Add skill lock configuration for evals-cli#301
Conversation
|
Looks good to me! Are you still planning more changes in this PR? I think going forward evaluating cases with SKILLS available will be very much needed |
|
@Kulikowski Thanks!
No, it's more in case we need more skills here that you folks are using and I may not be aware of. If we're OK with this, happy to merge.
Can you share a bit more on this? Like skills specific to testing? |
|
SKILLS specific to journeys - my bet is that going forward we will need a way to provide SKILLS for very complex worflows, like we have 140 tools, and a skill on how to complete specific journey with 10 tools only. It is still very early and there is no standardized way of providing SKILLS for WebMCP :) |
|
Definitely! Nice idea, I see how skills would make a lot of sense as journeys get more complex and involve more tools. Changed this PR to non-draft, feel free to review and I'll merge when LGTY. |
Add skill lock configuration for evals-cli, and vercel skill as per https://ai-sdk.dev/.
Are we happy with this living under evals-cli considering the evals tooling needs specific dependencies not needed in other directories, or do we think managing skills globally for webmcp-tools makes more sense?
Side note: while installing this skill I saw Vercel made a skill dedicated to help us migrate the AI SDK to the next version, could come in handy