Grant scoped tool permissions to skills under test#3
Merged
Conversation
Eval runs invoked claude -p with no tool permissions, so every Bash call was auto-denied with "requires approval". Skills that diagnose by running read-only commands (kubectl get, gh api, gcloud ... list) could never execute their steps and failed spuriously, the eval measured the permission gate rather than the skill. Add allowed-tools and permission-mode action inputs, forwarded to the execute call as claude --allowedTools / --permission-mode. Support per-case allowed_tools (string or list) and permission_mode overrides in eval YAML so a refusal-of-mutation case can widen scope while diagnosis cases stay read-only. Validate the new fields before any API calls and cover normalization plus validation with unit tests. Scoped allow-lists are preferred over a blanket bypass so read-only commands run while mutation-refusal behavior stays testable.
…ools-permission # Conflicts: # scripts/eval.py
CI's zizmor (unpinned, bumped to 1.26.1) now fails the adhoc-packages audit on `npm install -g @anthropic-ai/claude-code`: an ad-hoc install leaves transitive deps unpinned. Version pinning alone does not satisfy the audit, the documented remediation is a committed lockfile installed with `npm ci`. Add package.json + package-lock.json pinning the claude CLI and install it with `npm ci --prefix`. The eval step prepends the local node_modules/.bin to PATH for its own shell instead of writing to GITHUB_PATH (which trips zizmor's github-env audit). Ignore node_modules.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2.
Problem
Eval runs invoked
claude -pwith no tool permissions, so everyBashcall was auto-denied with "requires approval". Skills that diagnose by running read-only commands (kubectl get,gh api,gcloud ... list) could never execute their steps and failed spuriously. The eval ended up measuring the permission gate rather than the skill.Change
allowed-toolsandpermission-modeaction inputs, forwarded to the execute call asclaude --allowedTools/--permission-mode(flag names verified againstclaude --help).allowed_tools(string or list) andpermission_modeoverrides in eval YAML, so a refusal-of-mutation case can widen scope while diagnosis cases stay read-only.Scoped allow-lists are preferred over a blanket bypass so read-only commands run while mutation-refusal behavior stays testable.
Verification
python3 scripts/test_validation.pypassesruff check scripts/cleanaction.ymlpasses the CI schema checkDocs on skill-bench.dev are updated separately.