feat: add SKILL.md security scanner gate (QEMETRICS-2414)#42
Merged
Conversation
Add a SecurityGate implementation that scans SKILL.md files for security risks. Plugs into the gate registry alongside the existing Cisco gate. Patterns ported from harness-eval-lab (setup-eval). Checks: prompt injection (17 patterns), credential access (22 patterns), data exfiltration (8), reverse shells (10), obfuscation (8). Context-aware severity demotion for patterns inside code blocks or examples. Also refactors SecurityGate base class to extract shared JSON-reading and scoring logic, removing duplication between Cisco and the new gate.
- Wire skillmd-security-scan into the test phase as step 3 - Remove exit 1 from Tekton task; let aggregate_scorecard.py decide pass/fail via gate policy (matches Cisco pattern) - Narrow markdown image pattern to exclude docs/github/imgur domains - Narrow translate evasion pattern to exclude known languages - Add directory excludes (.git, node_modules, vendor, __pycache__) - Add false-positive tests for narrowed patterns and dir excludes
These are shell/code patterns (curl, wget, netcat, bash -i) that target Python and shell files. The Cisco scanner already covers these in .py files. In markdown files they almost exclusively appear inside code blocks, where they get demoted to LOW severity and add noise without signal.
The scan runs inline as step 3 in pipeline/tasks/phases/test.yaml. The standalone task file was never wired into any pipeline.
GuyZivRH
previously approved these changes
Jun 24, 2026
GuyZivRH
left a comment
GuyZivRH
left a comment
Collaborator
There was a problem hiding this comment.
solve minor linting issues and mereg
GuyZivRH
approved these changes
Jun 25, 2026
This was referenced Jun 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new SecurityGate implementation that scans SKILL.md files in submissions for security risks using deterministic regex checks, complementing the existing Cisco scanner. Security scanning patterns originate from the harness-eval-lab repo (setup-eval) and are adapted for ABEvalFlow's gate architecture.
Also refactors the
SecurityGatebase class to extract shared JSON-reading and scoring logic intoevaluate_scan_json(), removing ~130 lines of duplication. Both CiscoGate and the new SkillMdScannerGate now delegate to this shared method.What it scans
The scanner checks markdown files for 5 categories of security risk:
Context-aware severity: patterns inside code blocks or example text ("for example", blockquotes) are demoted to LOW instead of blocking.
Narrowed patterns: markdown image skips trusted domains (docs, github, imgur), translate evasion skips known languages. Directory excludes: .git, node_modules, vendor, pycache, .venv.
Files
abevalflow/security/skillmd_scanner.pyscripts/skillmd_security_scan.pyabevalflow/gates/security/skillmd_scanner.pyabevalflow/gates/security/base.pyevaluate_scan_json()abevalflow/gates/security/cisco.pypipeline/tasks/phases/test.yamltests/test_skillmd_scanner.pyHow it integrates
The gate registers via
@register_security_gate(\"skillmd-scanner\")and is auto-discovered byaggregate_scorecard.pythroughget_all_security_gates(). No changes needed to scorecard, certification, or Compass facts.Task always exits 0; gate decides pass/fail via policy (matches Cisco pattern).
Test plan
./scripts/misc/trigger_test_runs.sh)Ref: QEMETRICS-2414
Replaces PR #38 (moved from fork to org branch per review feedback).">