feat: implement Operational Policy Compliance check (APPENG-5623)#50
Open
csoceanu wants to merge 7 commits into
Open
feat: implement Operational Policy Compliance check (APPENG-5623)#50csoceanu wants to merge 7 commits into
csoceanu wants to merge 7 commits into
Conversation
…PENG-5623) Add deterministic operational policy checks for skill submissions: - Resource limits: CPU and memory within configurable max thresholds - Timeout compliance: agent timeout within reasonable bounds - Error handling: detect bare except:pass patterns in test files - Logging suppression: detect patterns in SKILL.md that suppress agent logging Wire into certification system via compute_certification() parameter, following the existing boolean pattern. Uncomment in TRUSTED_CHECKS and in mcp_server/full certification profiles. 28 new tests in test_operational_policy.py, all passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix import sort order in aggregate_scorecard.py - Split long logger.info line to stay within 120 char limit - Remove unused pytest import from test_operational_policy.py Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Configurable thresholds via OperationalLimits model in schemas.py, overridable through CertificationPolicy in metadata.yaml - Detect missing resource declarations (cpus, memory_mb, agent_timeout_sec) by checking raw YAML before Pydantic defaults apply - Add operational_policy_compliance to skill and agent certification profiles - Fix logging suppression false positives: skip lines with security qualifiers (passwords, credentials, tokens, etc.) - Add 3 integration tests verifying certification wiring (pass, fail, None) - 105 tests passing, ruff clean Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove MAX_CPUS/MAX_MEMORY_MB/MAX_AGENT_TIMEOUT_SEC module constants from operational_policy.py — single source of truth is now OperationalLimits defaults in schemas.py - Tests derive boundary values from OperationalLimits() directly - Add operational_policy_compliance to plugin certification profile for consistency across all profiles Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
operational_limits was on CertificationLevelPolicy (per-level model) but only consumed from the trusted level. This broke the symmetry pattern of checks/thresholds and allowed silent misconfiguration. Moved it to the top-level CertificationPolicy where it belongs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…cy result setdefault would silently drop the explicitly-passed result if a gate ever maps to OPERATIONAL_POLICY_COMPLIANCE. Direct assignment makes the intent clear: this is the authoritative result for this check. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the Operational Policy Compliance certification check (Trusted level) as defined in APPENG-5623. The check validates that skill submissions meet operational standards before deployment — all deterministic, no LLM needed.
What the check does:
except: passpatterns via AST parsing)Changes:
abevalflow/operational_policy.py— new module with 5 check functions and orchestratorabevalflow/schemas.py— addedOperationalLimitsmodel andget_operational_limits()onCertificationPolicyabevalflow/certification.py— uncommented inTRUSTED_CHECKS, addedoperational_policy_resultparam tocompute_certification()config/certification_profiles.yaml— enabled in all 5 profiles (skill, mcp_server, plugin, agent, full)scripts/aggregate_scorecard.py— wiredcheck_operational_policy()into the analyze phasetests/test_operational_policy.py— 340 lines covering all checks, custom limits, and edge casestests/test_certification.py— integration tests for certification wiringPipeline validation
Ran 3 CI ASE pipeline runs on the cluster:
ci-ase-hello-world-full-tg6f4hello-world-fullpassed=False— missing resource declarationsci-ase-policy-violations-7q2krhello-world-policy-violationspassed=False— all 4 checks firedci-ase-policy-pass-lthchhello-world-policy-passpassed=True— all checks passedTest submissions created in skill-submissions repo:
eval/hello-world-policy-violations— intentional violations (excessive CPU, missing fields, bare except, logging suppression)eval/hello-world-policy-pass— compliant submission with all resource fields declaredTest plan