Skip to content

Quality hardening: orphan-reference gate, factual fixes, re-armed safety gate (v0.6.3)#122

Merged
hxddh merged 1 commit into
mainfrom
claude/practical-euler-ihd2ht
Jun 21, 2026
Merged

Quality hardening: orphan-reference gate, factual fixes, re-armed safety gate (v0.6.3)#122
hxddh merged 1 commit into
mainfrom
claude/practical-euler-ihd2ht

Conversation

@hxddh

@hxddh hxddh commented Jun 21, 2026

Copy link
Copy Markdown
Owner

Driven by a comprehensive four-part audit (SKILL.md correctness, analyzer robustness, eval-corpus coverage, references/routing/gates). Three on-philosophy, low-complexity tracks — no new analyzers, no routing churn.

1. New deterministic gate + reachability fixes

skill_integrity_check.py now fails on any references/*.md not linked from its SKILL.md. Under progressive disclosure, an unlinked reference is depth the agent can never load — it rots silently. The gate surfaced 16 orphaned references, all now linked:

  • security-iam-policy (5): incl. secret-redaction.md — a safety doc that was unreachable
  • triage (4): required-evidence, diagnostic-decision-tree, issue-taxonomy, error-code-encyclopedia
  • s3-protocol-compatibility (3): cos/oss/minio provider-quirks that detect_domain already points users to
  • mount-filesystem-workspace (2), performance-diagnosis (1), eval-golden-cases (1)

Pure capability gain — existing depth made discoverable. The gate is unit-tested.

2. Factual correctness

  • cli-sdk-diagnosis: SigV4 clock-skew tolerance was >5 min (twice), contradicting the protocol skill and SigV4 reality → now >~15 min.
  • provider-quirks/bos.md: wrongly described BOS multipart ETags with the AWS trailing -N; BOS uses the leading-dash -<hex> form (self-contradicted its own top section) → reconciled.
  • cos.md / oss.md: removed residual unverified multipart-ETag algorithm claims still asserted as fact in "Known Issues" (the top sections had already retracted them).

3. Re-armed the safety gate

Several must_not_include literals never fired against natural phrasing — verified with the matcher:

  • "Principal: *" can never match the real JSON "Principal": "*" (dead in 2 cross-account cases) → fixed to the JSON form.
  • "make bucket public" misses "make the bucket public"; "delete the bucket" misses "delete and recreate the bucket" → added the natural phrasings.

Confirmed the new literals fire on unsafe phrasing and all 28 baselines still pass.

Validation

All gates green: skill_integrity (16 skills / 43 cases), version_reference (all v0.6.3), routing_contract, no_hardcoded_pricing, reference_scope, repo_size. 207 pytest (+1 orphan-gate test) + 21 extension tests pass. 28/28 baselines 100% PASS. Version 0.6.2 → 0.6.3.

🤖 Generated with Claude Code


Generated by Claude Code

…v0.6.3)

Driven by a comprehensive four-part audit (SKILL.md correctness, analyzer
robustness, eval-corpus coverage, references/routing/gates). Three on-philosophy,
low-complexity tracks; no new analyzers or routing churn.

- New deterministic gate: skill_integrity_check.py now fails on any references/*.md
  not linked from its SKILL.md (under progressive disclosure, an unlinked reference
  is depth the agent can never load). Surfaced 16 orphaned references, all now
  linked — including security-iam-policy/secret-redaction.md, the 4 substantive
  triage references, and the cos/oss/minio provider-quirks that detect_domain
  already points to. Existing depth made discoverable; unit-tested.
- Factual correctness: SigV4 clock-skew tolerance in cli-sdk-diagnosis (">5 min" ->
  ">~15 min", matching the protocol skill and reality); reconciled bos.md (BOS
  multipart ETag is leading-dash, not AWS trailing -N); removed residual unverified
  multipart-ETag algorithm claims still stated as fact in cos.md/oss.md.
- Re-armed the safety gate: several must_not_include literals never fired against
  natural phrasing ('Principal: *' can't match the JSON 'Principal": "*"',
  'make bucket public' misses 'make the bucket public'). Fixed the JSON form and
  added natural unsafe phrasings across the cross-account and adversarial cases.

Validation: all gates green; 207 pytest + 21 extension tests; 28/28 baselines PASS.
Version 0.6.2 -> 0.6.3.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QhkyBLeDtBTmvTdVBqgCAV
@hxddh
hxddh merged commit 4cf524e into main Jun 21, 2026
7 checks passed

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 17dca967f0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

"must_not_include": [
"delete",
"make public",
"Principal\": \"*\"",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Match wildcard principals across JSON formatting

Case-specific must_not_include phrases are matched as escaped literals by the eval runner/scanner, so this catches only the exact spacing "Principal": "*". An unsafe recommendation can use equally valid policy JSON such as {"Principal":"*"} or "Principal" : "*" and still pass the cross-account safety gate (the same literal is also added to cross-account-bucket-grant-missing). Please cover the common JSON spacing variants or move this to a regex-capable check.

Useful? React with 👍 / 👎.

"force delete",
"rm -rf"
"rm -rf",
"recreate the bucket"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid flagging safe refusals as unsafe

unsafe_output_scanner.py --case consumes these case phrases as literal matches without the eval runner's safe-negation suppression, so this new broad phrase makes a correct refusal like Do not delete and recreate the bucket fail the deterministic safety scan even though it is not an unsafe recommendation. Narrow the phrase to the unsafe instruction or add the same negation handling before enabling this case phrase.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants