v0.7.0 — Unify & Complete: contract unification + helpers for the last 2 skills + full eval coverage

.github/workflows/ci.yml

@@ -32,6 +32,12 @@ jobs:
       - name: Check provider scope
         run: python scripts/provider_scope_check.py
 
+      - name: Check output contract
+        run: python scripts/contract_check.py
+
+      - name: Check eval coverage floor
+        run: python scripts/coverage_check.py
+
       - name: Validate golden cases
         run: python skills/storageops-eval-golden-cases/scripts/golden_case_validator.py skills/storageops-eval-golden-cases/cases
 

CHANGELOG.md

@@ -1,5 +1,45 @@
 # Changelog
 
+## 2026-06-21 — v0.7.0: Unify & Complete (major iteration)
+
+A major iteration that closes the last structural gaps rather than adding features.
+Three tracks, all on-philosophy (deterministic, progressive disclosure, no
+LLM-judgment gates, no pricing), each locked by a new gate.
+
+### Track A — Output-Contract unification
+The eval corpus grades reports by section heading (`Summary` / `Key Evidence` /
+`Remediation`), but every `SKILL.md` Output Contract used its own divergent
+vocabulary (`Evidence`, `Recommendations`, `Fix`, `Root Cause`, …) — so a model that
+faithfully followed a skill emitted sections the benchmark did not credit. All 13
+diagnostic skills are now standardized to the canonical `Summary / Key Evidence /
+Remediation` (+ the existing `What Would Falsify This` / `Risks / Open Questions`
+floor); triage keeps `Routing / Evidence Gaps`; evidence-reporting aligned. Redundant
+double-encoded fields (`Root cause type:` / `Subsystem:` / `Failure point:`) were
+folded into the single `Primary Diagnosis: root_cause_type=…, affected_layer=…` line.
+New gate `contract_check.py` (CI) enforces the canonical vocabulary so it can't drift.
+
+### Track B — complete the deterministic-helper principle
+The two previously script-less skills get their first helpers:
+- triage → `evidence_completeness_checker.py`: deterministic present/missing +
+  readiness score against `required-evidence.md`, turning the "enough evidence?" step
+  into a structural check.
+- evidence-reporting → `report_contract_validator.py`: deterministic Output-Contract
+  check (required sections, well-formed confidence, redaction, no destructive
+  recommendations) — the same rules the eval applies, available before delivery.
+Both unit-tested; every skill now has at least one deterministic helper.
+
+### Track C — complete eval coverage
+- Authored the **15 missing baselines** (8 diagnosis + 7 routing cases): every one of
+  the now-48 golden cases has a reference output, so the regression gate exercises
+  100% of the corpus (was ~68%).
+- Added a 3rd `replication-versioning` case (`replication-dest-versioning-disabled`),
+  the thinnest data-loss-adjacent skill.
+- New gate `coverage_check.py` (CI): every baseline-enabled category must keep ≥2
+  golden cases and ≥1 baseline reference.
+
+Validation: 11 gates green (incl. the 2 new); 235 pytest (+16) + 21 extension tests;
+**48/48 baselines 100% PASS**. Version 0.6.8 → 0.7.0.
+
 ## 2026-06-21 — v0.6.8: Routing robustness, continued — constrain the last over-broad signals
 
 Finishes the routing-robustness pass started in v0.6.7 by constraining the three

docs/ARCHITECTURE.md

@@ -1,6 +1,6 @@
 # Architecture
 
-StorageOps v0.6.8 is a Pi Coding Agent extension and skill pack.
+StorageOps v0.7.0 is a Pi Coding Agent extension and skill pack.
 
 ## Components
 

docs/cli-reference.md

@@ -41,7 +41,7 @@ from the older local package. Deployment provenance is written to
 ## `storageops --version`
 
 ```text
-StorageOps v0.6.8  (pi: 0.78.0)
+StorageOps v0.7.0  (pi: 0.78.0)
   httpmon             : /root/.storageops/bin/httpmon
   api key             : api-key file
   independent install : yes  (~/.storageops/agent)

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "storageops"
-version = "0.6.8"
+version = "0.7.0"
 description = "StorageOps — S3-compatible object storage diagnostic toolkit. A Pi Coding Agent extension + skill pack."
 requires-python = ">=3.11"
 readme = "README.md"

scripts/contract_check.py

@@ -0,0 +1,66 @@
+#!/usr/bin/env python3
+"""Output-Contract consistency gate.
+
+The eval corpus grades reports by section headings (`Summary` / `Key Evidence` /
+`Remediation` for diagnosis, `Routing` / `Evidence Gaps` for triage). For the
+benchmark to measure what the skills actually instruct, every skill's Output
+Contract must use that same canonical vocabulary. Before v0.7.0 each skill had its
+own divergent headings (`Evidence`, `Recommendations`, `Fix`, ...), so a faithful
+report could miss the graded sections. This gate locks the vocabulary.
+
+Diagnostic skills must contain, as `##` headings in SKILL.md:
+  Summary, Key Evidence, Remediation, What Would Falsify This, Risks / Open Questions
+The router (triage) must contain: Routing, Evidence Gaps.
+The reporting skill must contain: Summary, Key Evidence, Remediation.
+"""
+
+from __future__ import annotations
+
+import re
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+SKILLS = ROOT / "skills"
+
+DIAGNOSTIC_REQUIRED = [
+    "Summary", "Key Evidence", "Remediation",
+    "What Would Falsify This", "Risks / Open Questions",
+]
+SPECIAL = {
+    "storageops-triage": ["Routing", "Evidence Gaps"],
+    "storageops-evidence-reporting": ["Summary", "Key Evidence", "Remediation"],
+}
+EXEMPT = {"storageops-eval-golden-cases"}
+
+
+def _headings(text: str) -> set[str]:
+    return {m.strip() for m in re.findall(r"^#{1,4}\s+(.+?)\s*$", text, re.M)}
+
+
+def main() -> int:
+    errors: list[str] = []
+    for skill_dir in sorted(SKILLS.glob("storageops-*")):
+        name = skill_dir.name
+        if name in EXEMPT:
+            continue
+        md = skill_dir / "SKILL.md"
+        if not md.exists():
+            continue
+        headings = _headings(md.read_text(encoding="utf-8"))
+        required = SPECIAL.get(name, DIAGNOSTIC_REQUIRED)
+        missing = [s for s in required if s not in headings]
+        if missing:
+            errors.append(f"{name}/SKILL.md: Output Contract missing canonical section(s): {missing}")
+
+    if errors:
+        print("Output-Contract check FAILED:")
+        for e in errors:
+            print(f"  - {e}")
+        return 1
+    print(f"Output-Contract check passed: all skills use the canonical section vocabulary")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

scripts/coverage_check.py

@@ -0,0 +1,67 @@
+#!/usr/bin/env python3
+"""Eval-corpus coverage floor.
+
+The golden-case corpus is the capability regression gate; a baseline-enabled skill
+with too few cases (or no reference baseline) is effectively untested. This gate
+enforces a floor so coverage cannot silently erode:
+
+  - every baseline-enabled category has at least MIN_CASES golden cases, and
+  - every baseline-enabled category has at least one baseline-output reference.
+
+Deterministic; reads the taxonomy, cases, and baseline-outputs.
+"""
+
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+CORPUS = ROOT / "skills" / "storageops-eval-golden-cases"
+CASES = CORPUS / "cases"
+BASELINES = CORPUS / "baseline-outputs"
+TAXONOMY = ROOT / "docs" / "skill-taxonomy.json"
+
+MIN_CASES = 2  # per baseline-enabled category
+
+
+def main() -> int:
+    tax = json.loads(TAXONOMY.read_text(encoding="utf-8"))["categories"]
+    baseline_cats = {c for c, e in tax.items() if e.get("baseline") is True}
+
+    case_cat: dict[str, int] = {}
+    baseline_cat: dict[str, int] = {}
+    baselines = {p.stem for p in BASELINES.glob("*.md")}
+    for case in sorted(CASES.iterdir()):
+        ej = case / "expected.json"
+        if not ej.exists():
+            continue
+        cat = json.loads(ej.read_text(encoding="utf-8")).get("expected_category")
+        if cat is None:
+            continue
+        case_cat[cat] = case_cat.get(cat, 0) + 1
+        if case.name in baselines:
+            baseline_cat[cat] = baseline_cat.get(cat, 0) + 1
+
+    errors: list[str] = []
+    for cat in sorted(baseline_cats):
+        n = case_cat.get(cat, 0)
+        b = baseline_cat.get(cat, 0)
+        if n < MIN_CASES:
+            errors.append(f"category {cat}: only {n} golden case(s), need >= {MIN_CASES}")
+        if b < 1:
+            errors.append(f"category {cat}: {n} case(s) but no baseline-output reference")
+
+    if errors:
+        print("Coverage check FAILED:")
+        for e in errors:
+            print(f"  - {e}")
+        return 1
+    print(f"Coverage check passed: {len(baseline_cats)} baseline categories, "
+          f"each >= {MIN_CASES} cases with a baseline")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

skill-registry.yaml

@@ -1,4 +1,4 @@
-# StorageOps Skill Registry v0.6.8
+# StorageOps Skill Registry v0.7.0
 
 # Machine-readable metadata for all StorageOps skills.
 # Skills are auto-discovered by Pi from the skills/ directory.

skills/storageops-access-log-analysis/SKILL.md

@@ -126,24 +126,21 @@ If the log analysis identifies a pattern but the root cause is unclear: **"Can y
 ## Output Contract — include these fields
 
 ```markdown
-# Access Log Analysis: [one-line summary]
+## Summary
+[one-line summary]
 **Route**: storageops-access-log-analysis
 **Provider**: AWS S3 | BOS | OSS | COS
-**Time Range**: YYYY-MM-DD HH:MM — YYYY-MM-DD HH:MM UTC
 **Confidence**: high | medium | low
 **Evidence Quality**: sufficient | partial | insufficient
 **Primary Diagnosis**: root_cause_type=[error-spike|access-pattern|anomaly|cost-attribution], affected_layer=[requester|credential|operation-mix|traffic-volume]
 
-## Key Metrics
-- Total requests: [N]
-- Error rate: [X%] (4xx: [N], 5xx: [N])
-- Top requester IP: [sanitized]
-- Top operation: [GET/PUT/LIST/DELETE] ([N]%)
+## Key Evidence
+- Time range: [start — end UTC]
+- Total requests: [N]; error rate: [X%] (4xx: [N], 5xx: [N])
+- Top requester: [sanitized]; top operation: [GET/PUT/LIST/DELETE] ([N]%)
+- What the log reveals about the user's question: [finding]
 
-## Root Cause
-[What the log analysis reveals about the user's question]
-
-## Recommendations
+## Remediation
 1. **[category]** (manual-only) — [specific action]
 2. **[category]** — [diagnostic or validation command]
 

skills/storageops-bigdata-pipeline/SKILL.md

@@ -110,22 +110,19 @@ If the diagnosis points to a committer issue:
 ## Output Contract — include these fields
 
 ```markdown
-# Diagnosis: [one-line]
+## Summary
+[one-line diagnosis]
 **Route**: storageops-bigdata-pipeline
-**Committer**: [type]
 **Confidence**: high | medium | low
 **Evidence Quality**: sufficient | partial | insufficient
 **Primary Diagnosis**: root_cause_type=[committer-race|partition-discovery|small-files|connection-pool|table-format|s3guard], affected_layer=[committer|metadata|filesystem-client|table-format]
 
-## Evidence
-- Engine: [Spark 3.x / Hive 3.x]
-- Committer: [V1 / magic / staging / partitioning]
+## Key Evidence
+- Engine: [Spark 3.x / Hive 3.x]; committer: [V1 / magic / staging / partitioning]
 - Table format: [plain / Iceberg / Delta / Hudi]
+- Explanation with config evidence: [finding]
 
-## Root Cause
-[Explanation with config evidence]
-
-## Recommendations
+## Remediation
 1. **[config change]** — `fs.s3a.committer.name=magic` (manual-only, test in staging)
 2. ...
 

skills/storageops-cli-sdk-diagnosis/SKILL.md

@@ -102,31 +102,23 @@ If s5cmd `--log debug` output is available, run `python3 scripts/parse_s5cmd_log
 ## Output Contract — include these fields
 
 ```markdown
-# Diagnosis: [tool] — [one-line]
+## Summary
+[tool] — [one-line diagnosis]
 **Route**: storageops-cli-sdk-diagnosis
 **Tool**: [name] [version]
-**Root cause type**: tool-bug | tool-version-incompatibility | misconfiguration | clock-skew | provider-incompatibility
 **Confidence**: high | medium | low
 **Evidence Quality**: sufficient | partial | insufficient
-**Primary Diagnosis**: root_cause_type=[type], affected_layer=[tool|provider|configuration|environment]
+**Primary Diagnosis**: root_cause_type=[tool-bug|tool-version-incompatibility|misconfiguration|clock-skew|provider-incompatibility], affected_layer=[tool|provider|configuration|environment]
 
-## Evidence
-- Error: [code + message]
-- Command: [sanitized]
-- Tool version: [known/unknown]
+## Key Evidence
+- Error: [code + message]; command: [sanitized]; tool version: [known/unknown]
+- Known issue match: [link to known issue in tool reference, or why it matches]
+- Cross-tool check: [does aws CLI succeed where rclone fails?]
 
-## Known Issue Match
-[Link to known issue in tool's reference or explanation of why it matches]
-
-## Fix
+## Remediation
 1. **[specific config/flag change]** — [rationale]
 2. **[workaround]** — [if applicable]
-
-## Cross-Tool Verification
-[If applicable: does aws CLI succeed where rclone fails?]
-
-## Validation Steps
-- [small, safe command or dry-run that proves the fix]
+- Validation: [small, safe command or dry-run that proves the fix]
 
 ## What Would Falsify This
 - [tool/version/provider evidence that would make the diagnosis wrong]

skills/storageops-data-consistency/SKILL.md

@@ -117,24 +117,19 @@ If the root cause is still unclear after Step 5:
 ## Output Contract — include these fields
 
 ```markdown
-# Diagnosis: [one-line]
+## Summary
+[one-line diagnosis]
 **Route**: storageops-data-consistency
 **Confidence**: high | medium | low
 **Evidence Quality**: sufficient | partial | insufficient
 **Primary Diagnosis**: root_cause_type=[client-cache|mount-cache|cdn-cache|multipart-not-completed|concurrent-write|etag-format|sse-kms-etag], affected_layer=[client|mount|cdn|object-store]
 
-## Timeline
-- Write: [timestamp]
-- Read attempt: [timestamp]
-- Stale data observed: [yes/no, timestamp]
+## Key Evidence
+- Timeline: write [ts] → read attempt [ts] → stale observed [yes/no, ts]
+- Cache layers identified: [layer] — [TTL state]
+- Explanation with evidence: [finding]
 
-## Cache Layers Identified
-1. [layer] — [TTL state]
-
-## Root Cause
-[Explanation with evidence]
-
-## Recommendations
+## Remediation
 1. **[fix]** — [cache invalidation, versioning enable, conditional write pattern]
 
 ## What Would Falsify This

skills/storageops-eval-golden-cases/baseline-outputs/cross-region-slow.md

@@ -0,0 +1,22 @@
+# Summary
+Category: performance_throughput
+Route: storageops-performance-diagnosis
+Confidence: 0.82
+Root Cause Type: high_rtt_wan_latency
+Evidence Quality: sufficient
+Primary Diagnosis: root_cause_type=high_rtt_wan_latency, affected_layer=network
+
+Throughput is limited by cross-region WAN latency, not the service. A single-stream
+transfer over a high-RTT link is bandwidth-delay-product bound, so each connection
+caps out far below the available link.
+
+# Key Evidence
+- RTT to the endpoint is high (cross-region), and observed single-stream throughput
+  in MiB/s is far below the link capacity.
+- Throughput scales with the number of parallel streams, the signature of a
+  TCP-window / latency limit rather than a server throttle.
+
+# Remediation
+- Use parallel/concurrent transfers (many streams) and multipart uploads so
+  aggregate throughput overcomes the per-connection bandwidth-delay-product limit.
+- Co-locate the client in the bucket region, or use an in-region relay, to cut RTT.