Skip to content

ci: consolidate hermetic validation - #65

Merged
steipete merged 4 commits into
mainfrom
codex/ci-docs-hardening
Aug 14, 2026
Merged

ci: consolidate hermetic validation#65
steipete merged 4 commits into
mainfrom
codex/ci-docs-hardening

Conversation

@steipete

Copy link
Copy Markdown
Collaborator

Summary

  • replace three overlapping test/lint workflows and disabled/soft-fail jobs with one blocking macOS/Linux CI contract
  • move credentialed provider smoke tests to a manual fail-closed workflow that requires at least one configured provider secret and enables both live-test gates
  • remove warning-only API-baseline/DocC checks that never validated this repository
  • remove the obsolete, unbuildable Agent CLI prototype, dead disabled/placeholder tests, and unlinked vendored planning/reference dumps
  • refresh model, MCP transport, testing, and local-provider docs to match the current source

Proof

  • TACHIKOMA_TEST_MODE=mock TACHIKOMA_DISABLE_API_TESTS=true swift test --parallel (837 Tachikoma tests + 57 MCP tests)
  • TACHIKOMA_TEST_MODE=mock TACHIKOMA_DISABLE_API_TESTS=true swift build -c release
  • swiftformat --lint .
  • swiftlint lint --config .swiftlint.yml --strict
  • actionlint .github/workflows/*.yml
  • package dump/describe/dependency-contract checks
  • local-link audit across all 18 remaining Markdown files
  • structured autoreview clean across the complete three-part bundle

Scope

No provider/model runtime API, release, or tag behavior changes.

@clawsweeper

clawsweeper Bot commented Aug 14, 2026

Copy link
Copy Markdown

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

@clawsweeper clawsweeper Bot added merge-risk: 🚨 security-boundary 🚨 Merging this PR could weaken sandboxing, authorization, credentials, or sensitive data. P1 Urgent regression or broken agent/channel workflow affecting real users now. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Aug 14, 2026
@clawsweeper

clawsweeper Bot commented Aug 14, 2026

Copy link
Copy Markdown

Codex review: needs maintainer review before merge. Reviewed August 14, 2026, 5:20 AM ET / 09:20 UTC.

ClawSweeper review

What this changes

Consolidates Swift validation into one hermetic macOS/Linux workflow, adds a default-branch-only manual provider smoke workflow, removes disconnected examples and documents, and updates related documentation.

Merge readiness

⚠️ Ready for maintainer review - 2 items remain

Keep open for an explicit collaborator-owned merge decision. The latest head fixes the prior false-green live-provider gate, and no blocking patch defect was found; branch-protection status checks must be confirmed against the consolidated workflow before merge.

Priority: P2
Reviewed head: 31f384ef8ff5be7fd5ba19dcaf2a9176713e756b
Owner decision: Required. See Decision needed.

Review scores

Measure Result What it means
Overall readiness 🐚 platinum hermit (4/6) The patch is coherent and resolves the previously identified false-green provider gate, pending confirmation of required-check continuity.
Proof confidence 🌊 off-meta tidepool Not applicable: This collaborator-authored CI and documentation change is not subject to the external contributor real-behavior proof gate; exact-head CI validation is still in progress.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Not applicable Not applicable: This collaborator-authored CI and documentation change is not subject to the external contributor real-behavior proof gate; exact-head CI validation is still in progress.
Evidence reviewed 6 items Prior live-gate blocker resolved: The provider suite now requires an eligible credential and lets configured-provider request failures and failed expectations fail the test, replacing the prior warning-only behavior identified in the previous review.
Credential boundary is constrained: The new live-provider workflow is manually dispatched, uses read-only contents permission, runs only on the default branch, and checks out that branch before exposing provider secrets.
Workflow status surface changes: The branch removes three workflow files and replaces prior jobs with new consolidated job names, so protected-branch required-check configuration needs an explicit compatibility check.
Findings None None.
Security None None.

How this fits together

GitHub Actions builds, tests, and styles the Swift package on supported runners, while a separate manual workflow runs credentialed provider requests. These workflows turn repository source and provider credentials into merge signals for contributors and maintainers.

flowchart LR
  A[Source change] --> B[Hermetic CI]
  B --> C[macOS and Linux validation]
  A --> D[Manual provider workflow]
  E[Repository secrets] --> D
  D --> F[Live provider smoke tests]
  C --> G[Merge status checks]
  F --> G
Loading

Decision needed

Question Recommendation
Should protected-branch required status checks be migrated to the new consolidated CI job names before this workflow replacement merges? Migrate required checks first: Confirm the new Swift, package, and style jobs provide the intended required checks and update any protected-branch rules before merge.

Why: The diff demonstrably removes old workflow jobs, but repository branch-protection configuration is an external maintainer-controlled contract that source inspection cannot determine.

Before merge

  • Complete next step (P2) - A collaborator should confirm required-check migration and review the exact-head CI results; there is no narrow source repair to dispatch.
Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Workflow consolidation 3 workflows removed, 1 consolidated workflow changed, 1 manual workflow added The PR intentionally changes the repository's merge-validation contract and its status-check names.
Repository cleanup 315 added, 28,546 removed across 23 files Most of the diff is removal of excluded example code and disconnected reference material, so the broad deletion warrants maintainer scope review.

Merge-risk options

  • This PR removes and renames CI status checks; if protected-branch rules still require the former workflow job names, future merges could be blocked or enforcement could be unintentionally reduced until those rules are migrated.

Maintainer options:

  1. Verify required-check continuity (recommended)
    Before merging, confirm that branch protection requires the intended replacement jobs rather than deleted workflow job names.
  2. Accept the status-check transition
    Merge as-is only after confirming that no protected branch rule or downstream automation relies on a removed check name.

Technical review

Best possible solution:

Land a single hermetic validation contract only after the protected-branch required-check set is confirmed or migrated to the new job names and the exact-head matrix completes successfully.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this is a CI and cleanup PR rather than a user-reported runtime failure; the updated workflow and test gates are directly inspectable in the proposed source.

Is this the best way to solve the issue?

Unclear until required-check continuity is confirmed. The consolidated workflow and fail-closed manual provider gate are coherent, but deleting old job names must not silently alter protected-branch enforcement.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 7ec311aec8d4.

Labels

Label changes:

  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: This collaborator-authored CI and documentation change is not subject to the external contributor real-behavior proof gate; exact-head CI validation is still in progress.
  • remove status: ⏳ waiting on author: Current PR status label is status: 👀 ready for maintainer look.
  • remove rating: 🦐 gold shrimp: Current PR rating is rating: 🐚 platinum hermit, so this older rating label is no longer current.

Label justifications:

  • P2: This is a bounded CI and repository-maintenance change with meaningful developer workflow impact but no demonstrated active user regression.
  • merge-risk: 🚨 automation: Replacing several workflow jobs changes the status-check interface that branch protection and downstream automation may consume.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: This collaborator-authored CI and documentation change is not subject to the external contributor real-behavior proof gate; exact-head CI validation is still in progress.

Evidence

What I checked:

Likely related people:

  • steipete: Current-main CI hardening and the current live-provider gate repair both trace to Peter Steinberger, who is also the collaborator author of this PR. (role: recent CI and test-history contributor; confidence: high; commits: a9ff00bef92c, 31f384ef8ff5; files: .github/workflows/ci.yml, .github/workflows/live-providers.yml, Tests/TachikomaTests/Providers/Integration/ProviderIntegrationTests.swift)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

  • Confirm protected-branch required status checks map to the consolidated jobs and let the current macOS, Linux, and style checks finish on this head.

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (3 earlier review cycles)
  • reviewed 2026-08-14T08:51:15.278Z sha 3962fbc :: needs changes before merge. :: [P1] Restrict secret-bearing runs to the default branch
  • reviewed 2026-08-14T08:59:21.988Z sha 01f2380 :: needs changes before merge. :: [P3] Remove release-owned changelog entries
  • reviewed 2026-08-14T09:07:50.831Z sha 9fa0932 :: needs changes before merge. :: [P1] Fail the live gate when a configured provider request fails

@steipete

Copy link
Copy Markdown
Collaborator Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Aug 14, 2026

Copy link
Copy Markdown

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

@clawsweeper clawsweeper Bot added P2 Normal priority bug or improvement with limited blast radius. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. P1 Urgent regression or broken agent/channel workflow affecting real users now. merge-risk: 🚨 security-boundary 🚨 Merging this PR could weaken sandboxing, authorization, credentials, or sensitive data. labels Aug 14, 2026
@steipete

Copy link
Copy Markdown
Collaborator Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Aug 14, 2026

Copy link
Copy Markdown

🦞👀
Exact review queued.

Re-review progress:

@clawsweeper clawsweeper Bot added merge-risk: 🚨 automation 🚨 Merging this PR could break CI, automerge, proof capture, label sync, or automation. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Aug 14, 2026
@steipete

Copy link
Copy Markdown
Collaborator Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Aug 14, 2026

Copy link
Copy Markdown

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Aug 14, 2026
@clawsweeper clawsweeper Bot removed status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. labels Aug 14, 2026
@steipete
steipete merged commit 0d2eb8d into main Aug 14, 2026
9 checks passed
@steipete
steipete deleted the codex/ci-docs-hardening branch August 14, 2026 09:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 automation 🚨 Merging this PR could break CI, automerge, proof capture, label sync, or automation. P2 Normal priority bug or improvement with limited blast radius. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant