arena: A6 agent drive + A7 fleet + hardening waves 1-2#357
Draft
JayFarei wants to merge 189 commits into
Draft
Conversation
fix(trails): deadline-aware event-index maintenance + cat-file timeout PID proof (H1)
…OBSERVE Add write_compatibility_report, an atomic to_dict() writer analogue of write_parity_report, and call it from the literal-bytes compatibility proof so the run emits an inspectable compatibility-report.json under the pytest base alongside the capture-result and parity-report artifacts. The report round-trips exactly to PersistentCompatibilityReport.to_dict(), flipping the RED control from the previous commit to GREEN. Closes #311 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
polish(arena): refusal payload, origin docs, and A3 parity/report follow-ups (H4)
The #328 constant-time compare edit to server.ts changes source_sha256 and therefore invalidates the trusted-build pin (the packaging/provenance gate working as designed). Regenerate the pin from one actual compile through the repository build path (`build_hf_emulator_binary(update_trusted_manifest=True)`, bun 1.3.6, bun-linux-arm64) per the derive.md re-pin flow and the A5 precedent (adb19cc). Evidence carries the honest placeholder until a new real-box proof is attached; never a fabricated run line. Refs #328 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tion Review repair on PR #352 (Category A). The #323 owning-companion helper inferred ownership from legacy traces/v1 trace.json presence. A supported trace_record_only=True ingest writes the fresh canonical v2 record plus labels companion but skips the trace.json projection, so a stale project could be the only one holding trace.json: the helper silently selected the stale owner, omitted the fresher project's current labels, and bypassed fail-closed ambiguity handling. Ownership now binds through the same canonical corpus/freshness resolution normal reads use (trace_corpus.resolve): when multiple cross-project companions exist, decode ONLY the resolved owner's companion (the fresher canonical project); when the corpus cannot bind exactly one owning companion, fail closed with LabelIntegrityError instead of silently picking a project or unioning rows. The boundedness win is kept: at most one companion is decoded regardless of how many matching companions exist. RED-first controls (both fail on the pre-repair head): - test_summary_follows_the_canonical_corpus_owner_not_a_stale_trace_json_owner combines the two test_origin_join halves into the stale-owner scenario (older trace.json owner vs fresher canonical-only owner); before: assert 1 == 2 (stale owner's single row), after: the fresher owner's full labels. - test_summary_fails_closed_when_no_canonical_owner_binds_the_companions; before: DID NOT RAISE, after: fails closed on genuine ambiguity. - The #323 read-counting control re-proves boundedness: still exactly one _decode_rows call with 64 matching companions. Refs #323. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review strengthening on PR #352 (Category B) for #320. The original control was ASCII-only length/prefix/suffix. Add an 11-case parametrized parity matrix that computes the expected preview via the OLD implementation verbatim (whole-file Path.read_text(errors="replace") then slice) and asserts the bounded reader's output is byte-identical: - exact 4096 vs 4097 character boundary (and far-past-limit) - a 4-byte character straddling the preview read window - multibyte characters throughout (buffer chunks split sequences) - malformed bytes early, a truncated sequence at the boundary, and a truncated sequence at EOF (errors="replace" parity incl. final flush) - \r\n and lone-\r universal-newline translation parity - empty file Also asserts the remaining-budget arithmetic matches the old implementation for every case. Refs #320. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Real-box verification of the re-pinned sidecar build on a fresh ubuntu:24.04 Crabbox local-container lease: bench run of tests/arena/scenarios/test_browser_auth_reaches_hf.py produced run_20260717T012738618117Z_bf05a900157d with verdict=pass, execution_status=complete. The run's readiness/launch block binds the exact rebuilt binary (binary_sha256 83653f38..., source_sha256 311eff9b... = the constant-time-compare server.ts) to the running opentraces-hf process via the launch nonce; provenance=verified; device-flow browser authorization and a separately invoked CLI whoami authenticated; final ledger retained (ledger_binding sha256 stamped); RunStore.verify=true. Follows the A5 stamping precedent (d2726a8). Refs #328 #333 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review finding 1 on the #337 repair: the shared boundary-safe selector refuses a single clean id fused to a non-ASCII separator (em dash, NBSP, emoji), where the old permissive first-match regex accepted it. Ruling option (b): keep the ASCII-only fail-closed rule — real Crabbox output delimits the lease id with ASCII whitespace/punctuation, and the timed-out warmup path has pinned exactly this refusal (the unicode-left/ right cases) since A7, so widening the completed path would re-split the two paths' accept sets. Document the narrowing in the parser docstring and pin the completed-path em-dash and NBSP cases as INTENDED refusals (no guessed inspect/stop). No behavior change. Refs #337 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Four Category A repairs to the #335 purge path plus one Category B test strengthening on the #334 scanner, all RED-first (each control shown failing on the previous head 9f5efb3): - Repair fallback no longer follows symlinks: only a genuine NotImplementedError (no lchmod on the platform) may fall back to a plain chmod, and the fallback re-lstats and skips anything that became a symlink; a generic OSError never routes there. RED: injected OSError plus a post-lstat symlink swap chmod-ed the link TARGET (0o400 -> 0o644). - Repair enqueue is bounded inside the scandir loop: appending stops the moment seen + stack reaches _MAX_PURGE_REPAIR_ENTRIES, so one wide directory cannot blow past the budget. RED: 50 children consumed against a budget of 2. - The cleanup-required marker is unconditionally credential-free: every field passes through _credential_free_marker_value (hash-replace on a sensitive-byte match, same posture as _safe_sensitive_path_finding), with sensitive_values threaded through the purge call sites. RED: a store root containing the credential put the bytes in the marker. - Exception discipline: the FIRST deletion exception is captured once and always re-raised (retries no longer overwrite it), and a marker IO failure is recorded via sanitize_reason but never masks the primary error. RED: the final retry's exception propagated; a marker mkdir failure replaced the purge error. - Category B: the chunk-straddle control now parametrizes the split so the secret crosses the boundary with exactly one byte in the next chunk — the case only a full credential_length - 1 overlap catches. Verified by mutation: with overlap L-2 applied the new param fails while the old mid-split passes; scanner logic itself is byte-identical to the reviewed (verified-correct) form. Refs #334 Refs #335 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fix(arena): bounded label and action-output reads + labels decomposition (H1)
fix(arena): emulator control-plane hardening + evidence page containment (H2/H3/H4)
fix(arena): credential custody scanning/cleanup + strict lease identity (H2/H3)
A2 follow-up (#305), three Category-B parts: F1 - publish the compiled Bun emulator binary and its provenance record as one crash-consistent generation. The prior code committed the pair with two separate atomic renames while holding the materialization lock, so a crash between them left `output` pointing at new bytes with a stale/absent provenance sidecar (a torn generation recoverable only by a full rebuild). `_publish_generation` now stages both files in a fresh generation directory and makes them visible through a single atomic symlink swap of `output`; `emulator_provenance_path` follows that symlink so a reader always pairs the binary with the provenance from the same generation. A crash can leave only the prior generation or a fully-populated new one - never a torn pair. F2 - add a real two-process cold-cache control that races the cross-process materialization lock with only a controlled `npx` materializer on PATH. Both workers must return the identical verified executable while the materializer runs exactly once. (Verified RED: neutering the lock yields two materializations.) F3 - label the retained remote-only blocker prose in the A2 transcript as HISTORICAL, linking the amended #264 ruling and the A7 follow-up (#281), without relabeling the honest satisfies_remote_real_lease_gate=false stamp. A new evidence-hygiene control asserts the transcript cannot be read as the current, operative A2 gate. RED-first: the F1 crash-injection control fails on the pre-fix two-rename publish and passes on the single-pointer generation swap. Closes #305 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…298) Move the duplicated otbox-captures-v1 release-asset provenance verifier (source release / snapshot name+size+digest / session+metadata digests / derived-fixture digest) into one shared helper, tests/helpers/release_capture.py::verify_release_asset_provenance, and delegate both prior sites to it: A4's tests/test_slicing.py::test_coordinate_fixture_requires_a3_verified_release_chain (retiring its TODO(#265,#270)) and A3's portable-capture literal-legacy-bytes test. No genuine-capture provenance check is weakened; the real coordinate regression is unchanged. Add an authentic non-null Context join fixture: context_stamped_capture_record derives a TraceRecord from the same real session via the genuine Context Tree capture path (emit_context_tree_events_from_record) and stamps Step.context_node_id with the identical step_node_id_map join the ingest pipeline uses, so the node ids are real content-addressed hashes, not synthetic. The new test asserts that non-null Context join survives slicing-position to step-address materialization across the full tiling and a partial span. RED verified locally: dropping context_node_id in _steps_for_range fails the join-equality assertion; source restored. Closes #298 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tests/manual/generate_a7_guarantees.py accepts an explicit repository path but imported process-cached `tests.*` modules, so generating guarantees for two worktrees in one process bound both manifests to the first worktree's verifier digest (or fail-closed on coordinate shadowing when both roots are reachable). `canonical_guarantees` now loads verifier modules in a fresh child process that imports the verifiers exclusively from the requested repository, prints the manifest, and leaves the parent's sys.path/sys.modules untouched. The in-process build path is retained behind an internal `--emit-manifest` child entrypoint; `--check`, the canonical identity helper, and the no-execution import controls are unchanged. RED-first: two-repo generation in one parent process (A then B, and B then A) binds each repository's own guarantees.py bytes, is deterministic on repeat, and mutates no parent import state. The pre-fix generator fails these controls. Closes #338 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
test: unify release-capture verifier + Context join fixture (#298, H1)
fix(arena): crash-consistent Bun cache + isolated guarantee generation (H3/H4)
diagnostics.sanitize_diagnostic_text handled key=value assignments, bearer tokens, and host paths, but a bare credential in an assertion reason (assert resp == "sk-live-...") is not an assignment and reached result.json and the rendered OUTCOME REASON. Add a second pass over reason/diagnostic strings (never raw exhaust) that reuses the known token shapes and the conservative high-entropy heuristic from security/secrets rather than inventing a divergent regex vocabulary; PII/network shapes are excluded so benign diagnostic prose (region ids, timestamps, short shas) survives. RED first (tests/core/arena/test_diagnostics.py): bare sk-live-/sk-proj-/ sk-ant-/ghp_/xoxb-/AKIA tokens in a reason and inside a structured JSON reason are redacted while "tokenization completed" and low-entropy tokens survive untouched. Full tests/core/arena/ stays green (451 passed). Refs #302 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… runtime (#302 F5) Two hardenings from the A1 review: 1. The install-only app_state digest was computed over the wheel/dependency recipe only; provider and image were excluded even though base-only binds them, so two install-only runs on different images could share one digest. Add provider + image to the recipe material. This INTENTIONALLY changes install-only digests versus the pre-#302 wheel-only shape (disclosed here). 2. CrabboxRuntime accumulates per-run _diagnostics/_run_evidence_root instance state and was safe only because the CLI mints one runtime per run. Enforce that contract: configure_run_evidence (the per-run hook BenchRun.__enter__ already calls once) is now single-use and refuses a reused runtime with CrabboxRefusal("runtime_reused") rather than silently cross-contaminating a second run's evidence. RED first (tests/core/arena/test_box.py): same wheels + differing image or provider now yield distinct digests; a second configure_run_evidence on one runtime is refused. Full tests/core/arena/ green (454 passed). Refs #302 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The F6 evidence/durability batch from the A1 review, each RED-first: - Claim cross-assert: after reconcile, `bench run` asserts the stored scenario.claim equals static discover_claim(target); a divergence (they agree by construction only) is now a machinery error (ClickException, exit 1). - Storage-failure UX: on StorageFinalizeError the run command prints "provisional result retained at <recovery_path>" to stderr and exits 1 cleanly instead of crashing mute — the verdict is already safe in recovery. - Assertion-only verifiers: a passing verifier with empty evidence_refs keeps completing (algebra unchanged) but is now stamped observed:"assertion-only" (vs "evidence-backed"), and the evidence page renders an ASSERTION-ONLY badge so zero-evidence passes are visible. Additive key on the verifier record only. - Durability: _bucket_io atomic writes now fsync the temp file before replace and best-effort fsync the parent directory after, closing the power-loss window (helper _fsync_directory degrades gracefully where unsupported). - JTBD wording: the `bench run` row in tests/otbox/jtbd-command-map.md now states its owning journey is a help-surface smoke only, with behavioral coverage in tests/core/arena + tests/cli/test_bench_cli.py; the SSoT gate (tests/otbox/test_jtbd_ssot.py) stays green. RED first for each control. tests/core/arena (475 with cli+jtbd), security (409), and core bucket (78) suites all green. result-v0 changes are additive-only. Closes #302 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…reasons (#302 review A1+A2) Non-author codex review repair, RED-first with the review's exact probes: A1 — the unrestricted high-entropy scan over-redacted legitimate operator evidence: a base64 sha256 value became "sha256:[redacted]" and a Nix-store path component was cut to "[redacted].4.10". Entropy-only matches (no known credential shape) are now redacted ONLY when outside digest contexts (sha*/md5/blake2/digest prefix, or a pure-hex token) and path components (token containing or adjacent to "/"), AND a credential-like word (token/ secret/key/credential/password/auth/bearer/api) appears on the containing line. Known credential shapes (sk-*, ghp_, xox*, AKIA, ...) redact regardless of context, unchanged. A2 — the shared security/secrets scanner returns early when the text STARTS with a recognized decorator, so "@pytest.mark.parametrize ... sk-live-..." passed a bare credential through unchanged. Per the review ruling the shared scanner is NOT touched (its detection logic is governed by SECURITY_VERSION policy); the diagnostics wrapper now rescans past the leading decorator token and merges the two scans' spans before replacement. RED first (tests/core/arena/test_diagnostics.py): base64-sha256 digest, Nix-store path, generic high-entropy path component, and the decorator-masked sk-live probe; plus pins for the already-safe hex digest and for the bounded catch keeping its teeth (entropy token WITH credential context still redacts). tests/core/arena green. Refs #302 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ew A3) The F6 cross-assert ran AFTER _finalize_after_pytest published an integrity-verified result: on divergence the CLI exited 1 but the finalized store still carried a consumable pass. The check now runs INSIDE _finalize_after_pytest, before finalize: on divergence the stored result is demoted to the machinery-error state (execution_status=error, verdict=null, reason.code=claim_reconciliation_mismatch, evidence.complete=false with a scenario.claim requirement) — consistent with the honesty-outcomes vocabulary — so the run is retained as evidence, never as a verdict. The CLI still surfaces the mismatch as a ClickException (exit 1). Fleet's caller is unchanged (it passes no expected_claim; bench run is the surface that owns static discovery). RED first: the repaired test asserts STORAGE state — after a divergent run, no result.json anywhere in the store is a consumable pass (verdict null, execution_status error, mismatch reason code); on the previous head the store held verdict="pass". Refs #302 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…review A4) The F5 provider/image binding lives in the recipe construction SHARED by install-only and agent-ready, so BOTH app_state digests changed — but only install-only was tested. Per the review ruling the shared behavior is KEPT (provider and image shape the agent-ready world too; excluding them there would reintroduce the same collision): this commit adds the agent-ready twin of the digest-divergence test so both seams are pinned, and discloses that agent-ready digests intentionally changed alongside install-only ones. Control proven RED by construction: with the two recipe binding lines removed, the twin test fails exactly like its install-only siblings (3 failed), then all 57 box tests green with the binding restored. Refs #302 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e is non-fatal (#302 review B) The F6 fsync change left failure semantics unpinned. Three controls: 1. Ordering pinned: the temp-content fsync must land BEFORE Path.replace commits the new directory entry (spy-sequenced test). 2. File-fsync failure semantics pinned as-is: a durability primitive must not pretend a failed fsync succeeded — the OSError propagates, the old target bytes survive untouched, and the temp file is cleaned up. 3. Fixed: an os.close failure on the directory fd inside _fsync_directory could escape AFTER the rename had already committed, surfacing a successful write as an error. Close failures there are now swallowed (the directory fsync itself was already best-effort). RED first for (3) (simulated os.close failure escaped on the previous head); (1) and (2) proven holding and locked so they cannot drift. Refs #302 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fix(arena): bench diagnostics, runtime lifecycle, and evidence durability (#302, H3)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Founder-gated integration of the
arenabranch intomain. DRAFT deliberately — the delivery agent does not merge to main.What this train carries
Verification at this head: arena + emulator + packaging + bench CLI + SSoT suites, 527 passed. Full narrative: #255 status comment, #348 wave reports, private ledger.
Closes on merge
Closes #275, Closes #276, Closes #277, Closes #278, Closes #279, Closes #280, Closes #281, Closes #282, Closes #283, Closes #284, Closes #285, Closes #286
(The 22 wave issues — #298, #302, #305, #309, #310, #311, #314, #316, #319, #320, #323, #324, #326, #328, #329, #330, #332, #334, #335, #337, #338, #343 — auto-close via the
Closeslines their implementation commits carry.)🤖 Generated with Claude Code