Skip to content

adv_run_test records run IDs that adv_subagent_report_submit cannot resolve — blocks acceptance/release gates #325

Description

@JRedeker

Summary

adv_run_test returns evidenceRecording: { status: "recorded", runId: "tr_..." } for a command, but adv_subagent_report_submit (and the verification-evidence gate consumer) then cannot resolve that same run ID — emitting consumer_warnings: [{ kind: "verification_missing", message: "No durable adv_run_test evidence found for run_id: tr_..." }].

The unresolved warning blocks the acceptance and release gates with VERIFICATION_EVIDENCE_MISSING, even though the test genuinely ran and the orchestrator recorded it. The only workaround is to call adv_verification_evidence_disposition with disposition: "fixed" after the final task report — and ordering matters: recording the disposition before submitting a new task report re-raises the concern.

Reproduction

Shipped two ADV changes today (pokeedge-web PRs #658 and #659, 2026-07-25). Both hit this on every evidence_policy: test and evidence_policy: static_check / artifact_reference task.

  1. Orchestrator runs a test via adv_run_test:
    adv_run_test({
      taskId: "tk-...",
      command: "bunx playwright test tests/e2e/card-detail-identity-chips.test.ts --project=mocked-mobile",
      phase: "green"
    })
    
  2. Tool returns success with a recorded run ID:
    {
      "success": true,
      "evidenceRecording": { "status": "recorded", "runId": "tr_ms0m3ov8_8557e960" }
    }
  3. A sub-agent (adv-designer, verify-and-bind) or the orchestrator submits a task report via adv_subagent_report_submit referencing that run ID in verification[].test_run_id.
  4. The report is accepted (success: true, duplicate: false) but the consumer emits:
    consumer_warnings: [{
      kind: "verification_missing",
      message: "No durable adv_run_test evidence found for run_id: tr_ms0m3ov8_8557e960"
    }]
    
  5. At adv_gate_complete for acceptance, the gate blocks:
    VERIFICATION_EVIDENCE_MISSING: Completed task tk-... (evidence_policy: test)
    has unresolved verification evidence: verification_missing.
    

Run IDs recorded today that fail to resolve

All returned evidenceRecording.status: "recorded" by adv_run_test; none resolvable by the report consumer:

  • tr_ms0m3ov8_8557e960
  • tr_ms0lj9qj_d625d6c6
  • tr_ms0lkf1w_6bee7c24
  • tr_ms0lme5w_8599a5bb
  • tr_ms0m0nc6_bf5e7462
  • (also observed on the prior change) tr_ms02ea84_f9753b9c, tr_ms02f5jd_fa2c0911, tr_ms02g1dd_78cc3308

Workaround in use today

For each blocked task:

  1. Submit an updated task report with evidence_binding_version: "typed-v1" and the run IDs in verification[].test_run_id.
  2. Call adv_verification_evidence_disposition with { disposition: "fixed", evidence: "<run IDs + reasoning>" } after the report submission. Ordering is load-bearing — a subsequent report re-raises the concern and requires re-recording the disposition.
  3. Retry adv_gate_complete.

This unblocks the gate but defeats the purpose of the durable-evidence check: the disposition is operator-attested, not registry-verified.

Expected behavior

A run ID returned by adv_run_test with evidenceRecording.status: "recorded" should be resolvable by adv_subagent_report_submit's verification consumer (and by the acceptance/release gate readiness check) without manual disposition.

Impact

  • Every evidence_policy: test / static_check / artifact_reference task on these two changes required manual disposition recovery at acceptance.
  • The pattern is deterministic, not flaky — it reproduces on every task with a proof-bearing evidence policy.
  • The verification gate's structural guarantee is currently being satisfied by operator attestation rather than registry proof, which weakens the evidence chain the gate is meant to enforce.

Environment

  • ADV plugin built from Sharper-Flow/Advance trunk (2026-07-25).
  • Target project: pokeedge-web (SvelteKit, Playwright E2E via bunx playwright test).
  • adv_run_test invoked from the orchestrator (main agent) root, not from sub-agents (sub-agent adv_run_test is role-firewalled).

Likely diagnostic question

Is adv_run_test's evidence-recording store the same store that adv_subagent_report_submit's verification consumer queries? The "recorded" status and the "not found" lookup point at either (a) two different stores, (b) a namespace/tenant mismatch, or (c) an async write that hasn't converged by the time the report consumer reads. Reproducing with a doctor pass immediately after adv_run_test may pinpoint which.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions