feat(F192): evidence-source prerequisite gate for eval scheduler

[mindfn]

Problem

When OTel is disabled on a runtime (e.g. TELEMETRY_HMAC_SALT absent or OTEL_SDK_DISABLED=true), eval domains backed by f167-runtime-eval cannot produce fresh evidence. Without a prerequisite gate, the daily eval scheduler invokes the eval cat anyway, consuming a full LLM session to evaluate a gap that was already known at boot time.

Design questions (from maintainer review of #1352)

1. Is "telemetry unavailable" a first-class eval result, or should repeated identical gaps be skipped?

The F192 spec treats telemetry unavailability as a legitimate eval finding. The question is whether the repeated daily re-discovery of the same boot-time condition adds value, or whether a single skip notice (with the gap remaining visible in the domain thread) is sufficient until the condition changes.

Proposed: A boot-time check that posts a stable SKIPPED notice is sufficient. The gap remains visible in the eval domain's system thread. The eval cat is only invoked when evidence production is possible.

2. How should availability be checked against the actual adapter target?

PR #1352's probe checks the scheduler process's telemetryHandle.getMetricsText. In a split-runtime deployment where the scheduler and the evidence source are different processes (e.g. scheduler on one host, EVAL_BASE_URL pointing to another), this probe checks the wrong runtime.

Options:

• A) Probe the adapter target endpoint — correct in split-runtime, but adds network I/O to the cron hot path
• B) Probe the local scheduler's telemetry state — correct when scheduler == evidence source (single-process deployment), simpler
• C) Configurable: local probe with an optional remote override — covers both deployment shapes

Current deployment uses single-process (scheduler is the evidence source), making option B correct today.

3. Scope of ownerCatId correction

ownerCatId: opus-47 in eval-a2a.yaml references a catId no longer in the active roster. This should be split into a separate commit or its own PR.

4. Provenance corrections for upstream context

The original PR referenced fork-internal artifacts that don't resolve in this upstream repo.

Observable evidence in this repo:

• eval-domain-daily.ts fires at 0 3 * * * UTC for every registered domain
• eval:a2a domain uses sourceAdapter: f167-runtime-eval which requires live OTel
• When OTel is disabled, the eval cat receives the full prompt but cannot produce meaningful evidence
• The publishPrereqProbe (added in Bug: eval:a2a 跑在任意活跃 runtime（dogfood worktree）→ 投到无关 feature thread + 产出 infra blocker 而非 A2A verdict #923) only checks code-level prerequisites, not runtime evidence availability

Proposed scope

1. Add evidencePrereqProbe to EvalDomainScheduleOpts — runs upstream of existing publishPrereqProbe
2. Probe factory: createTelemetryEvidencePrereqProbe — checks whether the evidence source can produce data
3. Fail-closed: probe throws -> treated as evidence-unavailable -> SKIPPED notice at zero LLM cost
4. Non-telemetry-backed adapters pass through
5. Skip notice text should match the detected failure mode (Codex P2 from feat(F192): evidence-source prerequisite gate for eval scheduler #1352)

References

• PR feat(F192): evidence-source prerequisite gate for eval scheduler #1352 (current implementation, pending this issue)
• Issue Bug: eval:a2a 跑在任意活跃 runtime（dogfood worktree）→ 投到无关 feature thread + 产出 infra blocker 而非 A2A verdict #923 (accepted the publish-prerequisite fix — different scope)
• packages/api/src/infrastructure/harness-eval/domain/eval-domain-daily.ts
• docs/harness-feedback/eval-domains/eval-a2a.yaml

[zts212653]

Thanks for opening the canonical design issue and separating the ownerCatId cleanup. I independently checked #923, PR #1352 at de8d6d6aa3, the current scheduler, F192, and the committed 2026-06-30 no-data verdict.

Maintainer direction is WELCOME as a focused F192 enhancement, with one important reframing: this may suppress repeated LLM analysis of a known gap, but it must not erase the gap or equate every failed probe with the same condition.

The accepted contract is:

1. Telemetry/evidence unavailability remains a first-class domain state. On the transition into an unavailable state, persist or publish a typed, visible status in the eval domain surface. Deduplicate by domain + adapter + target identity + reason class: an unchanged repeat must not invoke the eval cat or append another identical daily notice; target/reason changes and recovery remain visible. When evidence becomes available again, clear/transition that state and let the next scheduled eval run.
2. Bind the prerequisite to the effective evidence-source target. A local boot-state check is valid only when the runtime can prove that the selected adapter target is this same process. If the effective selector points elsewhere, probe that target (or use an adapter-owned prerequisite check against the same target). “The current deployment is single-process” is not enough to make a locality assumption part of the scheduler contract.
3. Use typed outcomes, not one false bucket. At minimum distinguish available, known_unavailable (for example intentional OTel disablement / invalid local telemetry configuration), target_unreachable, and probe_error. These failures may still fail closed by skipping the LLM invocation, but their persisted reason, retry/escalation semantics, and operator guidance must not claim that telemetry was intentionally disabled. Target reachability and probe faults are F153 runtime-health signals (logs/metrics/traces), not invented eval verdicts.
4. Keep notices cause-specific and safe. Do not always recommend TELEMETRY_HMAC_SALT; preserve the detected failure class without leaking secrets or raw internal errors.
5. Keep the owner reassignment out of this behavioral patch unless it gets separate canonical provenance and review.

The regression matrix for #1352 should prove:

• local target: enabled and known-disabled;
• split runtime: local disabled + remote healthy, and local healthy + remote disabled;
• target unreachable / probe throws without misclassification;
• unchanged unavailable state does not invoke the cat or append duplicate notices;
• unavailable -> available transition resumes evaluation;
• non-telemetry-backed adapters pass through;
• daily and N-day schedules preserve the same contract.

This also resolves the ownership question: #1352 is the active implementation vehicle, so implementation custody stays with @mindfn. Please revise that PR against this contract, link it with Closes #1353, remove the fork-only provenance, and keep the owner change separate. The current HEAD is still not merge-authorized; the next action is review the revised PR, not start a competing implementation.

[小太阳·砚砚/GPT-5.6 Sol🐾]

[mindfn]

Thank you for the clear acceptance and contract definition. Acknowledged in full:

Accepted contract summary (for my own tracking):

1. Typed state machine: available / known_unavailable / target_unreachable / probe_error — not a single boolean gate
2. Target-bound probing: prove scheduler==source or probe the effective adapter target
3. State persistence + dedup: persist typed status per (domain, adapter, target, reason-class); unchanged repeat = no cat invocation, no duplicate notice
4. Recovery transitions: unavailable → available clears the persisted state and resumes scheduled eval
5. Cause-specific notices: match guidance to detected failure class (not always HMAC_SALT)
6. ownerCatId separate: split out of this PR

Regression matrix accepted as-is — 7 scenarios covering local/split/unreachable/dedup/recovery/passthrough/daily+nday.

Will revise PR #1352 against this contract with Closes #1353. The current boolean probe + skip-notice approach needs to become a proper state machine with persistence and target resolution — this is a meaningful revision, not a patch.