feat: Add coordination scripts for managing gap slices and state

.gitignore

@@ -9,3 +9,10 @@ __pycache__/
 .ruff_cache/
 frontend/node_modules/
 frontend/dist/
+frontend/artifacts/
+docs/coordination.defaults.toml
+docs/coordination/current_gap_slice.md
+docs/coordination/focus.txt
+docs/coordination/inbox/
+docs/coordination/state.json
+docs/coordination/verification_target.md

AGENTS.md

@@ -18,6 +18,8 @@ Primary goals:
 ## Conventions
 - Treat the runner repo as the telemetry producer.
 - Consume telemetry through external interfaces; do not rely on runner internal imports as the primary path.
+- This repo owns the telemetry gap ledger and the operator question checklist.
+- Use dashboard-facing failures to define upstream telemetry work; do not silently normalize away telemetry truth gaps.
 - Prefer clear separation between:
   - decisions in `docs/decisions/`
   - implementation specs in `docs/specs/`

README.md

@@ -19,9 +19,21 @@ That means:
 
 The current architecture notes live in:
 - `docs/dashboard_repo_architecture.md`
+- `docs/gap_ledger.md`
+- `docs/current_state_audit.md`
+- `docs/operator_questions.md`
+- `docs/coordination_protocol.md`
 - `docs/decisions/`
 - `docs/specs/`
 
+## Product accountability
+This repo owns the operator-facing telemetry gap ledger.
+
+That means:
+- this repo defines what the dashboard still cannot answer
+- this repo records which upstream telemetry gaps are blocking usefulness
+- `model-runner` remains accountable for closing upstream telemetry truth gaps
+
 ## Setup
 ```bash
 python -m venv .venv
@@ -60,3 +72,72 @@ Integrated build served by Python:
 cd frontend && npm run build && cd ..
 ./scripts/dev-backend.sh --no-replay-fixtures
 ```
+
+## Coordination
+Push the current 3-5 upstream gap slice into `model-runner`:
+```bash
+./scripts/push-gap-slice.sh 5
+```
+
+Pull the latest upstream response back into this repo:
+```bash
+./scripts/pull-model-runner-response.sh
+```
+
+Run one upstream implementation pass automatically:
+```bash
+./scripts/run-model-runner-gap-pass.sh
+```
+
+Run the recursive loop:
+```bash
+./scripts/run-gap-loop.sh
+```
+
+Loop defaults live in:
+```bash
+docs/coordination.defaults.toml
+```
+
+Start from:
+```bash
+cp docs/coordination.defaults.example.toml docs/coordination.defaults.toml
+```
+
+Loop state lives in:
+```bash
+docs/coordination/state.json
+```
+
+Optional slice focus lives in:
+```bash
+docs/coordination/focus.txt
+```
+
+Start from:
+```bash
+cp docs/coordination/focus.example.txt docs/coordination/focus.txt
+```
+
+Add one `GAP-...` id per line to steer the next slice toward a narrow issue.
+
+The upstream mirror also includes a generated verification target so `model-runner` can run a real telemetry-producing command after emission changes instead of claiming fixes without evidence.
+
+## Dashboard Audit
+Single-command audit bundle:
+```bash
+./scripts/audit-dashboard.sh
+```
+
+If the backend is not already running:
+```bash
+./scripts/audit-dashboard.sh --start-backend --telemetry-jsonl /path/to/model-runner-telemetry.jsonl
+```
+
+Artifacts are written under `frontend/artifacts/`:
+- `latest.png`
+- `history-<timestamp>.png`
+- `api-status-<timestamp>.json`
+- `api-sessions-<timestamp>.json`
+- `api-session-detail-<timestamp>.json`
+- `audit-meta-<timestamp>.txt`

docs/coordination.defaults.example.toml

@@ -0,0 +1,8 @@
+model_runner_root = "/path/to/model-runner"
+telemetry_jsonl = "/path/to/model-runner/models/YourModel/backend/config/telemetry/session.jsonl"
+api_base = "http://127.0.0.1:8001"
+dashboard_url = "http://127.0.0.1:8001/"
+gap_slice_count = 5
+loop_iterations = 3
+codex_exec_mode = "danger-full-access"
+verification_policy = "auto"

docs/coordination/focus.example.txt

@@ -0,0 +1,8 @@
+# Optional focus steering for the gap slice generator.
+#
+# Copy to `docs/coordination/focus.txt` to activate.
+# Add one GAP id per line.
+#
+# Example:
+# GAP-001
+# GAP-015

docs/coordination_protocol.md

@@ -0,0 +1,105 @@
+# Coordination Protocol
+
+This repo owns the gap ledger.
+
+`model-runner` owns upstream telemetry implementation.
+
+The file-based loop is:
+1. Audit the dashboard in `lab-llm`.
+2. Update `docs/gap_ledger.md`.
+3. Generate a 3-5 gap slice.
+4. Push the mirrored bundle into `model-runner`.
+5. `model-runner` implements the slice, runs a real telemetry-producing verification path when telemetry behavior changed, and writes `docs/downstream/lab_llm/response.md`.
+6. Pull the response back into `lab-llm`.
+7. Re-run the dashboard audit, preferring the verified session id or evidence target from the upstream response when present.
+8. Only if the audit shows a genuine local projection/wiring issue, let `lab-llm` make the minimal read-model/API/UI change needed to expose the new telemetry truth.
+9. Audit again, update the ledger, and repeat.
+
+## Commands
+Generate and push the current slice to `model-runner`:
+```bash
+./scripts/push-gap-slice.sh
+```
+
+Pull the latest `model-runner` response back here:
+```bash
+./scripts/pull-model-runner-response.sh
+```
+
+Run one upstream implementation pass automatically:
+```bash
+./scripts/run-model-runner-gap-pass.sh
+```
+
+Pull response, rerun audit, and update the ledger here:
+```bash
+./scripts/reconcile-gap-ledger.sh
+```
+
+Run a multi-iteration loop:
+```bash
+./scripts/run-gap-loop.sh
+```
+
+All defaults come from:
+```bash
+docs/coordination.defaults.toml
+```
+
+Bootstrap from:
+```bash
+cp docs/coordination.defaults.example.toml docs/coordination.defaults.toml
+```
+
+Optional manual steering lives in:
+```bash
+docs/coordination/focus.txt
+```
+
+Bootstrap from:
+```bash
+cp docs/coordination/focus.example.txt docs/coordination/focus.txt
+```
+
+If that file contains one or more gap ids, the slice generator will prefer those gaps before normal ranking.
+This is the intended way to force a narrow proof pass without disabling automation.
+
+That file currently defines:
+- `model_runner_root`
+- `telemetry_jsonl`
+- `api_base`
+- `dashboard_url`
+- `gap_slice_count`
+- `loop_iterations`
+- `verification_policy`
+
+## Practical limit
+This loop can automate most coordination work, but it still depends on:
+- `model-runner` being able to make real upstream fixes without interactive blockers
+- a live or restartable dashboard/backend target for audit
+- telemetry changes being observable in audit artifacts
+
+It should reduce manual passes heavily, but “wake up with no gaps” is only realistic when the remaining gaps are actually automatable and verifiable from live artifacts.
+
+## Unattended mode
+`run-gap-loop.sh` uses `codex exec --dangerously-bypass-approvals-and-sandbox` for the outer loop so it does not stop for approvals overnight.
+
+That is intentional for unattended local automation.
+Use it only in the repo/workspace you are comfortable granting that level of autonomy.
+
+## Canonical locations
+Source of truth in this repo:
+- `docs/gap_ledger.md`
+- `docs/operator_questions.md`
+- `docs/current_state_audit.md`
+- `docs/coordination.defaults.example.toml`
+- `docs/coordination/focus.example.txt`
+
+Mirrored bundle in `model-runner`:
+- `docs/downstream/lab_llm/`
+
+Response file expected from `model-runner`:
+- `docs/downstream/lab_llm/response.md`
+
+Verification target mirrored into `model-runner`:
+- `docs/downstream/lab_llm/verification_target.md`

docs/current_state_audit.md

@@ -0,0 +1,59 @@
+# Current State Audit
+
+Date: 2026-03-16
+
+This file is a blunt assessment of the current dashboard against `docs/successful_lab.md`.
+
+## Latest evidence reviewed
+- `frontend/artifacts/latest.png`
+- `frontend/artifacts/history-2026-03-16T08-06-19-037Z.png`
+- `frontend/artifacts/api-status-20260316T080618Z.json`
+- `frontend/artifacts/api-sessions-20260316T080618Z.json`
+- `frontend/artifacts/api-session-detail-20260316T080618Z.json`
+- `docs/coordination/inbox/response.md`
+- `docs/coordination/inbox/artifacts/hf-qwen35-9b-sess_d8f3abe025ee4201abab2d30fa06188f.jsonl`
+
+## What already exists
+- fixture and JSONL ingestion
+- local REST/SSE service
+- session list/detail shell
+- basic throughput/GPU/KV charts
+- load/runtime/turn/log/inspect surfaces
+
+## What is already directionally correct
+- the repo split is correct
+- the dashboard is built around operator-facing sessions, not generic metrics alone
+- projected truth is available in-place
+- upstream canonical telemetry has started to improve for newly verified sessions:
+  - model display name is split from model path
+  - requested runtime intent is split from confirmed runtime truth, with mismatches called out explicitly
+  - activity state is split from session lifecycle status
+  - KV cache unavailability is explicit instead of silent omission
+  - throughput now carries trust/measurement-state semantics
+  - completed-turn throughput now separates raw completion speed from effective request speed
+  - turn-finished timing can include TTFT plus honest decode latency
+  - turn stop reason can include source attribution
+  - log messages can arrive without duplicated timestamp/source prefixes
+  - GPU samples can include utilization/power/temperature plus explicit diagnosis incompleteness
+
+## What is still failing usefulness
+- the audited frontend/API is still serving older path-only and numeric-only projections even though the newer verified finished session `sess_d8f3abe025ee4201abab2d30fa06188f` is already present in the sessions list
+- the selected audited session `sess_c7acaf69b1cc451081d7038203fd9c1e` started at `2026-03-16T06:01:57.731Z`, before the upstream verification run at `2026-03-16T08:03:29.432Z`, so the dashboard is still showing a pre-fix projection for the main detail pane
+- too many summaries are still thin wrappers around incomplete telemetry
+- model identity is still not trustworthy on the rendered surface
+- the top summary still says `Latest event Not yet` even though the audited detail payload has `session.last_event_at: 2026-03-16T08:06:17.491Z` and `latest_runtime.ts: 2026-03-16T08:06:17.488Z`
+- tok/s is still visible before its trustworthiness is clear
+- session lifecycle and current activity are still collapsed into one `running/live` story
+- GPU and KV cards still spend primary space on unavailable signals
+- the recent-turn strip still shows `TTFT Unavailable` and does not surface stop reason or plain-language timing
+- the rendered log rows are still duplicated/noisy, and the audited detail payload does not expose log entries for cross-checking
+- the inspect path stops at projected truth instead of exposing the canonical event fields that now matter
+- source/path status is still unavailable on the audited backend
+- the session list is still too ambiguous to help an operator pick the right run quickly, even though the sessions API already has timestamps and error state
+- bottleneck diagnosis is not possible yet
+- several important question areas from `docs/successful_lab.md` have no meaningful surface at all
+
+## Main conclusion
+The current app is now split between better upstream telemetry and an older dashboard projection. The newest verified runner artifact proves real improvement in canonical telemetry, but the newest audited `lab-llm` surface still does not carry forward most of that truth.
+
+That means the next useful slices should still be chosen from the gap ledger, with emphasis on read-model/API adoption of canonical identity, runtime-truth mismatch semantics, activity state, KV/throughput measurement semantics, richer turn timing, log normalization, GPU live signals, honest freshness/source states, and a less ambiguous session list.

docs/dashboard_repo_architecture.md

@@ -8,6 +8,7 @@ Purpose:
 - keep repo-facing notes about the downstream dashboard architecture
 - document the current expected boundary from `model-runner`
 - avoid turning this repo into the dashboard implementation home
+- reinforce that this repo owns the operator-facing gap ledger
 
 This file is intentionally downstream-facing and non-authoritative for the dashboard repo.
 It exists so this repo can remember what the downstream architecture expects today.
@@ -129,6 +130,14 @@ The downstream UI should be:
 - explicit about live vs completed state
 - explicit about unknown/unavailable metrics
 
+## Gap ownership
+This repo owns the product-facing telemetry gap list.
+
+That means:
+- `lab-llm` records which operator questions are not answerable
+- `lab-llm` records which telemetry gaps cause those failures
+- `model-runner` remains accountable for upstream telemetry truth improvements
+
 ## Open notes
 - If the dashboard repo needs long-retention storage, that should be solved there, not here.
 - If the dashboard repo wants richer compare workflows or annotations, that should be solved there, not here.

docs/decisions/0003-dashboard-owns-gap-ledger.md

@@ -0,0 +1,33 @@
+# 0003 - The dashboard repo owns the telemetry gap ledger
+
+Date: 2026-03-15
+
+## Context
+This repo cannot become useful by polishing around weak or incomplete telemetry.
+
+The runner repo emits telemetry, but the dashboard repo is where usability failures become obvious:
+- duplicated or misleading labels
+- unknown values shown as if they were facts
+- noisy or untrustworthy metrics
+- missing model/load/runtime truth
+- views that fail to answer operator questions
+
+If gap ownership stays split or informal, the dashboard will drift into generic UI work while telemetry truth problems remain unresolved.
+
+## Decision
+`lab-llm` owns the telemetry gap ledger.
+
+That means:
+- this repo records the operator questions the product must answer
+- this repo records the telemetry gaps preventing those answers
+- this repo defines the required upstream contract improvements
+- `model-runner` is accountable to that ledger for telemetry truth and completeness
+
+The runner repo still owns telemetry implementation.
+This repo owns the product-facing definition of what is missing and why it matters.
+
+## Consequences
+- Dashboard work should be driven by question-answering gaps, not generic UI iteration.
+- Every important dashboard slice should tie back to one or more operator questions.
+- Telemetry issues found here should be written as upstream contract gaps rather than silently worked around.
+- Raw JSON/debug surfaces remain important, but they are not the acceptance surface for usefulness.

docs/decisions/README.md

@@ -8,6 +8,7 @@ Architectural decisions for `lab-llm`.
 |---|---|---|
 | 0001 | Consume runner telemetry through external interfaces | This repo consumes canonical telemetry via external app/metrics boundaries rather than importing runner internals. |
 | 0002 | Build an observability-first web product with room for chat later | The primary UX is runtime observability now, with a shared browser shell that can host chat later. |
+| 0003 | The dashboard repo owns the telemetry gap ledger | `lab-llm` defines operator-facing telemetry gaps and holds `model-runner` accountable for closing them. |
 
 ## Format