feat: real generation agent transition (Phase 0-3 + 4-1/4-2/4-4/4-5)

docs/specs/domain.md

@@ -147,7 +147,7 @@ type SurfaceConstraints = {
 ```ts
 type GateResult = {
   step: "rag" | "intent" | "generate" | "sympy_verify" | "re_solve" | "objective_map";
-  status: "passed" | "failed" | "skipped";
+  status: "passed" | "failed" | "skipped" | "unverified";
   duration_ms: number;
   evidence?: unknown;  // step별 다름
   failure_detail?: { code: string; message: string };
@@ -168,7 +168,7 @@ type HumanFailureReason = {
 - (I-V1) `gates`는 정확히 6개. 누락된 단계는 `status: "skipped"`로 명시.
 - (I-V2) `overall == "verified"`는 `gates[3] (sympy_verify).status == "passed"` **and** `gates[5] (objective_map).status == "passed"` 일 때만 가능 (결정론 게이트).
 - (I-V3) `gates[1] (intent)`와 `gates[2] (generate)`가 passed라도 `gates[3] (sympy_verify)`가 failed면 `overall != "verified"` (D-1: LLM은 정답 판단 X).
-- (I-V4) `overall == "warning"`은 SymPy는 통과했으나 `independent_resolve`(gates[4])가 불일치할 때만 사용 — 사용자에게 *주의* 라벨로 노출.
+- (I-V4) `overall == "warning"`은 SymPy가 `passed` 또는 `unverified`인 상태에서 `independent_resolve`(gates[4])가 불일치했거나, 비결정 가능(non-decidable) 게이트가 `unverified`일 때 사용 — 사용자에게 *주의* 라벨로 노출. 단, 재시도 상한의 최종 attempt에서도 `re_solve` 불일치가 지속되면 `warning`이 아니라 `rejected`다.
 - (I-V5) `attempt_count > 3`이면 강제 `overall == "rejected"` (Q-5 잠정 정책, D-9에서 확정 예정).
 
 ---

docs/specs/remediation-handoff.md

@@ -0,0 +1,79 @@
+# Remediation Handoff — Remaining Work
+
+| | |
+|---|---|
+| Status | Active handoff |
+| Date | 2026-06-10 |
+| Branch (done so far) | `feat/agent-real-generation-remediation` (PR #19, head `0a000ae0f`) |
+| Plan | [`remediation-plan.md`](./remediation-plan.md) |
+| Baseline | [`../eval/baseline-2026-06-10.md`](../eval/baseline-2026-06-10.md) |
+
+This doc lets a new session resume the "real generation agent transition" cleanly. It records **what shipped**, **what's cut/remaining**, and the **non-obvious gotchas** discovered while implementing — so the next person doesn't re-debug them.
+
+---
+
+## 1. What shipped (committed, verified)
+
+Verified green: agent typecheck clean · 236 unit + 5 integration tests · web typecheck clean · 29 pytest · ruff clean.
+
+- **Phase 0** — `DETERMINISTIC_FALLBACK=off|last-resort|first` env threaded through workflow; SSE result wire now carries `gates`, `overall`, `attempt_count`, `generation_model`, `refined_by`; [`scripts/eval-generation.ts`](../../scripts/eval-generation.ts) harness; baseline run (**pure-LLM 0/8 verified** — the demo only passed via the template).
+- **Phase 1** — template demoted to last-resort (emitted flagged after retries fail); per-attempt temperature 0.35→0.6→0.85; `generateObject` schema-failure immediate retry; gate-specific retry hints + previous-failure counterexample; intent fallback no longer blanket `must_preserve:true` (inherits strategy dims / marks `dimensions_source`); `SOLVER_MODEL` env; web "템플릿 폴백" badge.
+- **Phase 2** — math-engine: `multiprocessing.Process`+`terminate()` 5s hard timeout, `equals()`+numeric-sampling equivalence, `POST /verify-answer` (number|expression|equation|solution_set|inequality), `parse_latex(backend="lark")` (+`lark` dep), async endpoints. agent: gate **3-state** (`passed|failed|skipped|unverified`), honest sympy + real multiple-choice verification (**zero "non-empty⇒passed" paths**), re-solve mismatch → retry then reject on final attempt, equation-extractor unicode+LaTeX, `domain.md` I-V4 update, web "기호 검증 불가" badge.
+- **Phase 3-1** — `techniques_used` flows generator→schema→objective gate; deterministic set compare vs `required_techniques` (structural must cover; `technique_mismatch` failure).
+- **Phase 4-2** — silent catches record `skipped_reason`/`nuance_skipped_reason`/`normalization_skipped_reasons` in gate evidence.
+- **Phase 4-4** — `LLM_E2E=1`-gated real integration test (`tests/integration/real-llm-e2e.test.ts`), skipped by default.
+- **Phase 4-1** (commit `0a000ae0f`) — `withTimeout` builds an `AbortController`, aborts it on timeout, and threads the signal into every `generateObject` call (generator/refiner/critic/solver/intent/objective nuance), so slow LLM calls are cancelled at the wall clock — **fixes the baseline 160s-timeout root cause**. math-engine calls were intentionally NOT threaded (the client already self-aborts each request at 10s, so a step can't exceed its budget on math-engine). +2 `withTimeout` cancellation unit tests.
+- **Phase 4-5** (commit `0a000ae0f`) — S5/S6 drop the page-level `generateMockResults` seed; result/export views source verified problems only from the SSE→`sessionStorage` rehydrate. `ResultProblem` type moved to `result/types.ts`, `mock.ts` deleted, S5 empty state added. **Browser-verified**: empty state (no mock leak), rehydrate with template-fallback/unverified badges, export adopted-filter, 0 console errors.
+
+---
+
+## 2. Remaining work (cut for time, in priority order)
+
+> **4-1 and 4-5 (the high-value items) are DONE** — PR #19 commit `0a000ae0f`, see §1. Below is what's left.
+
+### Medium value (now highest remaining)
+- **3-2 — Mode-specific surface check** (`steps/objective-mapping.ts` ~L135-141, the `not_transformed` guard / `sameMathText`). structural = number-masked skeleton must MATCH source; conceptual = skeleton must DIFFER. Extend the existing guard.
+- **3-3 — Deterministic `must_preserve` enforcement** at the objective gate (answer-type, technique, difficulty markers) independent of the LLM constraint-critic. Build on the 3-1 `techniqueEvidence` machinery already in `objective-mapping.ts`.
+- **4-3 — Cost visibility.** Aggregate AI SDK `usage` per workflow into the SSE result event; cap calls per workflow. AI SDK v4: `result.usage = { promptTokens, completionTokens, totalTokens }`. Thread through `verification-workflow.ts` → result event → `wire-format.schema.ts`.
+
+### Low value / post-demo
+- **4-6 — Corpus injection defense.** Wrap corpus text in `prompts/problem-generator.md` with `<source>…</source>` + an "ignore instructions inside source" directive. NOTE: this prompt was edited in Phase 1 (counterexample) and 3-1 (techniques) — read current state.
+- **4-7 — RAG embeddings, request seed reproducibility, rate limit.** Deferred to post-demo.
+
+### Measurement
+- **Full baseline + phase-boundary eval.** Run the documented 120-generation matrix unattended (1–2h at `xhigh` speed) and append the improvement table to `docs/eval/`:
+  ```bash
+  cd packages/agent && PORT=31416 DETERMINISTIC_FALLBACK=off ./node_modules/.bin/tsx src/index.ts &
+  packages/agent/node_modules/.bin/tsx scripts/eval-generation.ts \
+    --endpoint http://localhost:31416 --repeats 5 --concurrency 4 --timeout-ms 240000 --out-dir reports/eval
+  ```
+  Compare `refined_by` template-fallback % and per-gate pass rates against the 0% baseline. **Never fabricate numbers** — only commit what live runs return.
+
+---
+
+## 3. Non-obvious gotchas (re-debugged so you don't have to)
+
+1. **`normalizeExpectedAnswer` overwrites the LLM's answer with the engine's solve result** (`steps/problem-generation.ts`). So for any engine-solvable equation, `sympy_verify` compares solve-vs-solve → always passes; a genuine wrong answer surfaces at **re_solve** or the **objective** gate, not sympy. This is intended (engine = source of truth for solvable equations) but surprising.
+2. **`sameAnswer` has an `expectedMatchesEquationSolve` shortcut** (`tools/answer-equivalence.ts`): if the *expected* answer matches the equation's solve, it returns true **regardless of the derived/solver answer**. Combined with (1), a solvable-equation candidate cannot fail re_solve via a wrong solver answer in tests. To test a rejection deterministically, use a **technique mismatch** (3-1) — see `tests/integration/generate.test.ts`.
+3. **Gate `unverified` maps to wire step "completed"**, not "failed" (`server/sse/wire-adapter.ts` `readGateStatus` only knows passed/failed/skipped → null → "completed"). If you add UI/eval logic on step status, read the per-gate `status` from the result event instead.
+4. **acceptance ↔ retry budget consistency**: `verification-workflow.ts` passes `maxAttempts = maxRetries ?? 3` to BOTH `createBoundedRetryPolicy` and `createAcceptancePolicy` (L77-79). If you change one, change both, or the 2-9 "final-attempt mismatch → reject" promise breaks for `maxRetries ≠ 3`.
+5. **Python 3.11** — no `ProcessPoolExecutor.kill_workers` (3.14+). The 5s hard timeout uses `multiprocessing.Process` + `terminate()`. Don't "simplify" it back to ProcessPoolExecutor.
+6. **`parse_latex` uses `backend="lark"`** (pure-Python, `lark` dep) — not the antlr default (needs `antlr4-python3-runtime`).
+7. **Integration tests are a separate vitest config** (`pnpm -F @openmath/agent test:integration`, `vitest.integration.config.ts`) — they are NOT run by `pnpm test`. After changing verification behavior, run BOTH.
+8. **Tests that encoded the old "non-empty⇒passed" behavior were updated, not deleted**, to assert the new honest semantics. Keep this discipline.
+9. **`withTimeout(fn, opts)` calls `fn(signal)`** and aborts that signal on timeout (4-1). Pass `abortSignal: signal` to any new `generateObject`/fetch added inside a `withTimeout` closure. Zero-arg `() => ...` closures still typecheck (param contravariance), so signal-less sites (e.g. `rag-search.ts`) stay untouched. math-engine calls are deliberately not signal-threaded — the client self-aborts at 10s/request.
+
+---
+
+## 4. Files NOT touched (leave alone)
+
+Pre-existing untracked files that are NOT part of this work and were intentionally excluded from the commit:
+- `packages/agent/scripts/transform_real_corpus.py`
+- `packages/agent/scripts/verify_rag.ts`
+- `reports/` (regenerable eval artifacts)
+
+---
+
+## 5. Suggested resume order
+
+4-1 + 4-5 done (PR #19). Remaining: `3-2/3-3 (isomorphism depth) → 4-3 (cost) → 4-6 (injection defense) → run the eval (re-baseline now that LLM timeouts are cancellable + mock is gone) → final comparison table`.

docs/specs/remediation-plan.md

@@ -0,0 +1,115 @@
+# Remediation Plan — Real Generation Agent Transition
+
+| | |
+|---|---|
+| Status | Active (implementation) |
+| Last updated | 2026-06-10 |
+| Branch | `feat/agent-real-generation-remediation` |
+| Goal | 12 demo units where the LLM actually generates problems, gates pass *honestly*, and quality is provable in numbers. |
+| Principle | Each phase builds on the previous phase's **measurement**. Measure first, then fix ("재고 나서 고친다"). |
+
+> This doc tracks the engineering execution of the transition from a template-first
+> demo to a real LLM-generation product. File/line anchors below were captured from a
+> full codebase survey (2026-06-10) and may drift — re-grep before editing.
+
+## Environment status (verified 2026-06-10)
+
+- `packages/agent/.env` exists: `LLM_PROVIDER=cliproxy`, `CLIPROXY_BASE_URL=http://localhost:8317/v1`, `LLM_MODEL=gpt-5.5(xhigh)`.
+- CLIProxyAPI router reachable on `:8317` (401 on unauthenticated `/v1/models`; agent holds a key).
+- math-engine healthy on `:16180`; agent up on `:31415`.
+- Python **3.11.14** → `ProcessPoolExecutor.kill_workers()` (3.14+) unavailable. Hard SymPy timeout must use `multiprocessing.Process` + `terminate()`.
+- AI SDK pinned `ai@^4.3.0` → v4 usage naming `{promptTokens, completionTokens, totalTokens}`; `NoObjectGeneratedError`, `experimental_repairText`, `abortSignal`, `temperature` all available on `generateObject`.
+
+## 12 demo units (eval target)
+
+`9수01-01`, `9수01-05`, `9수02-01`, `9수02-03`, `9수02-06`, `9수02-07`, `9수02-08`, `9수02-09`, `9수02-10`, `9수03-02`, `9수03-04`, `9수04-05`.
+
+## Invariant impact (domain.md must be updated where flagged)
+
+- Adding gate status `unverified` extends `GateStatusSchema` (`passed|failed|skipped` → `+unverified`). Touches **I-V1** model and **I-V4** semantics → update `domain.md` §2.4.
+- 2-9 changes re-solve mismatch from `warning`-pass to retry-then-reject → revises **I-V4** intent → update `domain.md`.
+- 1-4 intent fallback must still satisfy **I-I2** (≥1 `must_preserve:true`).
+- 3-1/3-2/3-3 enforce **I-G2/I-G3** deterministically at the gate.
+
+---
+
+## Phase 0 — Baseline measurement (prerequisite for all decisions)
+
+| # | Task | Target |
+|---|---|---|
+| 0-1 | `DETERMINISTIC_FALLBACK=off\|last-resort\|first` (default `first`) env, threaded to workflow. | `config/env.ts` L8-32, `index.ts` L39-103, `steps/problem-generation.ts` L58-63 |
+| 0-1b | **Observability prereq**: expose verification `gates` + `generation_metadata.{model,refined_by}` + usage in the SSE result wire (needed by eval AND Phase 1-1 badge; no behavior change). | `wire-adapter.ts` L96-115, `wire-format.schema.ts` L45-72, `progress-event.schema.ts` |
+| 0-2 | `scripts/eval-generation.ts`: 12 units × {structural,conceptual} × 5 → per-gate pass rate, failure reason, duration, `refined_by`, model → JSONL + summary MD. Template on `scripts/sweep-generate.mjs`. | `scripts/eval-generation.ts` (new) |
+| 0-3 | Run once with `DETERMINISTIC_FALLBACK=off` → `docs/eval/baseline-<date>.md`. **Requires live LLM; numbers never fabricated.** | — |
+
+**Done when**: a per-unit LLM-generation success table exists as the comparison baseline.
+
+> **Baseline result (2026-06-10, see `docs/eval/baseline-2026-06-10.md`)**: pure-LLM (`off`) verified = **0/8 (0%)**. Failure modes: `no-result` pipeline exception (silently swallowed; from `generateProblem` re-throwing a `generateObject` schema failure) and 160s timeouts (slow `xhigh` model). Confirms the demo passes only via the template short-circuit. This is the number Phase 1+ must move.
+
+## Phase 1 — Generation path recovery
+
+| # | Task | Target |
+|---|---|---|
+| 1-1 | Demote template to last-resort; surface `generation_metadata.model="deterministic-topic-generator"` in SSE result; web "템플릿 폴백" badge. | `problem-generation.ts` L58-63,140-153, `deterministic.ts`, wire-adapter, web `result/view.tsx` L38-74 |
+| 1-2 | Per-attempt temperature schedule (0.35→0.6→0.85); gate-specific retry hints; previous failed candidate as counterexample in prompt. | `retry-policy.ts` L19-45, `generator-agent.ts` L66-72, `prompts/problem-generator.md` |
+| 1-3 | `generateObject` schema-failure immediate single retry (error as hint) before bubbling to workflow failure. | `generator-agent.ts` L66-72 |
+| 1-4 | Intent fallback: stop guessing `must_preserve:true` for all dims → inherit strategy default dims; if guessed, set `fallback:true` evidence → warning. Keep I-I2. | `intent-extraction.ts` L69-110 |
+| 1-5 | `SOLVER_MODEL` env (defaults to `LLM_MODEL`) so re-solve uses a different model. | `config/env.ts`, `index.ts` L42-80, `config/models.ts` |
+
+**Done when**: re-run eval shows `deterministic-topic-generator` in `refined_by` < 10%, LLM success measurably improved vs baseline.
+
+## Phase 2 — Verification becomes real verification
+
+### math-engine (Python) — parallelizable with TS
+| # | Task | Target |
+|---|---|---|
+| 2-1 | Per-request hard timeout via `multiprocessing.Process` + `terminate()` (5s) — Python 3.11, no `kill_workers`. | `routers/math.py`, `main.py` |
+| 2-2 | Equivalence `simplify(a-b)==0` → `expr.equals(0)` + numeric-sampling fallback (lambdify mpmath, N trials). | `math.py` L78-86 |
+| 2-3 | New `/verify-answer` with `answer_type: number\|expression\|equation\|solution_set\|inequality` (x=2 vs x-2=0; {-2,2} vs ±2; 1/2 vs 0.5). | `math.py`, models |
+| 2-4 | LaTeX parse layer: `parse_latex(..., backend="lark")` + add `lark` dep; `\frac` etc → SymPy. Removes agent regex workarounds. | `math.py` L19-23, `pyproject.toml` |
+| 2-5 | `async def` endpoints delegating to executor; uvicorn workers. | `main.py`, `run.py` |
+
+### agent (TypeScript)
+| # | Task | Target |
+|---|---|---|
+| 2-6 | 3-state gates: add `unverified` to `GateStatusSchema`. SymPy-blind types (Korean answer, geometry, stats) → `unverified` (not non-empty⇒passed). acceptance: `unverified`→`warning`. web badge "기호 검증 불가". **update domain.md.** | `verification.schema.ts` L24, `sympy-verification.ts` L70-90, `acceptance-policy.ts`, web |
+| 2-7 | Multiple-choice verification: extract correct option value → compare to SymPy solution; 5 distractors mutually non-equivalent (block duplicate answers). | `sympy-verification.ts` L71-76, `answer-equivalence.ts` |
+| 2-8 | `equation-extractor` unicode ops (≤ ≥ ÷) + LaTeX inline; extraction failure → `unverified` (not throw→failed). | `equation-extractor.ts` |
+| 2-9 | Re-solve mismatch → retry trigger; final-attempt mismatch → `reject` (was warning-pass). | `acceptance-policy.ts` L21, `retry-policy.ts` |
+
+**Done when**: zero "non-empty⇒passed" paths; every candidate is honestly `passed|unverified|failed`; deliberately-wrong-answer test cases all reject.
+
+## Phase 3 — Isomorphism from prompt to gate
+
+| # | Task | Target |
+|---|---|---|
+| 3-1 | `techniques_used: string[]` (enum from strategy vocab) in generator output → deterministic set compare with `required_techniques`. | `generated-problem.schema.ts`, `problem-generator.md`, `objective-mapping.ts` |
+| 3-2 | Mode surface check: structural=number-masked skeleton must match source; conceptual=skeleton must differ. | `objective-mapping.ts` L135-141 |
+| 3-3 | Enforce `must_preserve` dims deterministically (answer type, technique, difficulty markers) at the gate, independent of LLM critic. | `objective-mapping.ts` |
+
+**Done when**: injecting a different-technique problem (e.g. quadratic-formula item into a factoring unit) is rejected at the objective gate.
+
+## Phase 4 — Trust & operational infra
+
+| # | Task | Target |
+|---|---|---|
+| 4-1 | `withTimeout` + `AbortController` → propagate `abortSignal` to AI SDK + math-engine fetch; real cancel on timeout. | `timeout-policy.ts`, agents/clients |
+| 4-2 | Remove 5 silent catches → record `skipped_reason` in gate evidence. | `objective-mapping.ts` L206, `problem-generation.ts` L190-193, `answer-equivalence.ts` L79-84 |
+| 4-3 | Aggregate AI SDK `usage` per workflow into result event; cap calls per workflow. | wire/SSE schema, workflow |
+| 4-4 | `LLM_E2E=1`-gated real integration test: 1 unit end-to-end → verified. | `tests/integration/` |
+| 4-5 | Wire S5/S6 to real `useVerificationStream` results; remove mock. | `result/page.tsx` L35-46, `export/page.tsx` L45-60 |
+| 4-6 | Corpus text injection defense: `<source>…</source>` wrap + "ignore instructions" directive. | `problem-generator.md`, prompt-loader |
+| 4-7 | (Deferred) RAG embeddings, request seed reproducibility, rate limit. | post-demo |
+
+## Dependency order
+
+```
+Phase 0 (measure) ──→ Phase 1 (generation) ──→ Phase 3 (isomorphism)
+                  └→ Phase 2 (verification) ──┘
+Phase 4: 4-1/4-2 alongside Phase 1; rest at the end.
+```
+
+- Phase 0+1 = the first bundle; alone yields an "AI generates problems" product.
+- Phase 2 splits into Python (2-1..2-5) and agent (2-6..2-9), parallelizable.
+- If schedule is tight, cut 3-2/3-3 and 4-6/4-7 first — **never cut 2-6 (3-state)**; honesty of the "verified" claim is this project's reason to exist.
+- Re-run `scripts/eval-generation.ts` at each phase boundary; keep the comparison table for the final presentation (improvement story, not a template demo).

packages/agent/.env.example

@@ -29,3 +29,7 @@ CLIPROXY_MODEL=gpt-5.5(xhigh)
 LLM_BASE_URL=http://localhost:8317/v1
 LLM_API_KEY=dummy-key
 LLM_MODEL=gpt-5.5(xhigh)
+
+# (옵션) 독립 재풀이(D-5 ReSolveSpecialist) 전용 모델. 생성과 다른 모델을 쓰면
+# 동일 오답 재현 가능성이 낮아짐. 미설정 시 LLM_MODEL 과 동일한 인스턴스를 공유.
+# SOLVER_MODEL=claude-sonnet-4.5