From 4a26691b8c75a2a781aaef9d92d6247e127d7936 Mon Sep 17 00:00:00 2001 From: LeeJhin Date: Thu, 11 Jun 2026 12:54:53 +0900 Subject: [PATCH 1/2] feat: agent pipeline narration, run traces, verification fixes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit UX — the 6-step bar now tells what actually happened instead of bare 통과/실패: - Step summaries built from gate evidence (refs count, objective code, critic rounds/hints applied, SymPy solutions, re-solve answer + confidence, durations) - Emit the long-dead `preview` SSE event after generate (FE stage already existed) - Honest 3-state: `unverified` no longer masquerades as 통과 (△ badge), intent LLM fallback is labeled, new `attempt` event resets steps 3-6 with reason, new `runs` event surfaces parallel generation progress (n/m) DX — zero-logging agent now leaves a trail: - Per-run JSONL traces (request + every ProgressEvent incl. full gate evidence and critic/guard hints) under TRACE_DIR (default ./runs), plus llm.jsonl with per-call latency/tokens via a LanguageModel middleware; stdout narration per run Structure — fixes for flaws found in the D-5 review: - Split SOLVER_MODEL (gpt-5.4) so independent re-solve no longer shares the generator checkpoint (was sharing one gpt-5.5 instance, defeating D-5) - Generation failure degrades to a failed gate + skipped 4-6 and goes through the retry policy instead of killing the whole run (dead catch repaired) - RefinerAgent is now a real agent using prompts/refiner.md (repair-in-place, bounded change scope) instead of a disguised generator re-call - Critic critiques are preserved in gate evidence (critic_rounds, hints) - sympy_verify and re_solve run concurrently; driver errors are buffered so sibling runs can still deliver a result Wire contract (FE+BE updated together): step.status gains `unverified`, step.summary on success, new attempt/runs events; docs updated (D-6, web README). Tests: 247 passed (3 new test files), agent+web typecheck clean, live smoke verified against the running dev stack (preview/summaries/runs/traces observed). Co-Authored-By: Claude Fable 5 --- .gitignore | 3 + docs/specs/architecture.md | 2 +- packages/agent/.env.example | 15 +- packages/agent/prompts/refiner.md | 56 +++-- packages/agent/src/agents/generator-agent.ts | 73 +++++-- packages/agent/src/agents/refiner-agent.ts | 48 +++-- packages/agent/src/config/env.ts | 10 +- packages/agent/src/index.ts | 54 +++-- .../src/schemas/progress-event.schema.ts | 11 + .../agent/src/schemas/wire-format.schema.ts | 21 +- packages/agent/src/server/app.ts | 4 +- packages/agent/src/server/routes/generate.ts | 9 +- .../agent/src/server/sse/progress-stream.ts | 159 ++++++++++++-- packages/agent/src/server/sse/wire-adapter.ts | 153 ++++++++++++-- .../agent/src/steps/independent-resolve.ts | 5 +- packages/agent/src/steps/intent-extraction.ts | 6 +- .../agent/src/steps/problem-generation.ts | 32 ++- packages/agent/src/tools/index.ts | 2 + packages/agent/src/tools/llm-logging.ts | 66 ++++++ packages/agent/src/tools/run-trace.ts | 51 +++++ .../src/workflows/verification-workflow.ts | 119 +++++++---- packages/agent/tests/backend-contract.test.ts | 77 ++++++- .../tests/deterministic-fallback.test.ts | 12 +- .../tests/integration/real-llm-e2e.test.ts | 2 +- .../agent/tests/problem-generation.test.ts | 104 ++++++++- packages/agent/tests/refiner-agent.test.ts | 162 ++++++++++++++ ...fication-workflow-generate-failure.test.ts | 200 ++++++++++++++++++ packages/web/README.md | 8 +- packages/web/app/app/new/verify/view.tsx | 57 ++++- packages/web/app/globals.css | 10 + packages/web/hooks/use-verification-stream.ts | 87 +++++++- 31 files changed, 1438 insertions(+), 180 deletions(-) create mode 100644 packages/agent/src/tools/llm-logging.ts create mode 100644 packages/agent/src/tools/run-trace.ts create mode 100644 packages/agent/tests/refiner-agent.test.ts create mode 100644 packages/agent/tests/verification-workflow-generate-failure.test.ts diff --git a/.gitignore b/.gitignore index c6dc98a..8c6f1ee 100644 --- a/.gitignore +++ b/.gitignore @@ -44,3 +44,6 @@ packages/agent/data/corpus/*.summary.json # Local browser QA artifacts .playwright-mcp/ + +# Agent run observability traces (TRACE_DIR) +packages/agent/runs/ diff --git a/docs/specs/architecture.md b/docs/specs/architecture.md index 910c419..c80db70 100644 --- a/docs/specs/architecture.md +++ b/docs/specs/architecture.md @@ -159,7 +159,7 @@ - **Closes**: Q-6 ### D-6. 클라이언트 ↔ agent 프로토콜은 SSE -- **결정**: Hono `streamSSE` + 클라이언트 `EventSource`. `POST /api/generate`가 `text/event-stream`으로 응답. event 종류: `step` (단계 시작/완료), `result` (최종 문제 묶음), `error` (스테이지·메시지). +- **결정**: Hono `streamSSE` + 클라이언트 `EventSource`. `POST /api/generate`가 `text/event-stream`으로 응답. event 종류: `step` (단계 시작/완료/판정불가 + 단계 서사 summary), `preview` (3단계 직후 후보 미리보기), `attempt` (재생성 시작 — attempt/max_attempts/reason), `runs` (병렬 런 집계), `result` (최종 문제 묶음), `error` (스테이지·메시지). 상세 wire 계약: `packages/web/README.md` §SSE Consumption. - **대안**: (a) sync REST + 폴링 (step bar 라이브 노출 안됨), (b) job 큐 + 폴링/웹훅 (다중 사용자 동시성 1차 MVP scope 초과), (c) WebSocket (양방향 불필요). - **채택 사유**: D-4 `streamText` chunk + D-5 async generator emit이 SSE와 1:1 매칭. 단방향 progress 스트리밍 표준. 프론트 `EventSource` 빌트인. - **Closes**: Q-3 diff --git a/packages/agent/.env.example b/packages/agent/.env.example index e5c9136..5284072 100644 --- a/packages/agent/.env.example +++ b/packages/agent/.env.example @@ -30,6 +30,15 @@ LLM_BASE_URL=http://localhost:8317/v1 LLM_API_KEY=dummy-key LLM_MODEL=gpt-5.5(xhigh) -# (옵션) 독립 재풀이(D-5 ReSolveSpecialist) 전용 모델. 생성과 다른 모델을 쓰면 -# 동일 오답 재현 가능성이 낮아짐. 미설정 시 LLM_MODEL 과 동일한 인스턴스를 공유. -# SOLVER_MODEL=claude-sonnet-4.5 +# (권장) 독립 재풀이(D-5 ReSolveSpecialist) 전용 모델. 생성과 다른 모델을 쓰면 +# 동일 오답 재현 가능성이 낮아짐. 미설정 시 LLM_MODEL 과 동일한 인스턴스를 공유 — +# 이러면 "독립" 재풀이의 교차검증 효과가 사라지므로 반드시 분리할 것. +# 현재 cliproxy 가 노출하는 모델 중에서는 gpt-5.4 가 생성기(gpt-5.5)와 다른 체크포인트. +# Claude/Gemini 계열이 프록시에 생기면 다른 벤더 모델로 교체 권장. +SOLVER_MODEL=gpt-5.4 + +# --- 관측성 (DX) --- +# run 단위 JSONL 트레이스: ProgressEvent 전문(게이트 evidence 포함) + LLM 호출 지연/토큰. +# tail -f runs/run-*.jsonl | jq 로 에이전트 내부를 실시간 관찰 가능. +TRACE_ENABLED=true +TRACE_DIR=./runs diff --git a/packages/agent/prompts/refiner.md b/packages/agent/prompts/refiner.md index a4877fa..204d618 100644 --- a/packages/agent/prompts/refiner.md +++ b/packages/agent/prompts/refiner.md @@ -1,41 +1,71 @@ --- id: refiner -version: 0.1.0 +version: 0.2.0 model: gpt-4o temperature: 0.5 max_tokens: 2000 schema: GeneratedProblemSchema variables: - prior + - request + - generationKind - intent + - refs + - strategy - hints + - schemaError owner: 비할당 -updated: 2026-05-18 +updated: 2026-06-11 --- # Role -이전 생성된 문제를 *비평 힌트*에 따라 다시 작성하라. Intent는 유지하되 표면 표현만 개선. +당신은 한국 중학교 수학 문제의 *수선공*이다. 이전에 생성된 문제(Prior)를 아래 비평 힌트에 따라 고쳐 쓰라. + +- 비평이 지적한 부분만 고치고, 문제의 골격·소재·풀이 경로는 Prior에 최대한 가깝게 유지한다. +- 완전히 다른 문제를 새로 만들지 않는다. Prior와 동떨어진 문제는 실패로 간주된다. +- 학습 목표와 평가 차원(Intent)은 절대 바꾸지 않는다. # Prior -{{prior.question_text}} +문제: {{prior.question_text}} 정답: {{prior.expected_answer}} +풀이: {{prior.proposed_solution_trace}} + +# Critique hints (이 지적들을 반드시 해소하라) + +{{#each hints}}- {{this}} +{{/each}} # Intent (불변) -{{intent.objective_description}} -보존해야 할 평가 차원: {{intent.evaluation_dimensions}} +학습 목표: {{intent.objective_description}} (`{{intent.objective_code}}`) +보존해야 할 평가 차원: {{#each intent.evaluation_dimensions}}{{#if this.must_preserve}}{{this.description}}; {{/if}}{{/each}} +필수 기법: {{intent.required_techniques}} +금지 기법: {{intent.forbidden_techniques}} -# Critique hints +# Generation Kind -{{#each hints}}- {{this}} -{{/each}} +`{{generationKind}}` 유형을 유지하라. `{{request.difficulty}}` 난이도, `{{request.problem_type}}` 형식. -# Output +# Strategy (선택) + +{{strategy}} + +{{#if schemaError}} +# Schema repair hint (즉시 재시도) -다시 작성된 `GeneratedProblem` JSON. `inferred_intent.objective_code`와 `mode`는 prior 그대로 유지. +직전 응답이 JSON schema 검증에 실패했다. 오류: {{schemaError}} + +- 반드시 JSON object 하나만 출력하라. +- `question_text`, `expected_answer`, `techniques_used`, `proposed_solution_trace` 네 필드를 모두 채워라. +{{/if}} + +# Output -# TODO +JSON으로만 응답. 필드: `question_text`, `expected_answer`, `techniques_used`, `proposed_solution_trace`. -- prior와의 *변경 범위* 제약 (너무 다른 문제로 변하지 않게) +- JSON 문자열 안에서 raw backslash를 쓰지 말 것. `\(`, `\sqrt` 같은 LaTeX 명령 금지. +- 수식은 JSON 안전한 plain text로: 지수는 `x**2`, 곱셈은 `5*x`, 제곱근은 `sqrt(7)`. +- `expected_answer`는 정답만 간결하게. 수정 후에도 문제와 정답이 일치하는지 반드시 재검산하라. +- `techniques_used`는 실제 사용한 snake_case 기법 id 배열. diff --git a/packages/agent/src/agents/generator-agent.ts b/packages/agent/src/agents/generator-agent.ts index 758fe1e..2e7e27f 100644 --- a/packages/agent/src/agents/generator-agent.ts +++ b/packages/agent/src/agents/generator-agent.ts @@ -37,7 +37,7 @@ export interface GeneratorAgentDeps { prompts: PromptLoader; } -const LlmGeneratedCandidateSchema = z.object({ +export const LlmGeneratedCandidateSchema = z.object({ question_text: z .string() .min(1) @@ -56,7 +56,41 @@ const LlmGeneratedCandidateSchema = z.object({ .describe("Korean solution trace explaining the structural/conceptual transform"), }); -type LlmGeneratedCandidate = z.infer; +export type LlmGeneratedCandidate = z.infer; + +/** LLM raw output + 호출 컨텍스트 → 도메인 GeneratedProblem. Generator와 Refiner가 공유. */ +export function assembleGeneratedProblem(input: { + readonly request: GenerateRequest; + readonly intent: Intent; + readonly refs: RagResult[]; + readonly attempt: number; + readonly object: LlmGeneratedCandidate; + readonly modelId: string; + readonly temperature: number; + readonly promptId: string; + readonly promptVersion: string; +}): GeneratedProblem { + const generationKind = generationKindForTopic(getGenerateRequestTopicCode(input.request)); + return { + candidate_id: randomUUID(), + mode: input.request.mode === "conceptual" ? "conceptual" : "structural", + generation_kind: generationKind, + question_text: input.object.question_text, + expected_answer: input.object.expected_answer, + techniques_used: input.object.techniques_used ?? [], + proposed_solution_trace: input.object.proposed_solution_trace, + source_refs: input.refs.map((ref) => ref.item_id), + inferred_intent: input.intent, + generation_metadata: { + model: input.modelId, + temperature: input.temperature, + prompt_id: input.promptId, + prompt_version: input.promptVersion, + attempt: input.attempt, + generated_at: new Date().toISOString(), + }, + }; +} export function temperatureForGeneratorAttempt( attempt: number, @@ -95,30 +129,22 @@ export function createGeneratorAgent(deps: GeneratorAgentDeps): GeneratorAgent { }, }); - return { - candidate_id: randomUUID(), - mode: input.request.mode === "conceptual" ? "conceptual" : "structural", - generation_kind: generationKind, - question_text: object.question_text, - expected_answer: object.expected_answer, - techniques_used: object.techniques_used ?? [], - proposed_solution_trace: object.proposed_solution_trace, - source_refs: input.refs.map((ref) => ref.item_id), - inferred_intent: input.intent, - generation_metadata: { - model: deps.modelId, - temperature, - prompt_id: prompt.metadata.id, - prompt_version: prompt.metadata.version, - attempt: input.attempt, - generated_at: new Date().toISOString(), - }, - }; + return assembleGeneratedProblem({ + request: input.request, + intent: input.intent, + refs: input.refs, + attempt: input.attempt, + object, + modelId: deps.modelId, + temperature, + promptId: prompt.metadata.id, + promptVersion: prompt.metadata.version, + }); }, }; } -interface GenerateCandidateObjectInput { +export interface GenerateCandidateObjectInput { model: LanguageModel; prompt: string; temperature: number; @@ -126,7 +152,8 @@ interface GenerateCandidateObjectInput { retryPromptForSchemaError(schemaError: string): string; } -async function generateCandidateObject( +/** generateObject + schema-repair 1회 재시도. Generator와 Refiner가 공유. */ +export async function generateCandidateObject( input: GenerateCandidateObjectInput, ): Promise { try { diff --git a/packages/agent/src/agents/refiner-agent.ts b/packages/agent/src/agents/refiner-agent.ts index 11e9280..f10402d 100644 --- a/packages/agent/src/agents/refiner-agent.ts +++ b/packages/agent/src/agents/refiner-agent.ts @@ -1,9 +1,15 @@ -/** RefinerAgent — D-5 GenerationSpecialist team. Re-runs GeneratorAgent with critique hints. */ +/** RefinerAgent — D-5 GenerationSpecialist team. 비평 힌트를 받아 prior 후보를 *수선*한다. + * Generator의 백지 재생성과 달리 자기 프롬프트(refiner.md)로 변경 범위를 prior 근처로 제한. */ import type { LanguageModel } from "ai"; import type { GenerateRequest, GeneratedProblem, Intent, RagResult, Strategy } from "../schemas/index.js"; -import type { GeneratorAgent } from "./generator-agent.js"; +import { generationKindForTopic, getGenerateRequestTopicCode } from "../schemas/index.js"; +import type { PromptLoader } from "../tools/prompt-loader.js"; +import { + assembleGeneratedProblem, + generateCandidateObject, +} from "./generator-agent.js"; export interface RefineInput { prior: GeneratedProblem; @@ -24,27 +30,43 @@ export interface RefinerAgentDeps { model: LanguageModel; modelId: string; promptId: string; - generator: GeneratorAgent; + prompts: PromptLoader; } export function createRefinerAgent(deps: RefinerAgentDeps): RefinerAgent { return { async refine(input) { - const refinementHint = [ - "Prior candidate:", - input.prior.question_text, - "Critique hints:", - ...input.hints.map((hint) => `- ${hint}`), - ].join("\n"); + const prompt = await deps.prompts.load(deps.promptId); + const generationKind = generationKindForTopic(getGenerateRequestTopicCode(input.request)); + const basePromptVars = { + prior: input.prior, + request: input.request, + generationKind, + intent: input.intent, + refs: input.refs, + strategy: input.strategy === null ? "" : JSON.stringify(input.strategy, null, 2), + hints: input.hints, + }; + const object = await generateCandidateObject({ + model: deps.model, + prompt: prompt.render(basePromptVars), + temperature: prompt.metadata.temperature, + signal: input.signal, + retryPromptForSchemaError(schemaError) { + return prompt.render({ ...basePromptVars, schemaError }); + }, + }); - return deps.generator.generate({ + return assembleGeneratedProblem({ request: input.request, intent: input.intent, refs: input.refs, - strategy: input.strategy, attempt: input.attempt, - refinementHint, - signal: input.signal, + object, + modelId: deps.modelId, + temperature: prompt.metadata.temperature, + promptId: prompt.metadata.id, + promptVersion: prompt.metadata.version, }); }, }; diff --git a/packages/agent/src/config/env.ts b/packages/agent/src/config/env.ts index 2bfb964..41272aa 100644 --- a/packages/agent/src/config/env.ts +++ b/packages/agent/src/config/env.ts @@ -38,12 +38,18 @@ export const EnvSchema = z.object({ * workflow. Below ~45s, last-resort/off generation times out and fails. */ PER_STEP_TIMEOUT_MS: z.coerce.number().int().min(1000).default(60000), - /** `first` = template short-circuits LLM when refs exist (current behavior). + /** `first` = template short-circuits LLM when refs exist. * `off` = always go through LLM generator path; never substitute template. - * `last-resort` = placeholder, currently behaves like `first` (see TODO 1-1a). */ + * `last-resort` = LLM first; substitute the deterministic template only when the + * final verification verdict is `rejected` (transparent fallback, D-11). */ DETERMINISTIC_FALLBACK: z .enum(["off", "last-resort", "first"]) .default("first"), + + /** Per-run JSONL observability traces (ProgressEvent 전문 + LLM call 지연/토큰). + * Files land in TRACE_DIR: run--.jsonl, llm.jsonl. */ + TRACE_ENABLED: z.enum(["true", "false"]).default("true"), + TRACE_DIR: z.string().default("./runs"), }); export type Env = z.infer; diff --git a/packages/agent/src/index.ts b/packages/agent/src/index.ts index f8d6411..130f61d 100644 --- a/packages/agent/src/index.ts +++ b/packages/agent/src/index.ts @@ -14,14 +14,21 @@ import { } from "./agents/index.js"; import { createFsPromptLoader, + createFsRunTraceWriter, createFsStrategyLoader, createInMemoryRagClient, createMathEngineClient, resolveLanguageModel, + withLlmLogging, + type LlmCallRecord, } from "./tools/index.js"; export async function main(): Promise { const env = loadEnv(); + const trace = createFsRunTraceWriter({ + dir: resolve(env.TRACE_DIR), + enabled: env.TRACE_ENABLED === "true", + }); const mathEngine = createMathEngineClient({ baseUrl: env.MATH_ENGINE_URL, timeoutMs: env.PER_STEP_TIMEOUT_MS, @@ -40,27 +47,38 @@ export async function main(): Promise { const llmBaseUrl = env.LLM_BASE_URL ?? env.CLIPROXY_BASE_URL; const llmApiKey = env.LLM_API_KEY ?? env.CLIPROXY_API_KEY ?? env.OPENAI_API_KEY; const llmModel = env.LLM_MODEL ?? env.CLIPROXY_MODEL ?? env.OPENAI_MODEL ?? DEFAULT_MODELS.generator; + const logLlmCall = (record: LlmCallRecord): void => { + void trace.append("llm", record); + }; const llm = llmBaseUrl !== undefined || llmApiKey !== undefined - ? resolveLanguageModel({ - kind: llmKind, - modelId: llmModel, - baseUrl: llmBaseUrl, - apiKey: llmApiKey ?? "openmath-local", - allowedHosts: ["localhost", "127.0.0.1"], - }) + ? withLlmLogging( + resolveLanguageModel({ + kind: llmKind, + modelId: llmModel, + baseUrl: llmBaseUrl, + apiKey: llmApiKey ?? "openmath-local", + allowedHosts: ["localhost", "127.0.0.1"], + }), + llmModel, + logLlmCall, + ) : undefined; const solverModel = env.SOLVER_MODEL ?? llmModel; const solverLlm = llm === undefined ? undefined : solverModel === llmModel ? llm - : resolveLanguageModel({ - kind: llmKind, - modelId: solverModel, - baseUrl: llmBaseUrl, - apiKey: llmApiKey ?? "openmath-local", - allowedHosts: ["localhost", "127.0.0.1"], - }); + : withLlmLogging( + resolveLanguageModel({ + kind: llmKind, + modelId: solverModel, + baseUrl: llmBaseUrl, + apiKey: llmApiKey ?? "openmath-local", + allowedHosts: ["localhost", "127.0.0.1"], + }), + `${solverModel} (solver)`, + logLlmCall, + ); const generator = llm === undefined ? undefined : createGeneratorAgent({ @@ -77,13 +95,13 @@ export async function main(): Promise { promptId: "constraint-critic", prompts, }); - const refiner = llm === undefined || generator === undefined + const refiner = llm === undefined ? undefined : createRefinerAgent({ model: llm, modelId: llmModel, promptId: "refiner", - generator, + prompts, }); const solver = solverLlm === undefined ? undefined @@ -114,12 +132,16 @@ export async function main(): Promise { perStepTimeoutMs: env.PER_STEP_TIMEOUT_MS, deterministicFallback: env.DETERMINISTIC_FALLBACK, }, + trace, }); serve({ fetch: app.fetch, port: env.PORT }); console.log( `[openmath/agent] Listening on http://localhost:${env.PORT} (math-engine ${env.MATH_ENGINE_URL})`, ); + console.log( + `[openmath/agent] generator=${llmModel} solver=${solverModel} fallback=${env.DETERMINISTIC_FALLBACK} trace=${env.TRACE_ENABLED === "true" ? resolve(env.TRACE_DIR) : "off"}`, + ); } if (import.meta.url === `file://${process.argv[1]}`) { diff --git a/packages/agent/src/schemas/progress-event.schema.ts b/packages/agent/src/schemas/progress-event.schema.ts index d37368d..b8b62fe 100644 --- a/packages/agent/src/schemas/progress-event.schema.ts +++ b/packages/agent/src/schemas/progress-event.schema.ts @@ -32,12 +32,22 @@ export type StepEvent = z.infer; export const RetryEventSchema = z.object({ type: z.literal("retry"), attempt: z.number().int().min(1), + max_attempts: z.number().int().min(1), reason: z.string(), timestamp: z.string(), }); export type RetryEvent = z.infer; +/** 3단계(generate) 완료 직후 후보 문제 본문 미리보기. FE의 CANDIDATE PREVIEW 스테이지가 소비. */ +export const PreviewEventSchema = z.object({ + type: z.literal("preview"), + latex: z.string().min(1), + timestamp: z.string(), +}); + +export type PreviewEvent = z.infer; + export const ResultEventSchema = z.object({ type: z.literal("result"), candidates: z.array( @@ -65,6 +75,7 @@ export type ErrorEvent = z.infer; export const ProgressEventSchema = z.discriminatedUnion("type", [ StepEventSchema, RetryEventSchema, + PreviewEventSchema, ResultEventSchema, ErrorEventSchema, ]); diff --git a/packages/agent/src/schemas/wire-format.schema.ts b/packages/agent/src/schemas/wire-format.schema.ts index 912a4e6..f9dcab5 100644 --- a/packages/agent/src/schemas/wire-format.schema.ts +++ b/packages/agent/src/schemas/wire-format.schema.ts @@ -21,6 +21,8 @@ export const WireStepStatusSchema = z.enum([ "started", "completed", "failed", + /** 결정론 검증을 수행할 수 없었음 (예: SymPy가 못 푸는 유형). 실패가 아니라 "판정 불가". */ + "unverified", ]); export type WireStepStatus = z.infer; @@ -32,6 +34,21 @@ export const WireStepEventSchema = z.object({ }); export type WireStepEvent = z.infer; +/** 검증 실패 후 재생성 시작을 알림. FE는 3~6단계를 리셋하고 시도 배너를 띄운다. */ +export const WireAttemptEventSchema = z.object({ + attempt: z.number().int().min(1), + max_attempts: z.number().int().min(1), + reason: z.string().nullable(), +}); +export type WireAttemptEvent = z.infer; + +/** 병렬 생성 런 진행 상황 (count > 1 요청). 스텝바는 첫 런만 보여주므로 나머지 런의 완료를 집계로 알림. */ +export const WireRunsEventSchema = z.object({ + completed: z.number().int().min(0), + total: z.number().int().min(1), +}); +export type WireRunsEvent = z.infer; + export const WirePreviewEventSchema = z.object({ latex: z.string().min(1), }); @@ -97,10 +114,12 @@ export const WireErrorEventSchema = z.object({ }); export type WireErrorEvent = z.infer; -export type WireEventName = "step" | "preview" | "result" | "error"; +export type WireEventName = "step" | "preview" | "attempt" | "runs" | "result" | "error"; export type WireSseEvent = | { event: "step"; data: WireStepEvent } | { event: "preview"; data: WirePreviewEvent } + | { event: "attempt"; data: WireAttemptEvent } + | { event: "runs"; data: WireRunsEvent } | { event: "result"; data: WireResultEvent } | { event: "error"; data: WireErrorEvent }; diff --git a/packages/agent/src/server/app.ts b/packages/agent/src/server/app.ts index faf24f2..c499635 100644 --- a/packages/agent/src/server/app.ts +++ b/packages/agent/src/server/app.ts @@ -5,6 +5,7 @@ import { cors } from "hono/cors"; import type { MathEngineClient } from "../tools/math-engine-client.js"; import type { RagClient } from "../tools/rag-client.js"; +import type { RunTraceWriter } from "../tools/run-trace.js"; import type { RunOptions, VerificationWorkflowDeps } from "../workflows/verification-workflow.js"; import { createGenerateRoute } from "./routes/generate.js"; import { createHealthRoute } from "./routes/health.js"; @@ -15,6 +16,7 @@ export interface AppDeps { rag: RagClient; workflow: VerificationWorkflowDeps; workflowOptions?: RunOptions; + trace?: RunTraceWriter; } export function createApp(deps: AppDeps): Hono { @@ -31,7 +33,7 @@ export function createApp(deps: AppDeps): Hono { app.route("/", createHealthRoute(deps.mathEngine)); app.route("/", createSourceProblemsRoute(deps.rag)); - app.route("/", createGenerateRoute(deps.workflow, deps.workflowOptions)); + app.route("/", createGenerateRoute(deps.workflow, deps.workflowOptions, deps.trace)); return app; } diff --git a/packages/agent/src/server/routes/generate.ts b/packages/agent/src/server/routes/generate.ts index 79917d1..3c8cc08 100644 --- a/packages/agent/src/server/routes/generate.ts +++ b/packages/agent/src/server/routes/generate.ts @@ -6,13 +6,18 @@ import { Hono } from "hono"; import { streamSSE } from "hono/streaming"; import { GenerateRequestSchema } from "../../schemas/index.js"; +import type { RunTraceWriter } from "../../tools/run-trace.js"; import { pipeParallelProgressToSse } from "../sse/progress-stream.js"; import type { RunOptions, VerificationWorkflowDeps } from "../../workflows/verification-workflow.js"; import { runVerificationWorkflow } from "../../workflows/verification-workflow.js"; const MAX_PARALLEL_GENERATIONS = 3; -export function createGenerateRoute(deps: VerificationWorkflowDeps, options?: RunOptions): Hono { +export function createGenerateRoute( + deps: VerificationWorkflowDeps, + options?: RunOptions, + trace?: RunTraceWriter, +): Hono { const app = new Hono(); app.post( @@ -25,7 +30,7 @@ export function createGenerateRoute(deps: VerificationWorkflowDeps, options?: Ru const runs = Array.from({ length: count }, () => runVerificationWorkflow(deps, request, options), ); - await pipeParallelProgressToSse(stream, runs); + await pipeParallelProgressToSse(stream, runs, { trace, request }); }); }, ); diff --git a/packages/agent/src/server/sse/progress-stream.ts b/packages/agent/src/server/sse/progress-stream.ts index b4c8034..545b07f 100644 --- a/packages/agent/src/server/sse/progress-stream.ts +++ b/packages/agent/src/server/sse/progress-stream.ts @@ -1,9 +1,15 @@ -/** SSE adapter (D-6). Forwards ProgressEvent from workflow generator into Hono streamSSE. */ +/** SSE adapter (D-6). Forwards ProgressEvent from workflow generators into Hono streamSSE. + * + * 관측성: 모든 런(드라이버 + 백그라운드)의 이벤트를 run 단위 JSONL 트레이스와 + * stdout 한 줄 내레이션으로 남긴다. 트레이스 실패는 스트림에 영향을 주지 않는다. */ + +import { randomUUID } from "node:crypto"; import type { SSEStreamingApi } from "hono/streaming"; -import type { ProgressEvent, ResultEvent } from "../../schemas/index.js"; -import { toWireSseEvent } from "./wire-adapter.js"; +import type { ProgressEvent, ResultEvent, WireRunsEvent } from "../../schemas/index.js"; +import type { RunTraceWriter } from "../../tools/run-trace.js"; +import { toWireSseEvent, toWireStepEvent } from "./wire-adapter.js"; export async function pipeProgressToSse( stream: SSEStreamingApi, @@ -33,16 +39,35 @@ export async function pipeProgressToSse( } } -/** Runs N workflows for one request to produce `count` problems. Only the first - * run's step/preview/error events drive the user-visible bar; the rest are - * drained silently in parallel. Their result candidates are merged into a - * single result event so the FE receives all problems at once. A run that - * fails contributes no problem rather than aborting the others. */ +export interface ParallelProgressOptions { + trace?: RunTraceWriter; + /** 트레이스 run_start 레코드에 남길 원본 요청. */ + request?: unknown; +} + +/** Runs N workflows for one request to produce `count` problems. + * + * - 첫 번째 런(driver)의 step/preview/attempt 이벤트가 사용자 스텝바를 움직인다. + * 나머지 런은 백그라운드에서 트레이스만 남기며 돌고, 결과는 하나의 result로 합쳐진다. + * - 각 런이 끝날 때마다 `runs` 이벤트({completed, total})로 집계를 알린다 (count > 1일 때). + * - driver의 error 이벤트는 즉시 흘리지 않고 버퍼링한다: 다른 런이 문항을 만들어내면 + * 에러 대신 result를 보낸다. 모든 런이 빈손일 때만 에러를 보낸다. */ export async function pipeParallelProgressToSse( stream: SSEStreamingApi, runs: AsyncGenerator[], + options?: ParallelProgressOptions, ): Promise { - const [driver, ...others] = runs; + const total = runs.length; + if (total === 0) return; + + const traced = runs.map((run, index) => + withRunTrace(run, { + trace: options?.trace, + request: options?.request, + runLabel: `${index + 1}/${total}`, + }), + ); + const [driver, ...others] = traced; if (driver === undefined) return; const writeWire = (event: ProgressEvent): Promise => { @@ -52,6 +77,15 @@ export async function pipeParallelProgressToSse( const collected: ResultEvent[] = []; const failures: string[] = []; + const bufferedErrors: Array> = []; + + let completedRuns = 0; + const writeRunsEvent = async (): Promise => { + completedRuns += 1; + if (total <= 1) return; + const data: WireRunsEvent = { completed: completedRuns, total }; + await stream.writeSSE({ event: "runs", data: JSON.stringify(data) }); + }; const drained = others.map(async (run) => { try { @@ -61,21 +95,25 @@ export async function pipeParallelProgressToSse( } catch (err) { failures.push(err instanceof Error ? err.message : String(err)); } + await writeRunsEvent(); }); - let forwardedError = false; try { for await (const event of driver) { if (event.type === "result") { collected.push(event); continue; } - if (event.type === "error") forwardedError = true; + if (event.type === "error") { + bufferedErrors.push(event); + continue; + } await writeWire(event); } } catch (err) { failures.push(err instanceof Error ? err.message : String(err)); } + await writeRunsEvent(); await Promise.all(drained); @@ -84,14 +122,97 @@ export async function pipeParallelProgressToSse( await writeWire({ type: "result", candidates, timestamp: new Date().toISOString() }); return; } - if (!forwardedError) { - await writeWire({ - type: "error", - stage: "orchestrator", - code: "all_generations_failed", - message: failures[0] ?? "No problems were generated.", - recoverable: false, - timestamp: new Date().toISOString(), + const bufferedError = bufferedErrors[0]; + if (bufferedError !== undefined) { + await writeWire(bufferedError); + return; + } + await writeWire({ + type: "error", + stage: "orchestrator", + code: "all_generations_failed", + message: failures[0] ?? "No problems were generated.", + recoverable: false, + timestamp: new Date().toISOString(), + }); +} + +interface RunTraceContext { + trace?: RunTraceWriter; + request?: unknown; + runLabel: string; +} + +/** 런 하나의 이벤트를 그대로 통과시키며 JSONL 트레이스 + stdout 내레이션을 남기는 tee. */ +async function* withRunTrace( + run: AsyncGenerator, + ctx: RunTraceContext, +): AsyncGenerator { + const runId = randomUUID(); + const shortId = runId.slice(0, 8); + const fileKey = `run-${new Date().toISOString().slice(0, 10)}-${runId}`; + let seq = 0; + + await ctx.trace?.append(fileKey, { + run_id: runId, + seq, + event: { type: "run_start", run: ctx.runLabel }, + ...(ctx.request === undefined ? {} : { request: ctx.request }), + }); + + try { + for await (const event of run) { + seq += 1; + await ctx.trace?.append(fileKey, { run_id: runId, seq, event }); + narrate(shortId, ctx.runLabel, event); + yield event; + } + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + seq += 1; + await ctx.trace?.append(fileKey, { + run_id: runId, + seq, + event: { type: "exception", message }, }); + console.warn(`[run ${shortId} ${ctx.runLabel}] ✗ exception: ${message}`); + throw err; } } + +const STEP_STATUS_GLYPH: Record = { + completed: "✓", + failed: "✗", + unverified: "△", +}; + +function narrate(shortId: string, runLabel: string, event: ProgressEvent): void { + const prefix = `[run ${shortId} ${runLabel}]`; + if (event.type === "step") { + if (event.status === "start") return; // done 라인만 — 로그를 조용하게 유지 + const wire = toWireStepEvent(event); + const glyph = STEP_STATUS_GLYPH[wire.status] ?? "·"; + const summary = wire.summary === null || wire.summary === undefined ? "" : ` — ${wire.summary}`; + console.log(`${prefix} ${wire.index}/6 ${wire.name} ${glyph}${summary}`); + return; + } + if (event.type === "retry") { + console.log(`${prefix} ↻ 시도 ${event.attempt}/${event.max_attempts}: ${event.reason}`); + return; + } + if (event.type === "preview") { + console.log(`${prefix} ◇ 후보: ${truncate(event.latex, 80)}`); + return; + } + if (event.type === "result") { + const verdicts = event.candidates.map((candidate) => candidate.verification.overall).join(", "); + console.log(`${prefix} ● 결과 ${event.candidates.length}문항 (${verdicts})`); + return; + } + console.log(`${prefix} ✗ error ${event.stage}/${event.code}: ${event.message}`); +} + +function truncate(text: string, max: number): string { + const collapsed = text.replace(/\s+/g, " ").trim(); + return collapsed.length <= max ? collapsed : `${collapsed.slice(0, max - 1)}…`; +} diff --git a/packages/agent/src/server/sse/wire-adapter.ts b/packages/agent/src/server/sse/wire-adapter.ts index ad1e2eb..5c6a272 100644 --- a/packages/agent/src/server/sse/wire-adapter.ts +++ b/packages/agent/src/server/sse/wire-adapter.ts @@ -3,6 +3,11 @@ * * Keeps the internal BE domain model stable while matching the existing FE hook * contract in `packages/web/hooks/use-verification-stream.ts`. + * + * step 이벤트의 `summary`는 게이트 evidence에서 추린 단계별 한 줄 서사다. + * 실패는 `code: message` 원문을 그대로 노출하고 (D-11 투명성), 성공은 + * "무슨 일이 있었는지"를 한국어로 요약한다. evidence 필드가 없으면 조용히 + * 생략한다 — summary는 항상 best-effort. */ import type { @@ -37,24 +42,125 @@ function mapStepStatus(event: StepEvent): WireStepStatus { if (event.status === "info") return "failed"; const gateStatus = readGateStatus(event.data); if (gateStatus === "failed") return "failed"; + if (gateStatus === "unverified") return "unverified"; return "completed"; } export function toWireStepEvent(event: StepEvent): WireStepEvent { const meta = STEP_META[event.step]; + const status = mapStepStatus(event); return { index: meta.index, name: meta.name, - status: mapStepStatus(event), - summary: readGateFailureSummary(event.data), + status, + summary: stepSummary(event, status), }; } -function readGateStatus(data: unknown): "passed" | "failed" | "skipped" | null { +function stepSummary(event: StepEvent, status: WireStepStatus): string | null { + if (status === "started") return null; + const failure = readGateFailureSummary(event.data); + if (status === "failed" && failure !== null) return failure; + const gate = readGate(event.data); + if (gate === null) return failure; + const narrative = successSummary(event.step, gate); + if (narrative === null) return failure; + return withDuration(narrative, gate); +} + +/** 게이트 evidence → 단계별 성공/판정불가 서사. evidence 계약이 어긋나면 null. */ +function successSummary(step: StepName, gate: Record): string | null { + const evidence = asRecord(gate["evidence"]) ?? {}; + if (step === "rag") return ragSummary(evidence); + if (step === "intent") return intentSummary(evidence); + if (step === "generate") return generateSummary(evidence); + if (step === "sympy_verify") return sympySummary(gate, evidence); + if (step === "re_solve") return reSolveSummary(evidence); + return objectiveSummary(evidence); +} + +function ragSummary(evidence: Record): string | null { + const refs = asNumber(evidence["refs"]); + if (refs === null) return null; + return `참조 ${refs}문항 확보`; +} + +function intentSummary(evidence: Record): string | null { + const code = asString(evidence["objective_code"]); + if (code === null) return null; + const dimensions = asNumber(evidence["dimensions"]); + const base = dimensions === null ? `목표 ${code}` : `목표 ${code} · 평가차원 ${dimensions}개`; + if (evidence["fallback"] === true) { + return `${base} (LLM 응답 실패 — 시드 의도로 대체)`; + } + return base; +} + +function generateSummary(evidence: Record): string | null { + const refinedBy = asStringArray(evidence["refined_by"]) ?? []; + if (refinedBy.includes("deterministic-topic-generator")) { + return "결정론 템플릿 후보 사용"; + } + const model = asString(evidence["model"]); + if (model === null) return null; + const criticRounds = refinedBy.filter((name) => name === "constraint-critic").length; + const refinerRuns = refinedBy.filter((name) => name === "refiner").length; + const parts = [`후보 생성 (${model})`]; + if (criticRounds > 0) parts.push(`Critic ${criticRounds}라운드`); + if (refinerRuns > 0) parts.push(`수정 ${refinerRuns}회`); + const hints = asNumber(evidence["critic_hints_total"]); + if (hints !== null && hints > 0) parts.push(`지적 ${hints}건 반영`); + return parts.join(" · "); +} + +function sympySummary( + gate: Record, + evidence: Record, +): string | null { + if (gate["status"] === "unverified") { + const kind = asString(evidence["verification_kind"]); + const kindLabel = kind === null || kind === "equation" ? "" : ` (${kind} 유형)`; + return `기호 검증 불가${kindLabel} — 독립 재풀이로 확인`; + } + const sympyAnswer = asString(evidence["sympy_answer"]); + if (sympyAnswer === null) return "SymPy 검산 일치"; + return `SymPy 검산 일치 (해: ${sympyAnswer})`; +} + +function reSolveSummary(evidence: Record): string | null { + const derived = asString(evidence["derived_answer"]); + if (derived === null) return null; + const confidence = asString(evidence["confidence"]); + const confidenceLabel = + confidence === "high" ? "높음" : confidence === "medium" ? "중간" : confidence === "low" ? "낮음" : null; + return confidenceLabel === null + ? `재풀이 일치 (${derived})` + : `재풀이 일치 (${derived} · 확신도 ${confidenceLabel})`; +} + +function objectiveSummary(evidence: Record): string | null { + const used = asStringArray(evidence["techniques_used"]); + const overlapping = asStringArray(evidence["overlapping_techniques"]); + if (used === null) return "학습 목표 일치"; + if (overlapping !== null && overlapping.length > 0) { + return `학습 목표 일치 · 기법 ${overlapping.length}개 확인`; + } + return `학습 목표 일치 · 사용 기법 ${used.length}개`; +} + +function withDuration(summary: string, gate: Record): string { + const durationMs = asNumber(gate["duration_ms"]); + if (durationMs === null || durationMs < 100) return summary; + return `${summary} · ${(durationMs / 1000).toFixed(1)}초`; +} + +function readGateStatus(data: unknown): "passed" | "failed" | "skipped" | "unverified" | null { const gate = readGate(data); if (gate === null) return null; const status = gate["status"]; - if (status === "passed" || status === "failed" || status === "skipped") return status; + if (status === "passed" || status === "failed" || status === "skipped" || status === "unverified") { + return status; + } return null; } @@ -81,6 +187,19 @@ function asRecord(value: unknown): Record | null { return Object.fromEntries(entries); } +function asNumber(value: unknown): number | null { + return typeof value === "number" && Number.isFinite(value) ? value : null; +} + +function asString(value: unknown): string | null { + return typeof value === "string" && value.length > 0 ? value : null; +} + +function asStringArray(value: unknown): string[] | null { + if (!Array.isArray(value)) return null; + return value.filter((item): item is string => typeof item === "string"); +} + function mapVerificationStatus( overall: Verification["overall"], ): WireResultProblem["verification_status"] { @@ -141,15 +260,14 @@ function toWireErrorEvent(event: Extract): Wir }; } -function retryToWireStep(event: Extract): WireStepEvent { - return { - ...STEP_META.generate, - status: "started", - summary: `재시도 ${event.attempt}`, - }; -} - +/** 내부 메시지는 노출하지 않되, 알려진 실패 코드는 사용자가 행동할 수 있는 문구로 변환. */ function publicErrorMessage(event: Extract): string { + if (event.code === "no_refs") { + return "선택한 단원에서 참조 문항을 찾지 못했습니다. 다른 단원이나 기준 문항으로 시도해 주세요."; + } + if (event.code === "generation_failed" || event.code === "all_generations_failed") { + return "문제 생성에 반복 실패했습니다. 잠시 후 다시 시도해 주세요."; + } if (event.recoverable) { return "검증 중 복구 가능한 오류가 발생했습니다. 다시 시도합니다."; } @@ -161,7 +279,16 @@ export function toWireSseEvent(event: ProgressEvent): WireSseEvent { case "step": return { event: "step", data: toWireStepEvent(event) }; case "retry": - return { event: "step", data: retryToWireStep(event) }; + return { + event: "attempt", + data: { + attempt: event.attempt, + max_attempts: event.max_attempts, + reason: event.reason, + }, + }; + case "preview": + return { event: "preview", data: { latex: event.latex } }; case "result": return { event: "result", data: toWireResultEvent(event) }; case "error": diff --git a/packages/agent/src/steps/independent-resolve.ts b/packages/agent/src/steps/independent-resolve.ts index 67ed2d3..759cbcd 100644 --- a/packages/agent/src/steps/independent-resolve.ts +++ b/packages/agent/src/steps/independent-resolve.ts @@ -15,7 +15,8 @@ export interface IndependentResolveDeps { export interface IndependentResolveInput { candidate: GeneratedProblem; - sympyGate: GateResult; + /** 참고용 evidence로만 쓰인다. sympy_verify와 병렬 실행 시 생략. */ + sympyGate?: GateResult; } export interface IndependentResolveOutput { @@ -50,7 +51,7 @@ export async function independentResolve( confidence: attempt.confidence, derived_answer: attempt.derived_answer, expected_answer: input.candidate.expected_answer, - sympy_status: input.sympyGate.status, + ...(input.sympyGate === undefined ? {} : { sympy_status: input.sympyGate.status }), ...(answerDebug.skippedReasons.length === 0 ? {} : { answer_equivalence_skipped_reasons: answerDebug.skippedReasons }), diff --git a/packages/agent/src/steps/intent-extraction.ts b/packages/agent/src/steps/intent-extraction.ts index 3ad9ec5..495483e 100644 --- a/packages/agent/src/steps/intent-extraction.ts +++ b/packages/agent/src/steps/intent-extraction.ts @@ -64,7 +64,10 @@ export async function extractIntent( step: "intent", status: "passed", duration_ms: Date.now() - started, - evidence: { objective_code: intent.objective_code }, + evidence: { + objective_code: intent.objective_code, + dimensions: intent.evaluation_dimensions.length, + }, }, }; } catch (err) { @@ -79,6 +82,7 @@ export async function extractIntent( duration_ms: Date.now() - started, evidence: { objective_code: fallback.objective_code, + dimensions: fallback.evaluation_dimensions.length, fallback: true, dimensions_source: dimensionsSource, llm_error: err instanceof Error ? err.message : String(err), diff --git a/packages/agent/src/steps/problem-generation.ts b/packages/agent/src/steps/problem-generation.ts index 4a8075c..52df561 100644 --- a/packages/agent/src/steps/problem-generation.ts +++ b/packages/agent/src/steps/problem-generation.ts @@ -45,8 +45,17 @@ export interface ProblemGenerationInput { deterministicFallback?: DeterministicFallbackMode; } +export interface CriticRoundRecord { + round: number; + source: "constraint-critic" | "deterministic-topic-guard"; + passes: boolean; + hints: string[]; +} + export interface ProblemGenerationOutput { - data: GeneratedProblem; + /** null = 생성 자체가 실패 (timeout, provider error). 게이트는 failed로 남고 + * 오케스트레이터가 재시도 여부를 결정한다 — 워크플로우를 죽이지 않는다. */ + data: GeneratedProblem | null; gate: GateResult; refined_by: string[]; } @@ -58,6 +67,7 @@ export async function generateProblem( const started = Date.now(); const refinedBy: string[] = []; const normalizationSkippedReasons: string[] = []; + const criticRounds: CriticRoundRecord[] = []; try { const candidate = await withTimeout(async (signal) => { const fallbackMode: DeterministicFallbackMode = input.deterministicFallback ?? "first"; @@ -85,6 +95,12 @@ export async function generateProblem( const guardHints = deterministicTopicGuardHints(input.request, current); if (guardHints.length > 0) { refinedBy.push("deterministic-topic-guard"); + criticRounds.push({ + round: round + 1, + source: "deterministic-topic-guard", + passes: false, + hints: guardHints, + }); const replacement = deterministicGuardReplacement(input.request, current); if (replacement !== null) { current = replacement; @@ -112,6 +128,12 @@ export async function generateProblem( signal, }); refinedBy.push("constraint-critic"); + criticRounds.push({ + round: round + 1, + source: "constraint-critic", + passes: critique.passes, + hints: critique.hints, + }); if (critique.passes || critique.hints.length === 0) return current; current = await deps.refiner.refine({ prior: current, @@ -147,6 +169,8 @@ export async function generateProblem( expected_answer: candidate.expected_answer, model: candidate.generation_metadata.model, refined_by: refinedBy, + critic_rounds: criticRounds, + critic_hints_total: criticRounds.reduce((sum, record) => sum + record.hints.length, 0), ...(normalizationSkippedReasons.length === 0 ? {} : { normalization_skipped_reasons: normalizationSkippedReasons }), @@ -156,11 +180,15 @@ export async function generateProblem( }; } catch (err) { return { - data: await Promise.reject(err), + data: null, gate: { step: "generate", status: "failed", duration_ms: Date.now() - started, + evidence: { + refined_by: refinedBy, + critic_rounds: criticRounds, + }, failure_detail: { code: "generation_failed", message: err instanceof Error ? err.message : String(err), diff --git a/packages/agent/src/tools/index.ts b/packages/agent/src/tools/index.ts index da7a8d0..9bfb03e 100644 --- a/packages/agent/src/tools/index.ts +++ b/packages/agent/src/tools/index.ts @@ -1,6 +1,8 @@ +export * from "./llm-logging.js"; export * from "./llm-provider.js"; export * from "./latex-formatter.js"; export * from "./math-engine-client.js"; export * from "./prompt-loader.js"; export * from "./rag-client.js"; +export * from "./run-trace.js"; export * from "./schema-loader.js"; diff --git a/packages/agent/src/tools/llm-logging.ts b/packages/agent/src/tools/llm-logging.ts new file mode 100644 index 0000000..801bdbd --- /dev/null +++ b/packages/agent/src/tools/llm-logging.ts @@ -0,0 +1,66 @@ +/** + * LLM call logging middleware (DX). + * + * resolveLanguageModel이 만든 모델을 감싸 모든 generateObject/generateText 호출의 + * 지연·토큰·종료 사유를 stdout 한 줄 + (옵션) 트레이스 레코드로 남긴다. + * 에이전트 코드는 전혀 몰라도 된다 — 모델 인스턴스 단일 지점에서 계측. + */ + +import { wrapLanguageModel, type LanguageModel, type LanguageModelV1Middleware } from "ai"; + +export type LlmCallRecord = { + type: "llm_call"; + label: string; + duration_ms: number; + prompt_tokens?: number; + completion_tokens?: number; + finish_reason?: string; + error?: string; +}; + +export function withLlmLogging( + model: LanguageModel, + label: string, + onRecord?: (record: LlmCallRecord) => void, +): LanguageModel { + const middleware: LanguageModelV1Middleware = { + wrapGenerate: async ({ doGenerate }) => { + const started = Date.now(); + try { + const result = await doGenerate(); + const record: LlmCallRecord = { + type: "llm_call", + label, + duration_ms: Date.now() - started, + prompt_tokens: finiteOrUndefined(result.usage?.promptTokens), + completion_tokens: finiteOrUndefined(result.usage?.completionTokens), + finish_reason: result.finishReason, + }; + console.log( + `[llm] ${label} ${(record.duration_ms / 1000).toFixed(1)}s` + + ` in=${record.prompt_tokens ?? "?"} out=${record.completion_tokens ?? "?"}` + + ` (${record.finish_reason ?? "?"})`, + ); + onRecord?.(record); + return result; + } catch (err) { + const record: LlmCallRecord = { + type: "llm_call", + label, + duration_ms: Date.now() - started, + error: err instanceof Error ? err.message : String(err), + }; + console.warn( + `[llm] ${label} failed after ${(record.duration_ms / 1000).toFixed(1)}s: ${record.error}`, + ); + onRecord?.(record); + throw err; + } + }, + }; + return wrapLanguageModel({ model, middleware }); +} + +function finiteOrUndefined(value: number | undefined): number | undefined { + return value !== undefined && Number.isFinite(value) ? value : undefined; +} diff --git a/packages/agent/src/tools/run-trace.ts b/packages/agent/src/tools/run-trace.ts new file mode 100644 index 0000000..acb910e --- /dev/null +++ b/packages/agent/src/tools/run-trace.ts @@ -0,0 +1,51 @@ +/** + * Run trace writer — 워크플로우 런 단위 JSONL 관측 로그 (DX). + * + * `TRACE_DIR/run--.jsonl` 에 ProgressEvent 전문(게이트 evidence 포함)을, + * `TRACE_DIR/llm.jsonl` 에 LLM 호출 지연/토큰을 append한다. + * + * 트레이스는 절대 파이프라인을 깨면 안 된다 — 모든 실패는 warn 후 무시. + */ + +import { appendFile, mkdir } from "node:fs/promises"; +import { join } from "node:path"; + +export interface RunTraceWriter { + /** fileKey: 확장자 없는 파일 이름 (`run-...` 또는 `llm`). record는 JSON 직렬화 가능해야 한다. */ + append(fileKey: string, record: Record): Promise; +} + +export interface FsRunTraceWriterOptions { + dir: string; + enabled: boolean; +} + +export function createFsRunTraceWriter(opts: FsRunTraceWriterOptions): RunTraceWriter { + let dirReady: Promise | undefined; + + function ensureDir(): Promise { + dirReady ??= mkdir(opts.dir, { recursive: true }).then(() => undefined); + return dirReady; + } + + return { + async append(fileKey, record) { + if (!opts.enabled) return; + try { + await ensureDir(); + const line = JSON.stringify({ ts: new Date().toISOString(), ...record }); + await appendFile(join(opts.dir, `${fileKey}.jsonl`), `${line}\n`, "utf8"); + } catch (err) { + console.warn( + `[trace] write failed (${fileKey}): ${err instanceof Error ? err.message : String(err)}`, + ); + } + }, + }; +} + +export const NOOP_TRACE_WRITER: RunTraceWriter = { + async append() { + /* tracing disabled */ + }, +}; diff --git a/packages/agent/src/workflows/verification-workflow.ts b/packages/agent/src/workflows/verification-workflow.ts index ddfd99c..a660fcb 100644 --- a/packages/agent/src/workflows/verification-workflow.ts +++ b/packages/agent/src/workflows/verification-workflow.ts @@ -1,6 +1,8 @@ /** Verification workflow — deterministic Orchestrator (D-5 outer skeleton). * Owns the user-visible 6-step progress. Streams ProgressEvent via async generator. */ +import { randomUUID } from "node:crypto"; + import type { LanguageModel } from "ai"; import type { @@ -144,37 +146,53 @@ export async function* runVerificationWorkflow( const candidate = generation.data; yield step("generate", "done", timestamp(), generation.gate); - yield step("sympy_verify", "start", timestamp()); - const sympy = await verifyWithSympy( - { mathEngine: deps.mathEngine, perStepTimeoutMs }, - { candidate }, - ); - yield step("sympy_verify", "done", timestamp(), sympy.gate); + let gates: GateResult[]; + if (candidate === null) { + // 생성 실패는 failed 게이트로 강등하고 4~6단계를 skipped 처리한다. + // 재시도 정책이 generate 힌트로 다음 attempt를 시도할 수 있다 — 워크플로우는 죽지 않는다. + gates = [ + ragGate, + intentStep.gate, + generation.gate, + skippedGate("sympy_verify"), + skippedGate("re_solve"), + skippedGate("objective_map"), + ]; + } else { + yield { type: "preview", latex: candidate.question_text, timestamp: timestamp() }; - yield step("re_solve", "start", timestamp()); - const reSolve = await independentResolve( - { solver: deps.solver, mathEngine: deps.mathEngine, perStepTimeoutMs }, - { candidate, sympyGate: sympy.gate }, - ); - yield step("re_solve", "done", timestamp(), reSolve.gate); + // sympy_verify(결정론)와 re_solve(LLM)는 서로 독립 — 동시 실행으로 지연을 줄인다. + yield step("sympy_verify", "start", timestamp()); + yield step("re_solve", "start", timestamp()); + const [sympy, reSolve] = await Promise.all([ + verifyWithSympy({ mathEngine: deps.mathEngine, perStepTimeoutMs }, { candidate }), + independentResolve( + { solver: deps.solver, mathEngine: deps.mathEngine, perStepTimeoutMs }, + { candidate }, + ), + ]); + yield step("sympy_verify", "done", timestamp(), sympy.gate); + yield step("re_solve", "done", timestamp(), reSolve.gate); - yield step("objective_map", "start", timestamp()); - const objective = await mapObjective( - { llm: deps.objectiveLlm, prompts: deps.prompts, perStepTimeoutMs }, - { request, refs, candidate, intent: intentStep.data, strategy }, - ); - yield step("objective_map", "done", timestamp(), objective.gate); + yield step("objective_map", "start", timestamp()); + const objective = await mapObjective( + { llm: deps.objectiveLlm, prompts: deps.prompts, perStepTimeoutMs }, + { request, refs, candidate, intent: intentStep.data, strategy }, + ); + yield step("objective_map", "done", timestamp(), objective.gate); + + gates = [ + ragGate, + intentStep.gate, + generation.gate, + sympy.gate, + reSolve.gate, + objective.gate, + ]; + } - const gates = [ - ragGate, - intentStep.gate, - generation.gate, - sympy.gate, - reSolve.gate, - objective.gate, - ]; const verification: Verification = { - candidate_id: candidate.candidate_id, + candidate_id: candidate?.candidate_id ?? randomUUID(), overall: acceptance.decide(gates, attempt), gates, attempt_count: attempt, @@ -193,6 +211,17 @@ export async function* runVerificationWorkflow( candidate, verification, }); + if (finalResult === null) { + yield { + type: "error", + stage: "generate", + code: "generation_failed", + message: "Problem generation failed on every attempt", + recoverable: false, + timestamp: timestamp(), + }; + return { verifications }; + } if (finalResult.verification !== verification) { verifications[verifications.length - 1] = finalResult.verification; } @@ -210,27 +239,43 @@ export async function* runVerificationWorkflow( yield { type: "retry", attempt, + max_attempts: maxAttempts, reason: retry.refinementHint ?? "verification failed", timestamp: timestamp(), }; } } +function skippedGate(stepName: StepName): GateResult { + return { + step: stepName, + status: "skipped", + duration_ms: 0, + failure_detail: { + code: "skipped_generation_failed", + message: "Skipped because problem generation failed", + }, + }; +} + +/** 최종 attempt의 결과 선택. candidate가 null(생성 실패)이고 결정론 템플릿도 없으면 + * null을 반환한다 — 호출자는 result 대신 error 이벤트를 emit해야 한다. */ function selectFinalResult(input: { readonly deterministicFallback: DeterministicFallbackMode; readonly request: GenerateRequest; readonly intent: Intent; readonly refs: readonly RagResult[]; readonly attempt: number; - readonly candidate: GeneratedProblem; + readonly candidate: GeneratedProblem | null; readonly verification: Verification; -}): { problem: GeneratedProblem; verification: Verification } { - if (input.deterministicFallback !== "last-resort") { - return { problem: input.candidate, verification: input.verification }; - } - if (input.verification.overall !== "rejected") { - return { problem: input.candidate, verification: input.verification }; - } +}): { problem: GeneratedProblem; verification: Verification } | null { + const keepCandidate = (): { problem: GeneratedProblem; verification: Verification } | null => + input.candidate === null + ? null + : { problem: input.candidate, verification: input.verification }; + + if (input.deterministicFallback !== "last-resort") return keepCandidate(); + if (input.verification.overall !== "rejected") return keepCandidate(); const fallback = deterministicInitialCandidate({ request: input.request, @@ -238,9 +283,7 @@ function selectFinalResult(input: { refs: input.refs, attempt: input.attempt, }); - if (fallback === null) { - return { problem: input.candidate, verification: input.verification }; - } + if (fallback === null) return keepCandidate(); const problem = withDeterministicGeneratorMarker(fallback); const verification = { ...input.verification, candidate_id: problem.candidate_id }; diff --git a/packages/agent/tests/backend-contract.test.ts b/packages/agent/tests/backend-contract.test.ts index a887048..30e442d 100644 --- a/packages/agent/tests/backend-contract.test.ts +++ b/packages/agent/tests/backend-contract.test.ts @@ -112,20 +112,20 @@ describe("SSE wire adapter", () => { }); }); - it("sanitizes retry reasons and internal errors", () => { + it("maps retry events to attempt events and sanitizes internal errors", () => { const retry = toWireSseEvent({ type: "retry", attempt: 2, - reason: "internal stack/path/token should not leak", + max_attempts: 3, + reason: "SymPy 검산 불일치: 기대답 2, 엔진 결과 3 — 계수를 다시 검산하라", timestamp: "2026-05-21T00:00:00.000Z", }); expect(retry).toEqual({ - event: "step", + event: "attempt", data: { - index: 3, - name: "문제 생성", - status: "started", - summary: "재시도 2", + attempt: 2, + max_attempts: 3, + reason: "SymPy 검산 불일치: 기대답 2, 엔진 결과 3 — 계수를 다시 검산하라", }, }); @@ -146,6 +146,69 @@ describe("SSE wire adapter", () => { }); }); + it("maps preview events and unverified gates onto the wire", () => { + const preview = toWireSseEvent({ + type: "preview", + latex: "x**2 - 5*x + 6 = 0", + timestamp: "2026-05-21T00:00:00.000Z", + }); + expect(preview).toEqual({ + event: "preview", + data: { latex: "x**2 - 5*x + 6 = 0" }, + }); + + const unverified = toWireSseEvent({ + type: "step", + step: "sympy_verify", + status: "done", + timestamp: "2026-05-21T00:00:00.000Z", + data: { + gate: { + step: "sympy_verify", + status: "unverified", + duration_ms: 42, + evidence: { engine: "sympy", verification_kind: "geometry" }, + failure_detail: { + code: "sympy_unverified", + message: "No deterministic SymPy verifier for geometry", + }, + }, + }, + }); + expect(unverified.event).toBe("step"); + if (unverified.event !== "step") throw new Error("expected step event"); + expect(unverified.data.status).toBe("unverified"); + expect(unverified.data.summary).toContain("기호 검증 불가"); + }); + + it("narrates successful gates in step summaries", () => { + const wire = toWireSseEvent({ + type: "step", + step: "generate", + status: "done", + timestamp: "2026-05-21T00:00:00.000Z", + data: { + gate: { + step: "generate", + status: "passed", + duration_ms: 12_300, + evidence: { + candidate_id: "00000000-0000-0000-0000-000000000001", + model: "gpt-5.5(xhigh)", + refined_by: ["constraint-critic", "refiner", "constraint-critic"], + critic_hints_total: 2, + }, + }, + }, + }); + expect(wire.event).toBe("step"); + if (wire.event !== "step") throw new Error("expected step event"); + expect(wire.data.status).toBe("completed"); + expect(wire.data.summary).toBe( + "후보 생성 (gpt-5.5(xhigh)) · Critic 2라운드 · 수정 1회 · 지적 2건 반영 · 12.3초", + ); + }); + it("flattens domain result events for the frontend", () => { const problem: GeneratedProblem = { candidate_id: "00000000-0000-0000-0000-000000000001", diff --git a/packages/agent/tests/deterministic-fallback.test.ts b/packages/agent/tests/deterministic-fallback.test.ts index 5d6bc60..3bc08fb 100644 --- a/packages/agent/tests/deterministic-fallback.test.ts +++ b/packages/agent/tests/deterministic-fallback.test.ts @@ -141,7 +141,7 @@ describe("generateProblem deterministicFallback behavior", () => { expect(spy).not.toHaveBeenCalled(); expect(result.refined_by).toContain("deterministic-topic-generator"); - expect(result.data.generation_metadata.model).toBe("deterministic-topic-generator"); + expect(result.data?.generation_metadata.model).toBe("deterministic-topic-generator"); expect(result.gate.status).toBe("passed"); }); @@ -154,7 +154,7 @@ describe("generateProblem deterministicFallback behavior", () => { ); expect(spy).not.toHaveBeenCalled(); - expect(result.data.generation_metadata.model).toBe("deterministic-topic-generator"); + expect(result.data?.generation_metadata.model).toBe("deterministic-topic-generator"); }); it("skips the deterministic template when mode='off' (goes through LLM generator)", async () => { @@ -167,8 +167,8 @@ describe("generateProblem deterministicFallback behavior", () => { expect(spy).toHaveBeenCalledTimes(1); expect(result.refined_by).not.toContain("deterministic-topic-generator"); - expect(result.data.generation_metadata.model).toBe("llm-test-model"); - expect(result.data.candidate_id).toBe(llmCandidate.candidate_id); + expect(result.data?.generation_metadata.model).toBe("llm-test-model"); + expect(result.data?.candidate_id).toBe(llmCandidate.candidate_id); }); it("skips the deterministic template up-front when mode='last-resort'", async () => { @@ -188,7 +188,7 @@ describe("generateProblem deterministicFallback behavior", () => { expect(spy).toHaveBeenCalledTimes(1); expect(result.refined_by).not.toContain("deterministic-topic-generator"); - expect(result.data.generation_metadata.model).toBe("llm-test-model"); - expect(result.data.candidate_id).toBe(llmCandidate.candidate_id); + expect(result.data?.generation_metadata.model).toBe("llm-test-model"); + expect(result.data?.candidate_id).toBe(llmCandidate.candidate_id); }); }); diff --git a/packages/agent/tests/integration/real-llm-e2e.test.ts b/packages/agent/tests/integration/real-llm-e2e.test.ts index df60812..c53fe04 100644 --- a/packages/agent/tests/integration/real-llm-e2e.test.ts +++ b/packages/agent/tests/integration/real-llm-e2e.test.ts @@ -176,7 +176,7 @@ async function buildRealWorkflow(): Promise { model: llm, modelId: llmModelId, promptId: "refiner", - generator, + prompts, }); const solver = createSolverAgent({ model: solverLlm, diff --git a/packages/agent/tests/problem-generation.test.ts b/packages/agent/tests/problem-generation.test.ts index 185593b..5c02888 100644 --- a/packages/agent/tests/problem-generation.test.ts +++ b/packages/agent/tests/problem-generation.test.ts @@ -83,7 +83,7 @@ describe("generateProblem", () => { ); expect(result.gate.status).toBe("passed"); - expect(result.data.expected_answer).toBe("5, -2"); + expect(result.data?.expected_answer).toBe("5, -2"); expect(result.gate.evidence).toMatchObject({ normalization_skipped_reasons: ["answer normalization skipped: math-engine rejected extracted equation"], }); @@ -121,7 +121,7 @@ describe("generateProblem", () => { }, ); - expect(result.data.expected_answer).toBe("②"); + expect(result.data?.expected_answer).toBe("②"); }); it("refines integer-and-rational candidates that drift into radical calculation", async () => { @@ -169,7 +169,7 @@ describe("generateProblem", () => { }, ); - expect(result.data.question_text).toBe(goodCandidate.question_text); + expect(result.data?.question_text).toBe(goodCandidate.question_text); expect(hints.join("\n")).toContain("정수와 유리수"); expect(result.refined_by).toContain("deterministic-topic-guard"); }); @@ -213,7 +213,101 @@ describe("generateProblem", () => { }, ); - expect(result.data.question_text).toBe(goodCandidate.question_text); + expect(result.data?.question_text).toBe(goodCandidate.question_text); + }); + + it("returns a failed gate with null data instead of throwing when generation fails", async () => { + const result = await generateProblem( + { + generator: { + generate: async () => { + throw new Error("provider exploded"); + }, + }, + critic, + refiner, + mathEngine, + }, + { + request: { + grade: 3, + topic: "9수02-10", + topic_name: "이차방정식의 활용", + mode: "structural", + school_level: "middle", + dims: ["A"], + count: 1, + difficulty: "medium", + problem_type: "objective", + }, + intent: candidate.inferred_intent, + refs: [], + strategy: null, + attempt: 1, + deterministicFallback: "off", + }, + ); + + expect(result.data).toBeNull(); + expect(result.gate.status).toBe("failed"); + expect(result.gate.failure_detail?.code).toBe("generation_failed"); + expect(result.gate.failure_detail?.message).toContain("provider exploded"); + }); + + it("records critic rounds and hints in gate evidence", async () => { + const fixedCandidate: GeneratedProblem = { + ...candidate, + question_text: "다음 방정식을 풀어라. x**2 - 9*x + 20 = 0", + expected_answer: "4, 5", + }; + let critiqueCalls = 0; + const result = await generateProblem( + { + generator: { generate: async () => candidate }, + critic: { + critique: async () => { + critiqueCalls += 1; + return critiqueCalls === 1 + ? { passes: false, hints: ["문구를 더 명확히 하라"] } + : { passes: true, hints: [] }; + }, + }, + refiner: { refine: async () => fixedCandidate }, + mathEngine, + }, + { + request: { + grade: 3, + topic: "9수02-10", + topic_name: "이차방정식의 활용", + mode: "structural", + school_level: "middle", + dims: ["A"], + count: 1, + difficulty: "medium", + problem_type: "objective", + }, + intent: candidate.inferred_intent, + refs: [], + strategy: null, + attempt: 1, + deterministicFallback: "off", + }, + ); + + expect(result.gate.evidence).toMatchObject({ + critic_hints_total: 1, + critic_rounds: [ + { + round: 1, + source: "constraint-critic", + passes: false, + hints: ["문구를 더 명확히 하라"], + }, + { round: 2, source: "constraint-critic", passes: true, hints: [] }, + ], + }); + expect(result.refined_by).toEqual(["constraint-critic", "refiner", "constraint-critic"]); }); it("refines geometry candidates that depend on an unseen figure", async () => { @@ -255,6 +349,6 @@ describe("generateProblem", () => { }, ); - expect(result.data.question_text).toBe(goodCandidate.question_text); + expect(result.data?.question_text).toBe(goodCandidate.question_text); }); }); diff --git a/packages/agent/tests/refiner-agent.test.ts b/packages/agent/tests/refiner-agent.test.ts new file mode 100644 index 0000000..89db3af --- /dev/null +++ b/packages/agent/tests/refiner-agent.test.ts @@ -0,0 +1,162 @@ +import type { LanguageModel } from "ai"; +import { beforeEach, describe, expect, it, vi } from "vitest"; + +import { createRefinerAgent } from "../src/agents/refiner-agent.js"; +import type { GenerateRequest, GeneratedProblem, Intent } from "../src/schemas/index.js"; +import type { LoadedPrompt, PromptLoader } from "../src/tools/prompt-loader.js"; + +const aiMock = vi.hoisted(() => ({ + generateObject: vi.fn(), + NoObjectGeneratedError: { + isInstance(error: unknown) { + return error instanceof Error && error.name === "NoObjectGeneratedError"; + }, + }, +})); + +vi.mock("ai", () => ({ + generateObject: aiMock.generateObject, + NoObjectGeneratedError: aiMock.NoObjectGeneratedError, +})); + +const request: GenerateRequest = { + grade: 3, + topic: "9수02-10", + topic_name: "이차방정식의 활용", + mode: "structural", + school_level: "middle", + dims: ["A"], + count: 1, + difficulty: "medium", + problem_type: "objective", +}; + +const intent: Intent = { + objective_code: "9수02-10", + objective_description: "이차방정식을 활용하여 문제를 해결한다.", + evaluation_dimensions: [{ id: "A", description: "이차방정식 활용", must_preserve: true }], + required_techniques: ["quadratic_equation"], + forbidden_techniques: [], + surface_constraints: { difficulty: "medium", problem_type: "objective" }, +}; + +const prior: GeneratedProblem = { + candidate_id: "00000000-0000-0000-0000-000000000077", + mode: "structural", + generation_kind: "equation", + question_text: "두 수의 곱이 12이고 합이 7일 때 두 수를 구하시오.", + expected_answer: "3, 4", + techniques_used: ["quadratic_equation"], + proposed_solution_trace: "x(7 - x) = 12를 풀어 두 수를 구한다.", + source_refs: ["ref-1"], + inferred_intent: intent, + generation_metadata: { + model: "test-model", + temperature: 0.35, + prompt_id: "problem-generator", + prompt_version: "0.1.2", + attempt: 2, + generated_at: "2026-06-11T00:00:00.000Z", + }, +}; + +const refinedObject = { + question_text: "두 자연수의 곱이 20이고 합이 9일 때 두 수를 구하시오.", + expected_answer: "4, 5", + techniques_used: ["quadratic_equation"], + proposed_solution_trace: "x(9 - x) = 20을 정리해 이차방정식으로 푼다.", +}; + +beforeEach(() => { + aiMock.generateObject.mockReset(); +}); + +describe("createRefinerAgent", () => { + it("refines via its own prompt with prior + hints (not a blank regeneration)", async () => { + aiMock.generateObject.mockResolvedValue({ object: refinedObject }); + const render = vi.fn((vars: Record) => JSON.stringify(vars)); + const agent = createRefinerAgent({ + model: {} as LanguageModel, + modelId: "test-model", + promptId: "refiner", + prompts: promptLoader(render), + }); + + const refined = await agent.refine({ + prior, + request, + intent, + refs: [], + strategy: null, + attempt: 2, + hints: ["선택지가 없는데 객관식으로 서술돼 있다"], + }); + + expect(refined.question_text).toBe(refinedObject.question_text); + expect(refined.generation_metadata.prompt_id).toBe("refiner"); + expect(refined.generation_metadata.model).toBe("test-model"); + expect(refined.generation_metadata.temperature).toBe(0.5); + expect(refined.generation_metadata.attempt).toBe(2); + expect(refined.inferred_intent.objective_code).toBe(intent.objective_code); + + expect(render).toHaveBeenCalledTimes(1); + expect(render.mock.calls[0]?.[0]).toMatchObject({ + prior, + hints: ["선택지가 없는데 객관식으로 서술돼 있다"], + }); + expect(aiMock.generateObject).toHaveBeenCalledWith( + expect.objectContaining({ temperature: 0.5 }), + ); + }); + + it("retries once with a schema hint when the model output fails validation", async () => { + const schemaError = new Error("expected_answer is required"); + schemaError.name = "NoObjectGeneratedError"; + aiMock.generateObject + .mockRejectedValueOnce(schemaError) + .mockResolvedValueOnce({ object: refinedObject }); + const render = vi.fn((vars: Record) => JSON.stringify(vars)); + const agent = createRefinerAgent({ + model: {} as LanguageModel, + modelId: "test-model", + promptId: "refiner", + prompts: promptLoader(render), + }); + + const refined = await agent.refine({ + prior, + request, + intent, + refs: [], + strategy: null, + attempt: 1, + hints: ["정답 표기를 plain text로"], + }); + + expect(refined.expected_answer).toBe(refinedObject.expected_answer); + expect(aiMock.generateObject).toHaveBeenCalledTimes(2); + expect(render.mock.calls[1]?.[0]).toMatchObject({ + schemaError: "expected_answer is required", + }); + }); +}); + +function promptLoader(render: LoadedPrompt["render"]): PromptLoader { + return { + load: async (id) => ({ + metadata: { + id, + version: "0.2.0", + model: "test-model", + temperature: 0.5, + max_tokens: 2000, + schema: "GeneratedProblemSchema", + variables: ["prior", "request", "intent", "refs", "strategy", "hints"], + owner: "비할당", + updated: "2026-06-11", + }, + rawBody: "", + render, + }), + }; +} diff --git a/packages/agent/tests/verification-workflow-generate-failure.test.ts b/packages/agent/tests/verification-workflow-generate-failure.test.ts new file mode 100644 index 0000000..1833059 --- /dev/null +++ b/packages/agent/tests/verification-workflow-generate-failure.test.ts @@ -0,0 +1,200 @@ +import { MockLanguageModelV1 } from "ai/test"; +import { describe, expect, it, vi } from "vitest"; + +import type { + ConstraintCriticAgent, + GeneratorAgent, + RefinerAgent, + SolverAgent, +} from "../src/agents/index.js"; +import type { + GenerateRequest, + ProgressEvent, + RagResult, + Strategy, +} from "../src/schemas/index.js"; +import type { MathEngineClient } from "../src/tools/math-engine-client.js"; +import type { PromptLoader } from "../src/tools/prompt-loader.js"; +import type { RagClient } from "../src/tools/rag-client.js"; +import type { StrategyLoader } from "../src/tools/schema-loader.js"; +import { runVerificationWorkflow } from "../src/workflows/verification-workflow.js"; + +const request: GenerateRequest = { + grade: 3, + topic: "9수02-10", + topic_name: "이차방정식의 활용", + mode: "structural", + school_level: "middle", + dims: ["활용"], + count: 1, + difficulty: "medium", + problem_type: "objective", +}; + +const strategy: Strategy = { + code: "9수02-10", + title: "이차방정식의 활용", + school_level: "middle", + grade: 3, + techniques: { + required_at_least_one_of: ["quadratic_equation"], + forbidden: [], + }, + evaluation_dimensions: [{ id: "A", description: "활용", must_preserve: true }], + difficulty_range: ["medium"], + problem_types_supported: ["objective"], + structural_transforms: [{ kind: "coefficient_swap", range: [1, 9], exclude_zero: true }], + conceptual_transforms: [], +}; + +const refs: RagResult[] = [ + { + item_id: "ref-1", + match_reason: "hybrid", + problem: { + item_id: "ref-1", + source_dataset: "111", + split: "train", + source_label_type: "problem_label", + school_level: "middle", + grade: 3, + semester: null, + topic_code: "9수02-10", + topic_name: "이차방정식의 활용", + achievement_standard: "이차방정식을 활용하여 문제를 해결한다.", + question_text: "가로가 세로보다 3cm 긴 직사각형의 넓이가 10cm^2이다.", + answer_text: "2cm, 5cm", + explanation_text: null, + choice_blocks: [], + problem_type_norm: "objective", + difficulty_norm: "medium", + question_image_relpath: null, + answer_image_relpath: null, + question_json_relpath: null, + answer_json_relpath: null, + }, + }, +]; + +const critic: ConstraintCriticAgent = { + critique: async () => ({ passes: true, hints: [] }), +}; + +const refiner: RefinerAgent = { + refine: async (input) => input.prior, +}; + +const solver: SolverAgent = { + solve: async (candidate) => ({ + derived_answer: candidate.expected_answer, + trace: "독립 풀이", + confidence: "high", + }), +}; + +const mathEngine: MathEngineClient = { + health: async () => ({ status: "ok", engine: "sympy" }), + solve: async () => ({ solutions: ["2", "3"] }), + verify: async () => ({ equivalent: true, diff: "0" }), + simplify: async ({ expr }) => ({ simplified: expr }), + differentiate: async () => ({ derivative: "" }), + limit: async () => ({ limit: "" }), +}; + +const rag: RagClient = { search: async () => refs }; + +const prompts: PromptLoader = { + load: async (id) => { + throw new Error(`prompt ${id} is not used in this test`); + }, +}; + +const strategies: StrategyLoader = { + load: async () => strategy, + loadAll: async () => [strategy], +}; + +async function collectEvents( + generator: GeneratorAgent, + deterministicFallback: "off" | "last-resort", + maxRetries: number, +): Promise { + const events: ProgressEvent[] = []; + for await (const event of runVerificationWorkflow( + { + rag, + mathEngine, + prompts, + strategies, + intentModel: new MockLanguageModelV1(), + generator, + critic, + refiner, + solver, + }, + request, + { deterministicFallback, maxRetries, perStepTimeoutMs: 1_000 }, + )) { + events.push(event); + } + return events; +} + +describe("runVerificationWorkflow when generation itself fails", () => { + it("degrades to a failed gate, retries, then falls back to the deterministic template (last-resort)", async () => { + const generate = vi.fn(async () => { + throw new Error("LLM provider unreachable"); + }); + + const events = await collectEvents({ generate }, "last-resort", 2); + + expect(generate).toHaveBeenCalledTimes(2); + + const retries = events.filter((event) => event.type === "retry"); + expect(retries).toHaveLength(1); + expect(retries[0]).toMatchObject({ attempt: 2, max_attempts: 2 }); + + const result = events.find((event) => event.type === "result"); + expect(result).toBeDefined(); + if (result?.type !== "result") throw new Error("expected result event"); + const candidate = result.candidates[0]; + expect(candidate?.problem.generation_metadata.model).toBe("deterministic-topic-generator"); + expect(candidate?.verification.overall).toBe("rejected"); + expect(candidate?.verification.gates.map((gate) => [gate.step, gate.status])).toEqual([ + ["rag", "passed"], + ["intent", "passed"], + ["generate", "failed"], + ["sympy_verify", "skipped"], + ["re_solve", "skipped"], + ["objective_map", "skipped"], + ]); + }); + + it("emits a generation_failed error instead of a result when no fallback exists (off)", async () => { + const generate = vi.fn(async () => { + throw new Error("LLM provider unreachable"); + }); + + const events = await collectEvents({ generate }, "off", 2); + + expect(generate).toHaveBeenCalledTimes(2); + expect(events.find((event) => event.type === "result")).toBeUndefined(); + const error = events.find((event) => event.type === "error"); + expect(error).toMatchObject({ + type: "error", + stage: "generate", + code: "generation_failed", + recoverable: false, + }); + }); + + it("does not emit preview events for failed generations", async () => { + const generate = vi.fn(async () => { + throw new Error("LLM provider unreachable"); + }); + + const events = await collectEvents({ generate }, "off", 1); + + expect(events.filter((event) => event.type === "preview")).toHaveLength(0); + }); +}); diff --git a/packages/web/README.md b/packages/web/README.md index d22d107..82e1001 100644 --- a/packages/web/README.md +++ b/packages/web/README.md @@ -65,8 +65,12 @@ npx @google/design.md lint DESIGN.md 현재 구현은 (A) `@microsoft/fetch-event-source` 채택. `useChat` (Vercel AI SDK) 은 AI SDK 의 `data-*` 메시지 스트림 프로토콜 전용이라 임의 SSE 이벤트 이름 (`step_started`, `pipeline_completed`) 과 호환 안 됨. 이벤트 종류 (`docs/specs/architecture.md` D-6): -- `step` — `{ index: 1-6, name: string, status: 'started' | 'completed' | 'failed' }` -- `preview` — `{ latex: string }` (중간 후보 LaTeX 미리보기) +- `step` — `{ index: 1-6, name: string, status: 'started' | 'completed' | 'failed' | 'unverified', summary?: string }` + - `summary` 는 성공 시에도 단계 서사가 온다 (예: `후보 생성 (gpt-5.5) · Critic 1라운드 · 3.2초`) + - `unverified` = 결정론 검증 불가 (실패 아님; 독립 재풀이로만 확인) +- `preview` — `{ latex: string }` (3/6 완료 직후 후보 문제 미리보기) +- `attempt` — `{ attempt, max_attempts, reason }` (검증 실패로 재생성 시작; FE 는 3~6 단계 리셋) +- `runs` — `{ completed, total }` (병렬 생성 런 집계; count > 1 일 때만) - `result` — `GeneratedProblem[]` (검증 통과한 문항 묶음) - `error` — `{ stage: string, message: string }` diff --git a/packages/web/app/app/new/verify/view.tsx b/packages/web/app/app/new/verify/view.tsx index e3694c4..92ba3f6 100644 --- a/packages/web/app/app/new/verify/view.tsx +++ b/packages/web/app/app/new/verify/view.tsx @@ -34,8 +34,14 @@ const STATUS_ICON: Record = { active: "", pass: "✓", fail: "✗", + unverified: "△", }; +/* unverified 는 디자인 토큰의 warn 변형을 재사용한다 (실패 아님 — 판정 불가). */ +function rowClass(status: Step["status"]): string { + return status === "unverified" ? "warn" : status; +} + function StepIcon({ step }: { step: Step }) { if (step.status === "active") { return