feat: real generation agent transition (Phase 0-3 + 4-1/4-2/4-4/4-5)#19
Merged
Conversation
Phase 0: DETERMINISTIC_FALLBACK env (off|last-resort|first), SSE wire exposes gates+generation_model+refined_by, scripts/eval-generation.ts harness, baseline doc (pure-LLM 0/8 verified). Phase 1: template demoted to last-resort, per-attempt temperature schedule, generateObject schema-failure retry, gate-specific retry hints + counterexample, intent fallback no longer blanket must_preserve:true, SOLVER_MODEL env, web 템플릿 폴백 badge. Phase 2: math-engine multiprocessing 5s hard-timeout, equals()+numeric-sampling equivalence, /verify-answer (number|expression|equation|solution_set|inequality), lark LaTeX parsing, async endpoints; agent 3-state gates (passed|failed|skipped|unverified), honest sympy/multiple-choice verification (no more non-empty=>passed), re-solve mismatch->retry/reject, equation-extractor unicode+LaTeX, domain.md I-V4 update + web 기호검증불가 badge. Phase 3-1: techniques_used deterministic objective-gate comparison. Phase 4-2: silent catches record skipped_reason. Phase 4-4: env-gated LLM_E2E integration test. Verified: agent typecheck clean, 236 unit + 5 integration tests pass; web typecheck clean; 29 pytest pass, ruff clean.
Covers cut tasks 4-1/4-3/4-5/4-6, 3-2/3-3, full 120-run eval, plus non-obvious gotchas (answer normalization, sameAnswer shortcut, unverified->completed wire mapping, acceptance/retry budget, py3.11 multiprocessing, lark backend).
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
20 issues found across 64 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/agent/src/workflows/verification-workflow.ts">
<violation number="1" location="packages/agent/src/workflows/verification-workflow.ts:244">
P1: Last-resort fallback returns a different problem without re-verifying it, but reuses prior gate results. This can emit inconsistent result payloads where verification evidence does not belong to the returned problem.</violation>
</file>
<file name="packages/agent/src/schemas/wire-format.schema.ts">
<violation number="1" location="packages/agent/src/schemas/wire-format.schema.ts:84">
P2: Wire schema does not enforce the 6-gate invariant. Require `.length(6)` to keep FE payload validation aligned with verification invariants.</violation>
</file>
<file name="packages/agent/src/policies/acceptance-policy.ts">
<violation number="1" location="packages/agent/src/policies/acceptance-policy.ts:17">
P1: Configurable maxAttempts is inconsistent with invariant I-V5 fixed at 3, causing runtime invariant failures when maxAttempts > 3. Align policy limit with invariant or update invariant to use same configured cap.</violation>
</file>
<file name="packages/agent/src/schemas/source-problem.schema.ts">
<violation number="1" location="packages/agent/src/schemas/source-problem.schema.ts:62">
P2: I-S2 condition is over-restricted; `achievement_standard` is required even when `topic_code` is absent. This can reject valid normalized source rows.</violation>
<violation number="2" location="packages/agent/src/schemas/source-problem.schema.ts:65">
P1: I-S3 regex is stricter than spec and will fail valid LaTeX questions. It should only block non-normalized fraction commands.</violation>
<violation number="3" location="packages/agent/src/schemas/source-problem.schema.ts:68">
P1: I-S4 check incorrectly forbids all null explanations. This will fail legitimate dataset-30 rows that are explicitly allowed by domain invariants.</violation>
<violation number="4" location="packages/agent/src/schemas/source-problem.schema.ts:71">
P2: I-S5 condition is too broad; high-school rows can legally have null grade. Current check rejects valid source problems.</violation>
</file>
<file name="packages/agent/src/tools/latex-formatter.ts">
<violation number="1" location="packages/agent/src/tools/latex-formatter.ts:20">
P1: `\pm` expansion is algebraically incorrect because it removes the left operand. This can produce wrong candidate roots/answers for SymPy verification.</violation>
<violation number="2" location="packages/agent/src/tools/latex-formatter.ts:33">
P2: Display-math segments `\[...\]` are not extracted from prose. This leaves delimiter artifacts in normalized expressions and can break downstream parsing.</violation>
<violation number="3" location="packages/agent/src/tools/latex-formatter.ts:47">
P2: Equation extraction only matches `=` and drops inequality-only statements from prose. This can pass non-math text to SymPy normalization.</violation>
</file>
<file name="packages/agent/src/config/env.ts">
<violation number="1" location="packages/agent/src/config/env.ts:40">
P3: JSDoc for `last-resort` mode says "currently behaves like `first`" but the implementation treats it differently: generation skips template first (unlike `first`) and final-result fallback is already gated on `last-resort`. Stale docs risk misinforming configuration decisions.</violation>
</file>
<file name="docs/specs/domain.md">
<violation number="1" location="docs/specs/domain.md:171">
P3: Stale name `independent_resolve` conflicts with `re_solve` in same invariant — gate[4] step is `re_solve` per GateResult.</violation>
</file>
<file name="packages/agent/src/tools/answer-equivalence.ts">
<violation number="1" location="packages/agent/src/tools/answer-equivalence.ts:428">
P1: Choice-label regex misclassifies decimal answers as choice labels. This can pick the wrong expected choice index and produce incorrect equivalence/reject decisions.</violation>
</file>
<file name="packages/agent/src/schemas/verification.schema.ts">
<violation number="1" location="packages/agent/src/schemas/verification.schema.ts:24">
P2: New `unverified` gate status is not handled in SSE step-status mapping, causing unverified checks to be reported as completed. This mislabels verification progress/UX and hides non-decidable outcomes during the run.</violation>
</file>
<file name="scripts/eval-generation.ts">
<violation number="1" location="scripts/eval-generation.ts:474">
P2: `unverified` gate statuses are silently discarded during SSE parsing. This skews gate diagnostics and failure attribution in the eval report.</violation>
</file>
<file name="packages/agent/src/schemas/generated-problem.schema.ts">
<violation number="1" location="packages/agent/src/schemas/generated-problem.schema.ts:61">
P1: I-G2 check is applied to conceptual candidates too, causing valid conceptual transforms to be rejected when required techniques differ.</violation>
<violation number="2" location="packages/agent/src/schemas/generated-problem.schema.ts:80">
P2: I-G3 comparison is one-way subset, not equality; conceptual candidates can add extra must_preserve dimensions without failing invariant.</violation>
</file>
<file name="packages/web/hooks/use-verification-stream.ts">
<violation number="1" location="packages/web/hooks/use-verification-stream.ts:212">
P2: Gate status is not validated against the 4-state contract, so invalid payloads are treated as valid results. This can hide the “기호 검증 불가” signal instead of failing the stream payload.</violation>
</file>
<file name="packages/agent/src/steps/intent-extraction.ts">
<violation number="1" location="packages/agent/src/steps/intent-extraction.ts:72">
P3: `dimensions_source` provenance is inaccurate when strategy is null but request dimensions are provided. This weakens observability for fallback quality analysis.</violation>
<violation number="2" location="packages/agent/src/steps/intent-extraction.ts:124">
P2: Fallback intent incorrectly preserves only the first requested dimension. This can make objective mapping/verification ignore user-selected preserve constraints beyond index 0.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| } | ||
|
|
||
| const problem = withDeterministicGeneratorMarker(fallback); | ||
| const verification = { ...input.verification, candidate_id: problem.candidate_id }; |
There was a problem hiding this comment.
P1: Last-resort fallback returns a different problem without re-verifying it, but reuses prior gate results. This can emit inconsistent result payloads where verification evidence does not belong to the returned problem.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/agent/src/workflows/verification-workflow.ts, line 244:
<comment>Last-resort fallback returns a different problem without re-verifying it, but reuses prior gate results. This can emit inconsistent result payloads where verification evidence does not belong to the returned problem.</comment>
<file context>
@@ -183,6 +214,55 @@ export async function* runVerificationWorkflow(
+ }
+
+ const problem = withDeterministicGeneratorMarker(fallback);
+ const verification = { ...input.verification, candidate_id: problem.candidate_id };
+ assertVerificationInvariants(verification);
+ return { problem, verification };
</file context>
| return { | ||
| decide(gates, attemptCount) { | ||
| if (attemptCount > 3) return "rejected"; | ||
| if (attemptCount > maxAttempts) return "rejected"; |
There was a problem hiding this comment.
P1: Configurable maxAttempts is inconsistent with invariant I-V5 fixed at 3, causing runtime invariant failures when maxAttempts > 3. Align policy limit with invariant or update invariant to use same configured cap.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/agent/src/policies/acceptance-policy.ts, line 17:
<comment>Configurable maxAttempts is inconsistent with invariant I-V5 fixed at 3, causing runtime invariant failures when maxAttempts > 3. Align policy limit with invariant or update invariant to use same configured cap.</comment>
<file context>
@@ -6,20 +6,33 @@ export interface AcceptancePolicy {
return {
decide(gates, attemptCount) {
- if (attemptCount > 3) return "rejected";
+ if (attemptCount > maxAttempts) return "rejected";
const byStep = new Map(gates.map((gate) => [gate.step, gate]));
</file context>
| if (/\\(?:d?frac|sqrt|cdot|left|right|pi|pm)/u.test(problem.question_text)) { | ||
| throw new Error(`I-S3 violated: source problem ${problem.item_id} question_text must be normalized plain text`); | ||
| } | ||
| if (problem.explanation_text === null || problem.explanation_text.trim().length === 0) { |
There was a problem hiding this comment.
P1: I-S4 check incorrectly forbids all null explanations. This will fail legitimate dataset-30 rows that are explicitly allowed by domain invariants.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/agent/src/schemas/source-problem.schema.ts, line 68:
<comment>I-S4 check incorrectly forbids all null explanations. This will fail legitimate dataset-30 rows that are explicitly allowed by domain invariants.</comment>
<file context>
@@ -54,3 +54,21 @@ export const SourceProblemSchema = z.object({
+ if (/\\(?:d?frac|sqrt|cdot|left|right|pi|pm)/u.test(problem.question_text)) {
+ throw new Error(`I-S3 violated: source problem ${problem.item_id} question_text must be normalized plain text`);
+ }
+ if (problem.explanation_text === null || problem.explanation_text.trim().length === 0) {
+ throw new Error(`I-S4 violated: source problem ${problem.item_id} needs explanation_text`);
+ }
</file context>
Suggested change
| if (problem.explanation_text === null || problem.explanation_text.trim().length === 0) { | |
| if ((problem.explanation_text === null && problem.source_dataset !== "30") || (problem.explanation_text !== null && problem.explanation_text.trim().length === 0)) { |
| if (problem.achievement_standard === null || problem.achievement_standard.trim().length === 0) { | ||
| throw new Error(`I-S2 violated: source problem ${problem.item_id} needs achievement_standard`); | ||
| } | ||
| if (/\\(?:d?frac|sqrt|cdot|left|right|pi|pm)/u.test(problem.question_text)) { |
There was a problem hiding this comment.
P1: I-S3 regex is stricter than spec and will fail valid LaTeX questions. It should only block non-normalized fraction commands.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/agent/src/schemas/source-problem.schema.ts, line 65:
<comment>I-S3 regex is stricter than spec and will fail valid LaTeX questions. It should only block non-normalized fraction commands.</comment>
<file context>
@@ -54,3 +54,21 @@ export const SourceProblemSchema = z.object({
+ if (problem.achievement_standard === null || problem.achievement_standard.trim().length === 0) {
+ throw new Error(`I-S2 violated: source problem ${problem.item_id} needs achievement_standard`);
+ }
+ if (/\\(?:d?frac|sqrt|cdot|left|right|pi|pm)/u.test(problem.question_text)) {
+ throw new Error(`I-S3 violated: source problem ${problem.item_id} question_text must be normalized plain text`);
+ }
</file context>
| .replace(/\\leq?|≤/g, "<=") | ||
| .replace(/\\geq?|≥/g, ">=") | ||
| .replace(/\\pi/g, "pi") | ||
| .replace(/\\pm\s*([^\s,]+)/g, "-$1, $1") |
There was a problem hiding this comment.
P1: \pm expansion is algebraically incorrect because it removes the left operand. This can produce wrong candidate roots/answers for SymPy verification.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/agent/src/tools/latex-formatter.ts, line 20:
<comment>`\pm` expansion is algebraically incorrect because it removes the left operand. This can produce wrong candidate roots/answers for SymPy verification.</comment>
<file context>
@@ -4,6 +4,50 @@ export function formatLatex(value: string): string {
+ .replace(/\\leq?|≤/g, "<=")
+ .replace(/\\geq?|≥/g, ">=")
+ .replace(/\\pi/g, "pi")
+ .replace(/\\pm\s*([^\s,]+)/g, "-$1, $1")
+ .replace(/\^/g, "**");
+ expression = extractEquationText(expression)
</file context>
Suggested change
| .replace(/\\pm\s*([^\s,]+)/g, "-$1, $1") | |
| .replace(/([^,\s]+)\s*\\pm\s*([^\s,]+)/g, "$1-$2, $1+$2") |
| const step = asString(o.step); | ||
| const status = asString(o.status); | ||
| if (step === null || status === null) return null; | ||
| return { step, status }; |
There was a problem hiding this comment.
P2: Gate status is not validated against the 4-state contract, so invalid payloads are treated as valid results. This can hide the “기호 검증 불가” signal instead of failing the stream payload.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/web/hooks/use-verification-stream.ts, line 212:
<comment>Gate status is not validated against the 4-state contract, so invalid payloads are treated as valid results. This can hide the “기호 검증 불가” signal instead of failing the stream payload.</comment>
<file context>
@@ -188,6 +203,15 @@ function parsePreview(raw: unknown): string | null {
+ const step = asString(o.step);
+ const status = asString(o.status);
+ if (step === null || status === null) return null;
+ return { step, status };
+}
+
</file context>
| return descriptions.map((description, index) => ({ | ||
| id: String.fromCharCode(65 + index), | ||
| description, | ||
| must_preserve: index === 0, |
There was a problem hiding this comment.
P2: Fallback intent incorrectly preserves only the first requested dimension. This can make objective mapping/verification ignore user-selected preserve constraints beyond index 0.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/agent/src/steps/intent-extraction.ts, line 124:
<comment>Fallback intent incorrectly preserves only the first requested dimension. This can make objective mapping/verification ignore user-selected preserve constraints beyond index 0.</comment>
<file context>
@@ -116,3 +112,15 @@ function buildSeedIntent(
+ return descriptions.map((description, index) => ({
+ id: String.fromCharCode(65 + index),
+ description,
+ must_preserve: index === 0,
+ }));
+}
</file context>
Suggested change
| must_preserve: index === 0, | |
| must_preserve: true, |
|
|
||
| /** `first` = template short-circuits LLM when refs exist (current behavior). | ||
| * `off` = always go through LLM generator path; never substitute template. | ||
| * `last-resort` = placeholder, currently behaves like `first` (see TODO 1-1a). */ |
There was a problem hiding this comment.
P3: JSDoc for last-resort mode says "currently behaves like first" but the implementation treats it differently: generation skips template first (unlike first) and final-result fallback is already gated on last-resort. Stale docs risk misinforming configuration decisions.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/agent/src/config/env.ts, line 40:
<comment>JSDoc for `last-resort` mode says "currently behaves like `first`" but the implementation treats it differently: generation skips template first (unlike `first`) and final-result fallback is already gated on `last-resort`. Stale docs risk misinforming configuration decisions.</comment>
<file context>
@@ -29,10 +34,19 @@ export const EnvSchema = z.object({
+
+ /** `first` = template short-circuits LLM when refs exist (current behavior).
+ * `off` = always go through LLM generator path; never substitute template.
+ * `last-resort` = placeholder, currently behaves like `first` (see TODO 1-1a). */
+ DETERMINISTIC_FALLBACK: z
+ .enum(["off", "last-resort", "first"])
</file context>
| - (I-V2) `overall == "verified"`는 `gates[3] (sympy_verify).status == "passed"` **and** `gates[5] (objective_map).status == "passed"` 일 때만 가능 (결정론 게이트). | ||
| - (I-V3) `gates[1] (intent)`와 `gates[2] (generate)`가 passed라도 `gates[3] (sympy_verify)`가 failed면 `overall != "verified"` (D-1: LLM은 정답 판단 X). | ||
| - (I-V4) `overall == "warning"`은 SymPy는 통과했으나 `independent_resolve`(gates[4])가 불일치할 때만 사용 — 사용자에게 *주의* 라벨로 노출. | ||
| - (I-V4) `overall == "warning"`은 SymPy가 `passed` 또는 `unverified`인 상태에서 `independent_resolve`(gates[4])가 불일치했거나, 비결정 가능(non-decidable) 게이트가 `unverified`일 때 사용 — 사용자에게 *주의* 라벨로 노출. 단, 재시도 상한의 최종 attempt에서도 `re_solve` 불일치가 지속되면 `warning`이 아니라 `rejected`다. |
There was a problem hiding this comment.
P3: Stale name independent_resolve conflicts with re_solve in same invariant — gate[4] step is re_solve per GateResult.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At docs/specs/domain.md, line 171:
<comment>Stale name `independent_resolve` conflicts with `re_solve` in same invariant — gate[4] step is `re_solve` per GateResult.</comment>
<file context>
@@ -168,7 +168,7 @@ type HumanFailureReason = {
- (I-V2) `overall == "verified"`는 `gates[3] (sympy_verify).status == "passed"` **and** `gates[5] (objective_map).status == "passed"` 일 때만 가능 (결정론 게이트).
- (I-V3) `gates[1] (intent)`와 `gates[2] (generate)`가 passed라도 `gates[3] (sympy_verify)`가 failed면 `overall != "verified"` (D-1: LLM은 정답 판단 X).
-- (I-V4) `overall == "warning"`은 SymPy는 통과했으나 `independent_resolve`(gates[4])가 불일치할 때만 사용 — 사용자에게 *주의* 라벨로 노출.
+- (I-V4) `overall == "warning"`은 SymPy가 `passed` 또는 `unverified`인 상태에서 `independent_resolve`(gates[4])가 불일치했거나, 비결정 가능(non-decidable) 게이트가 `unverified`일 때 사용 — 사용자에게 *주의* 라벨로 노출. 단, 재시도 상한의 최종 attempt에서도 `re_solve` 불일치가 지속되면 `warning`이 아니라 `rejected`다.
- (I-V5) `attempt_count > 3`이면 강제 `overall == "rejected"` (Q-5 잠정 정책, D-9에서 확정 예정).
</file context>
Suggested change
| - (I-V4) `overall == "warning"`은 SymPy가 `passed` 또는 `unverified`인 상태에서 `independent_resolve`(gates[4])가 불일치했거나, 비결정 가능(non-decidable) 게이트가 `unverified`일 때 사용 — 사용자에게 *주의* 라벨로 노출. 단, 재시도 상한의 최종 attempt에서도 `re_solve` 불일치가 지속되면 `warning`이 아니라 `rejected`다. | |
| - (I-V4) `overall == "warning"`은 SymPy가 `passed` 또는 `unverified`인 상태에서 `re_solve`(gates[4])가 불일치했거나, 비결정 가능(non-decidable) 게이트가 `unverified`일 때 사용 — 사용자에게 *주의* 라벨로 노출. 단, 재시도 상한의 최종 attempt에서도 `re_solve` 불일치가 지속되면 `warning`이 아니라 `rejected`다. |
| } catch (err) { | ||
| const fallback = buildSeedIntent(input.request, input.strategy, input.refs); | ||
| assertIntentInvariants(fallback); | ||
| const dimensionsSource = input.strategy === null ? "guessed" : "strategy"; |
There was a problem hiding this comment.
P3: dimensions_source provenance is inaccurate when strategy is null but request dimensions are provided. This weakens observability for fallback quality analysis.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/agent/src/steps/intent-extraction.ts, line 72:
<comment>`dimensions_source` provenance is inaccurate when strategy is null but request dimensions are provided. This weakens observability for fallback quality analysis.</comment>
<file context>
@@ -69,6 +69,7 @@ export async function extractIntent(
} catch (err) {
const fallback = buildSeedIntent(input.request, input.strategy, input.refs);
assertIntentInvariants(fallback);
+ const dimensionsSource = input.strategy === null ? "guessed" : "strategy";
return {
data: fallback,
</file context>
Suggested change
| const dimensionsSource = input.strategy === null ? "guessed" : "strategy"; | |
| const dimensionsSource = input.strategy !== null ? "strategy" : input.request.dims.length > 0 ? "request" : "guessed"; |
4-1: withTimeout aborts an AbortController on timeout and threads the signal into every generateObject call (generator/refiner/critic/solver/intent/objective nuance), so slow LLM calls are cancelled instead of running to the wall clock. math-engine calls stay bounded by the client's existing internal 10s controller. Adds withTimeout cancellation tests. 4-5: drop the page-level generateMockResults seed on S5/S6; result/export views now source verified problems only from the SSE sessionStorage rehydrate. Relocate ResultProblem type to result/types.ts, delete mock.ts, add an S5 empty state. Verified S5/S6 in a real browser.
There was a problem hiding this comment.
2 issues found across 72 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/agent/src/tools/answer-equivalence.ts">
<violation number="1" location="packages/agent/src/tools/answer-equivalence.ts:65">
P2: New answer-equivalence decision flow is duplicated across two functions, creating avoidable divergence risk. Extract shared comparison logic and pass context-specific messages/inputs.</violation>
</file>
<file name="packages/agent/src/steps/objective-mapping.ts">
<violation number="1" location="packages/agent/src/steps/objective-mapping.ts:202">
P1: Technique gate for non-conceptual mode is stricter than schema semantics; it requires all required techniques instead of at least one. This can incorrectly reject otherwise valid generations.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
Comment on lines
+202
to
+206
| } else if (missingRequired.length > 0) { | ||
| failures.push({ | ||
| code: "technique_mismatch", | ||
| message: `Candidate techniques ${formatTechniqueList(used)} do not cover required techniques ${formatTechniqueList(missingRequired)}`, | ||
| }); |
There was a problem hiding this comment.
P1: Technique gate for non-conceptual mode is stricter than schema semantics; it requires all required techniques instead of at least one. This can incorrectly reject otherwise valid generations.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/agent/src/steps/objective-mapping.ts, line 202:
<comment>Technique gate for non-conceptual mode is stricter than schema semantics; it requires all required techniques instead of at least one. This can incorrectly reject otherwise valid generations.</comment>
<file context>
@@ -140,7 +169,70 @@ function evaluateDeterministically(input: ObjectiveMappingInput): Array<{ code:
+ message: `Candidate techniques ${formatTechniqueList(used)} do not overlap required or related techniques ${formatTechniqueList(related)}`,
+ });
+ }
+ } else if (missingRequired.length > 0) {
+ failures.push({
+ code: "technique_mismatch",
</file context>
Suggested change
| } else if (missingRequired.length > 0) { | |
| failures.push({ | |
| code: "technique_mismatch", | |
| message: `Candidate techniques ${formatTechniqueList(used)} do not cover required techniques ${formatTechniqueList(missingRequired)}`, | |
| }); | |
| } else if (required.length > 0 && missingRequired.length === required.length) { | |
| failures.push({ | |
| code: "technique_mismatch", | |
| message: `Candidate techniques ${formatTechniqueList(used)} do not overlap required techniques ${formatTechniqueList(required)}`, | |
| }); |
| return { status: "undecidable", reason: withSkippedReasons("math-engine could not compare the answers", debug) }; | ||
| } | ||
|
|
||
| export async function decideAnswerMatchesSolutions( |
There was a problem hiding this comment.
P2: New answer-equivalence decision flow is duplicated across two functions, creating avoidable divergence risk. Extract shared comparison logic and pass context-specific messages/inputs.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/agent/src/tools/answer-equivalence.ts, line 65:
<comment>New answer-equivalence decision flow is duplicated across two functions, creating avoidable divergence risk. Extract shared comparison logic and pass context-specific messages/inputs.</comment>
<file context>
@@ -3,10 +3,123 @@ import type { MathEngineClient } from "./math-engine-client.js";
+ return { status: "undecidable", reason: withSkippedReasons("math-engine could not compare the answers", debug) };
+}
+
+export async function decideAnswerMatchesSolutions(
+ mathEngine: MathEngineClient,
+ answer: string,
</file context>
smilebank7
changed the title
Contributor
Author
추가: Phase 4-1 + 4-5 (commit
|
The 30s per-step timeout killed the generate step in last-resort/off mode: real generation runs generator + 2 critic + 2 refiner LLM calls (~90s observed), and a single intent call alone takes ~36s on a reasoning model. With the 4-1 abort wired in, that timeout cancelled the in-flight LLM call and surfaced a generic orchestrator error, which is why the demo silently fell back to the deterministic template. Raise the default per-step timeout to 60s and give the generate step a budget scaled to its max LLM-call count (perStep x (1 + 2*maxCriticRounds)). Verified: last-resort now yields a real gpt-5.5-generated, SymPy-verified problem.
The workflow produces one problem per run; the demo expects multiple (3문항). Fan out at the route/SSE layer: run min(count,3) workflows concurrently, drive the user-visible step bar from the first run only, drain the rest silently, and merge their result candidates into one result event. Orchestrator and its 238 tests are untouched. A failed parallel run contributes no problem instead of aborting the others. Verified live: count=5 -> 3 distinct gpt-5.5 verified problems, one 12-event step bar, 169s.
There was a problem hiding this comment.
2 issues found across 3 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/agent/src/server/routes/generate.ts">
<violation number="1" location="packages/agent/src/server/routes/generate.ts:23">
P2: Route hard-caps `request.count` to 3, violating the API contract and silently returning fewer problems than requested.</violation>
</file>
<file name="packages/agent/src/server/sse/progress-stream.ts">
<violation number="1" location="packages/agent/src/server/sse/progress-stream.ts:73">
P1: Forwarding driver error events in parallel mode causes the frontend to abort before merged results arrive. A single failed driver run can hide successful candidates from other runs.</violation>
</file>
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
| collected.push(event); | ||
| continue; | ||
| } | ||
| if (event.type === "error") forwardedError = true; |
There was a problem hiding this comment.
P1: Forwarding driver error events in parallel mode causes the frontend to abort before merged results arrive. A single failed driver run can hide successful candidates from other runs.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/agent/src/server/sse/progress-stream.ts, line 73:
<comment>Forwarding driver error events in parallel mode causes the frontend to abort before merged results arrive. A single failed driver run can hide successful candidates from other runs.</comment>
<file context>
@@ -32,3 +32,66 @@ export async function pipeProgressToSse(
+ collected.push(event);
+ continue;
+ }
+ if (event.type === "error") forwardedError = true;
+ await writeWire(event);
+ }
</file context>
| zValidator("json", GenerateRequestSchema), | ||
| (c) => { | ||
| const request = c.req.valid("json"); | ||
| const count = Math.min(Math.max(request.count, 1), MAX_PARALLEL_GENERATIONS); |
There was a problem hiding this comment.
P2: Route hard-caps request.count to 3, violating the API contract and silently returning fewer problems than requested.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/agent/src/server/routes/generate.ts, line 23:
<comment>Route hard-caps `request.count` to 3, violating the API contract and silently returning fewer problems than requested.</comment>
<file context>
@@ -18,8 +20,12 @@ export function createGenerateRoute(deps: VerificationWorkflowDeps, options?: Ru
zValidator("json", GenerateRequestSchema),
(c) => {
const request = c.req.valid("json");
+ const count = Math.min(Math.max(request.count, 1), MAX_PARALLEL_GENERATIONS);
return streamSSE(c, async (stream) => {
- await pipeProgressToSse(stream, runVerificationWorkflow(deps, request, options));
</file context>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
목표
데모 12개 단원에서 LLM이 실제로 문제를 생성하고, 검증 게이트가 정직하게 통과하며, 품질을 수치로 증명. (플랜:
docs/specs/remediation-plan.md)핵심 발견 (Phase 0 기준선)
DETERMINISTIC_FALLBACK=off로 측정한 순수 LLM 성공률 = 0/8 (0%). 기존 데모는 결정론 템플릿이 LLM을 가려서 통과하던 것 —docs/eval/baseline-2026-06-10.md. 이 PR는 그 0%를 움직이기 위한 기반 + 검증 정직화.변경 요약
DETERMINISTIC_FALLBACK플래그, SSE 와이어에 gate/overall/model/refined_by 노출,scripts/eval-generation.ts평가 하니스, 기준선 측정.generateObject스키마실패 재시도, 게이트별 힌트+반례, intent fallback 수정(blanketmust_preserve:true제거),SOLVER_MODEL.equals()+수치샘플링,/verify-answer(5종 답유형), lark LaTeX, async. agent: 게이트 3-상태(unverified추가), 정직한 sympy/객관식 검증("비어있으면 통과" 0개), 재풀이 불일치→재시도/최종 reject, 유니코드+LaTeX 추출,domain.mdI-V4 갱신, 웹 "기호 검증 불가" 배지.techniques_used결정론 비교(구조 동형 기법 커버리지 강제).skipped_reason기록,LLM_E2E게이트 통합 테스트.검증
범위 밖 (핸드오프)
시간상 단축안으로 진행 — 남은 작업(4-1 AbortController, 4-5 웹 실연동, 3-2/3-3, 4-3, 4-6, 풀 120런 eval)과 디버깅 gotcha는
docs/specs/remediation-handoff.md에 정리.주의
main직접 변경 없음. 사용자 작업이 아닌 untracked 파일 2개(packages/agent/scripts/{transform_real_corpus.py,verify_rag.ts})와reports/아티팩트는 커밋 제외.Summary by cubic
Switches the demo to real LLM problem generation with honest, 3‑state verification and stronger math checks. Adds abortable per‑step timeouts (60s default; scaled budget for generate), wires S5/S6 to live SSE results, and runs multi‑problem requests in parallel (cap 3).
New Features
DETERMINISTIC_FALLBACK(off|last-resort|first); per‑attempt temperature; schema‑failure retry with hint; counterexample feedback.SOLVER_MODELbuilds a separate solver to decorrelate re‑solve from generation.PER_STEP_TIMEOUT_MS60s; generate step budget scaled; threadsAbortSignalinto generator/refiner/critic/solver/intent/nuance).passed|failed|unverified. Re‑solve mismatch triggers retry, then reject on budget.techniques_usedfor structural coverage; safer normalization and explicit skipped reasons./verify-answerfor number/expression/equation/solution_set/inequality. LaTeX parsing withlark. Async endpoints.gates,overall,attempt_count,generation_model,refined_by.min(count, 3)parallel runs; step bar streams from the first run; merges result candidates; a failed run is dropped without aborting others.result/types.ts.scripts/eval-generation.tsbatch harness andLLM_E2Egated integration test; timeout‑cancellation coverage; remediation handoff updated to mark 4‑1 and 4‑5 done.Dependencies
packages/math-engine: addlark; tuneuvicornrunner (workers/reload)..envcan optionally includeSOLVER_MODEL.Written for commit 112c802. Summary will update on new commits.