Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions packages/agent/prompts/problem-classifier.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
---
id: problem-classifier
version: 0.1.0
model: gpt-4o
temperature: 0.1
max_tokens: 800
schema: LlmClassificationSchema
variables:
- catalog
- questionText
- choices
owner: 비할당
updated: 2026-06-11
---

# Role

너는 한국 수학 교육과정 단원 분류기다. 주어진 문제를 아래 **단원 카탈로그**의 코드 중 하나로 분류하라.

# 단원 카탈로그 (이 코드들 중에서만 고른다)

{{catalog}}

# 분류할 문제

발문: {{questionText}}

보기: {{choices}}

# 출력 규칙

- `topic_code`: 위 카탈로그에 **존재하는 코드**만 사용한다. 카탈로그 밖의 코드를 만들지 말 것.
- `topic_name`: 그 코드의 단원 이름(카탈로그 표기 그대로).
- `generation_kind`: 문제의 **풀이 종류**를 8개 중에서 고른다 — `equation`(방정식 해), `inequality`(부등식), `system`(연립방정식), `expression`(식 계산·전개·인수분해·근호), `function`(함수·그래프), `geometry`(도형·삼각비), `probability`(경우의 수·확률), `statistics`(통계). 단원이 애매해도 이 종류는 **문제 형태로** 판단한다.
- `problem_type`: `objective`(객관식·보기 있음) / `short_answer`(짧은 답) / `essay`(서술형) / `subjective`(주관식) 중 하나.
- `difficulty`: `easy` / `medium` / `hard` 중 하나. 단원 표준 난이도 기준.
- `confidence`: 분류 확신(0~1). 여러 단원에 걸치거나 교육과정 밖으로 보이면 낮춘다.
- `alternatives`: 헷갈리는 다른 후보 단원(카탈로그 코드)을 최대 2개. 없으면 빈 배열.

# 주의

- 문제를 풀지 말고 **무엇을 평가하는 단원인지**만 판단한다.
- 학년이 애매하면 발문의 핵심 개념(예: 이차방정식 → 중3, 순환소수 → 중2)으로 판단한다.
33 changes: 33 additions & 0 deletions packages/agent/prompts/problem-extractor.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
---
id: problem-extractor
version: 0.1.0
model: gpt-4o
temperature: 0.1
max_tokens: 1500
schema: ExtractionSchema
variables: []
owner: 비할당
updated: 2026-06-11
---

# Role

너는 한국 중·고등학교 수학 문제를 정확히 읽어 들이는 OCR·표기 정리 도우미다.
입력으로 **이미지 한 장** 또는 **텍스트 한 덩어리**가 주어진다. 거기서 수학 문제 **한 개**를 읽어 구조화된 JSON으로 내보내라.

# 출력 규칙

- `question_text`: 문제 발문과 풀이에 필요한 수식을 그대로 옮긴다.
- 수식은 인라인 LaTeX `$...$` 로 적는다. 예: `$x^2 - 5x + 6 = 0$`, `$\frac{2}{3}$`, `$\sqrt{12}$`.
- 객관식 보기(①~⑤ 등)는 여기 넣지 말고 `choices` 로 분리한다.
- 문제 번호("3.", "12번")·배점·출처 표시는 제거한다.
- `choices`: 객관식이면 보기 텍스트 배열(마커 포함, 예: `["① $2$", "② $3$"]`). 객관식이 아니면 `null`.
- `answer_text`: 원본에 정답이 함께 보이면 그 값을 옮기고, 없으면 `null`.
- `figure_dependent`: 그림·그래프·도형·표가 있어야만 풀 수 있고 본문 텍스트만으로 답이 결정되지 않으면 `true`. 좌표·길이·각이 본문에 모두 적혀 있어 텍스트만으로 풀리면 `false`.
- `confidence`: 읽어 들인 본문의 정확도 확신(0~1). 흐릿함·손글씨·잘림·수식 누락이 있으면 낮춘다.

# 금지

- 보이지 않는 내용을 추측해 채우지 말 것. 안 보이면 비우고 `confidence` 를 낮춰라.
- 문제를 풀거나 정답을 새로 계산하지 말 것. 원본에 적힌 것만 옮긴다.
- 여러 문제가 보이면 가장 완전한 **한 개**만 고른다.
7 changes: 7 additions & 0 deletions packages/agent/prompts/problem-generator.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ variables:
- refinementHint
- counterexample
- schemaError
- attached
owner: 비할당
updated: 2026-06-11
---
Expand Down Expand Up @@ -54,6 +55,12 @@ updated: 2026-06-11
- conceptual: 평가 차원은 보존하되 식의 형태를 바꾼다. 예: `x**2 - a = 0` 기준이면 `(x - h)**2 - a = 0` 또는 전개형 `x**2 + b*x + c = 0`처럼 다른 표현 경로를 사용한다.
- `{{request.difficulty}}` 난이도에 맞춘다. medium 이상에서는 `x**2 - a = 0`처럼 너무 직접적인 기본형만 내지 말고 한 단계 변형된 식을 낸다.

{{#if attached}}
# 첨부 문제 우선 (중요)

기준 문제는 사용자가 직접 첨부한 문제다. 아래 References(참조문항)가 첨부 문제와 다르면 **무조건 첨부 문제를 따른다**. References 는 표현·형식 힌트로만 쓰고, 풀이 구조와 평가 요소는 첨부 문제에서 가져온다. References 가 비어 있거나 동떨어져 있어도 정상이다 — 첨부 문제 하나만으로 동형 문제를 만든다.
{{/if}}

# Intent

학습 목표: {{intent.objective_description}} (`{{intent.objective_code}}`)
Expand Down
148 changes: 148 additions & 0 deletions packages/agent/src/agents/classifier-agent.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,148 @@
/** ClassifierAgent — 추출 문제를 교육과정 카탈로그 단원으로 분류한 뒤 코드로 스냅.
*
* LLM 은 카탈로그 안의 코드를 고르고, resolveClassification 이 그 결과를
* 카탈로그(curriculum-topics)에 맞춰 검증·보정한다. 코드를 못 맞추면 이름으로,
* 그것도 안 되면 확신을 크게 낮춰 사용자가 확인 화면에서 직접 고르게 한다.
*/

import { generateObject, type LanguageModel } from "ai";
import { z } from "zod";

import {
ClassificationAlternativeSchema,
DifficultySchema,
GenerationKindSchema,
ProblemTypeSchema,
findCurriculumTopic,
findCurriculumTopicByName,
formatCurriculumCatalog,
type Classification,
type Extraction,
} from "../schemas/index.js";
import type { PromptLoader } from "../tools/prompt-loader.js";

export const LlmClassificationSchema = z.object({
topic_code: z.string().min(1),
topic_name: z.string().min(1),
generation_kind: GenerationKindSchema,
problem_type: ProblemTypeSchema,
difficulty: DifficultySchema,
confidence: z.number().min(0).max(1),
alternatives: z.array(ClassificationAlternativeSchema).default([]),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The prompt instructs the LLM to return at most 2 alternatives, but the LlmClassificationSchema does not enforce this limit on the alternatives array. If the model returns more than 2 entries, the frontend could render an unbounded number of chip buttons, breaking the UI contract. Consider adding .max(2) to the Zod array definition to make the schema match the prompt instruction and protect the UI boundary.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/agent/src/agents/classifier-agent.ts, line 31:

<comment>The prompt instructs the LLM to return at most 2 alternatives, but the `LlmClassificationSchema` does not enforce this limit on the `alternatives` array. If the model returns more than 2 entries, the frontend could render an unbounded number of chip buttons, breaking the UI contract. Consider adding `.max(2)` to the Zod array definition to make the schema match the prompt instruction and protect the UI boundary.</comment>

<file context>
@@ -0,0 +1,148 @@
+  problem_type: ProblemTypeSchema,
+  difficulty: DifficultySchema,
+  confidence: z.number().min(0).max(1),
+  alternatives: z.array(ClassificationAlternativeSchema).default([]),
+});
+
</file context>

});

export type LlmClassification = z.infer<typeof LlmClassificationSchema>;

export interface ClassifierAgentInput {
extraction: Pick<Extraction, "question_text" | "choices">;
signal?: AbortSignal;
}

export interface ClassifierAgent {
classify(input: ClassifierAgentInput): Promise<Classification>;
}

export interface ClassifierAgentDeps {
model: LanguageModel;
modelId: string;
promptId: string;
prompts: PromptLoader;
}

export function createClassifierAgent(deps: ClassifierAgentDeps): ClassifierAgent {
return {
async classify(input) {
const prompt = await deps.prompts.load(deps.promptId);
const rendered = prompt.render({
catalog: formatCurriculumCatalog(),
questionText: input.extraction.question_text,
choices:
input.extraction.choices === null || input.extraction.choices.length === 0
? "(없음)"
: input.extraction.choices.join(" / "),
});
const { object } = await generateObject({
model: deps.model,
schema: LlmClassificationSchema,
mode: "json",
temperature: prompt.metadata.temperature,
prompt: rendered,
abortSignal: input.signal,
});
return resolveClassification(object);
},
};
}

/** LLM 출력의 topic_code 를 카탈로그로 스냅. 코드 일치 우선 → 이름 매칭 → 폴백(확신 강등). */
export function resolveClassification(raw: LlmClassification): Classification {
const byCode = findCurriculumTopic(raw.topic_code);
const matched = byCode ?? findCurriculumTopicByName(raw.topic_name);

const alternatives = dedupeByCode(
raw.alternatives
.map((alt) => findCurriculumTopic(alt.topic_code) ?? findCurriculumTopicByName(alt.topic_name))
.filter((topic): topic is NonNullable<typeof topic> => topic !== undefined)
.map((topic) => ({ topic_code: topic.code, topic_name: topic.name })),
);

if (matched === undefined) {
// 코드·이름 모두 카탈로그 밖. 유효한 대안이 있으면 그 첫 번째로 강등 매칭한다
// (학교급을 raw 코드 접두로만 추정하면 오분류 위험 — 대안의 실제 학교급을 쓴다).
const firstAlt = alternatives[0];
const promoted = firstAlt === undefined ? undefined : findCurriculumTopic(firstAlt.topic_code);
if (promoted !== undefined) {
return {
school_level: promoted.school_level,
grade: promoted.grade,
topic_code: promoted.code,
topic_name: promoted.name,
generation_kind: raw.generation_kind,
problem_type: raw.problem_type,
difficulty: raw.difficulty,
confidence: Math.min(raw.confidence, 0.4),
alternatives: alternatives.filter((alt) => alt.topic_code !== promoted.code),
};
}
// 대안도 없음 — 단원을 비워 사용자가 확인 화면에서 직접 고르게 한다.
return {
school_level: raw.topic_code.startsWith("10") ? "high" : "middle",
Comment thread
cubic-dev-ai[bot] marked this conversation as resolved.
grade: null,
topic_code: "",
topic_name: "",
generation_kind: raw.generation_kind,
problem_type: raw.problem_type,
difficulty: raw.difficulty,
confidence: Math.min(raw.confidence, 0.3),
alternatives,
};
}

// 코드는 못 맞췄지만 이름으로 맞춘 경우 확신을 약간 강등.
const confidence = byCode === undefined ? Math.min(raw.confidence, 0.5) : raw.confidence;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Name-fallback classifications are capped at exactly 0.5, but the extract route warns only when classification.confidence < 0.5 (strict). This means users won't be warned when the classifier had to fall back from an invalid code to a name match, even though that path is explicitly treated as uncertain ("약간 강등"). For consistency with the alternative-promotion (0.4) and no-match (0.3) paths — both of which correctly trigger the warning — the name-fallback cap should be below the threshold, or the comparison should include the boundary.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/agent/src/agents/classifier-agent.ts, line 122:

<comment>Name-fallback classifications are capped at exactly 0.5, but the extract route warns only when `classification.confidence < 0.5` (strict). This means users won't be warned when the classifier had to fall back from an invalid code to a name match, even though that path is explicitly treated as uncertain ("약간 강등"). For consistency with the alternative-promotion (0.4) and no-match (0.3) paths — both of which correctly trigger the warning — the name-fallback cap should be below the threshold, or the comparison should include the boundary.</comment>

<file context>
@@ -0,0 +1,148 @@
+  }
+
+  // 코드는 못 맞췄지만 이름으로 맞춘 경우 확신을 약간 강등.
+  const confidence = byCode === undefined ? Math.min(raw.confidence, 0.5) : raw.confidence;
+  return {
+    school_level: matched.school_level,
</file context>
Suggested change
const confidence = byCode === undefined ? Math.min(raw.confidence, 0.5) : raw.confidence;
const confidence = byCode === undefined ? Math.min(raw.confidence, 0.49) : raw.confidence;

return {
school_level: matched.school_level,
grade: matched.grade,
topic_code: matched.code,
topic_name: matched.name,
generation_kind: raw.generation_kind,
problem_type: raw.problem_type,
difficulty: raw.difficulty,
confidence,
alternatives: alternatives.filter((alt) => alt.topic_code !== matched.code),
};
}

/** 같은 topic_code 중복 제거(첫 항목 유지) — LLM 이 같은 대안을 반복해도 UI 중복을 막는다. */
function dedupeByCode(
items: { topic_code: string; topic_name: string }[],
): { topic_code: string; topic_name: string }[] {
const seen = new Set<string>();
const out: { topic_code: string; topic_name: string }[] = [];
for (const item of items) {
if (seen.has(item.topic_code)) continue;
seen.add(item.topic_code);
out.push(item);
}
return out;
}
76 changes: 76 additions & 0 deletions packages/agent/src/agents/extractor-agent.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
/** ExtractorAgent — 첨부 문제(이미지 또는 텍스트)를 읽어 Extraction 으로 변환.
*
* 이미지: 비전 모델에 image 파트로 첨부. 텍스트: 프롬프트에 본문을 덧붙여 정규화.
* 출력 스키마는 ExtractionSchema 그대로 (메타데이터 조립 없음).
*/

import { generateObject, type LanguageModel } from "ai";

import { ExtractionSchema, type Extraction } from "../schemas/index.js";
import type { PromptLoader } from "../tools/prompt-loader.js";

export interface ExtractorImageInput {
kind: "image";
bytes: Uint8Array;
mediaType: string;
}

export interface ExtractorTextInput {
kind: "text";
text: string;
}

export type ExtractorInput = (ExtractorImageInput | ExtractorTextInput) & {
signal?: AbortSignal;
};

export interface ExtractorAgent {
extract(input: ExtractorInput): Promise<Extraction>;
}

export interface ExtractorAgentDeps {
model: LanguageModel;
modelId: string;
promptId: string;
prompts: PromptLoader;
}

export function createExtractorAgent(deps: ExtractorAgentDeps): ExtractorAgent {
return {
async extract(input) {
const prompt = await deps.prompts.load(deps.promptId);
const instructions = prompt.render({});
const temperature = prompt.metadata.temperature;

if (input.kind === "image") {
const { object } = await generateObject({
model: deps.model,
schema: ExtractionSchema,
mode: "json",
temperature,
messages: [
{
role: "user",
content: [
{ type: "text", text: instructions },
{ type: "image", image: input.bytes, mimeType: input.mediaType },
],
},
],
abortSignal: input.signal,
});
return object;
}

const { object } = await generateObject({
model: deps.model,
schema: ExtractionSchema,
mode: "json",
temperature,
prompt: `${instructions}\n\n# 입력 문제 (텍스트)\n\n${input.text}`,
abortSignal: input.signal,
});
return object;
},
};
}
11 changes: 9 additions & 2 deletions packages/agent/src/agents/generator-agent.ts
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,10 @@ export function assembleGeneratedProblem(input: {
readonly promptId: string;
readonly promptVersion: string;
}): GeneratedProblem {
const generationKind = generationKindForTopic(getGenerateRequestTopicCode(input.request));
const generationKind =
input.request.source_origin === "attached" && input.request.generation_kind !== undefined
? input.request.generation_kind
: generationKindForTopic(getGenerateRequestTopicCode(input.request));
return {
candidate_id: randomUUID(),
mode: input.request.mode === "conceptual" ? "conceptual" : "structural",
Expand Down Expand Up @@ -115,14 +118,18 @@ export function createGeneratorAgent(deps: GeneratorAgentDeps): GeneratorAgent {
return {
async generate(input) {
const prompt = await deps.prompts.load(deps.promptId);
const generationKind = generationKindForTopic(getGenerateRequestTopicCode(input.request));
const generationKind =
input.request.source_origin === "attached" && input.request.generation_kind !== undefined
? input.request.generation_kind
: generationKindForTopic(getGenerateRequestTopicCode(input.request));
const temperature = temperatureForGeneratorAttempt(
input.attempt,
prompt.metadata.temperature,
);
const basePromptVars = {
request: input.request,
generationKind,
attached: input.request.source_origin === "attached",
intent: input.intent,
refs: input.refs,
strategy: input.strategy === null ? "" : JSON.stringify(input.strategy, null, 2),
Expand Down
2 changes: 2 additions & 0 deletions packages/agent/src/agents/index.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
export * from "./classifier-agent.js";
export * from "./constraint-critic-agent.js";
export * from "./extractor-agent.js";
export * from "./generator-agent.js";
export * from "./refiner-agent.js";
export * from "./solver-agent.js";
4 changes: 4 additions & 0 deletions packages/agent/src/config/env.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,10 @@ export const EnvSchema = z.object({
* with a separate LanguageModel instance to decorrelate errors with generation.
* When unset, the solver shares the generator's resolved model. */
SOLVER_MODEL: z.string().min(1).optional(),
/** Optional override for the attached-problem extractor (POST /api/extract).
* Image upload needs a vision-capable model; set this to a vision model id
* when the main LLM_MODEL is text-only. When unset, reuses LLM_MODEL. */
EXTRACT_MODEL: z.string().min(1).optional(),
OPENAI_API_KEY: z.string().min(1).optional(),
OPENAI_MODEL: z.string().min(1).optional(),
CLIPROXY_BASE_URL: z.string().url().optional(),
Expand Down
Loading
Loading