Skip to content

fix(prompts): preserve verification_expression through refiner#22

Merged
smilebank7 merged 1 commit into
mainfrom
fix/refiner-verification-expression
Jun 11, 2026
Merged

fix(prompts): preserve verification_expression through refiner#22
smilebank7 merged 1 commit into
mainfrom
fix/refiner-verification-expression

Conversation

@smilebank7

@smilebank7 smilebank7 commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

문제

PR #21의 검증식 평가 기능이 머지된 뒤 실생성으로 재검증한 결과, **refiner를 거친 후보는 여전히 "기호 검증 불가"**가 떴습니다 (1차 라운드: 1/3 verified, 2/3 warning — warning 2건 모두 refined_by에 refiner 포함).

원인: refiner.md 프롬프트에 verification_expression 관련 지시가 전혀 없어, refiner가 문제를 고쳐 쓰면서 검증식 필드를 출력하지 않음 → sympy_verify 게이트가 평가할 식이 없어 unverified로 폴백.

수정 (refiner.md v0.2.0 → v0.3.0)

  • Prior 섹션: 기존 검증식을 컨텍스트로 전달 ({{#if prior.verification_expression}})
  • Output 섹션: verification_expression 필드 요구 + 문제 수정 시 검증식도 함께 갱신하라는 지시 (problem-generator.md와 동일한 SymPy 표기 규칙)

검증 (실LLM 라이브 생성, topic=10공수05-02 순열과 조합, mode=structural)

라운드 결과 refiner 경유 후보
수정 전 1/3 verified, 2/3 warning 2건 모두 검증식 누락 → unverified
수정 후 3/3 verified 2건 모두 검증식 보존, expression_check: true

수정 후 게이트 evidence 예시 (refiner 경유 후보):

{"engine": "sympy", "status": "passed", "expected_answer": "3456",
 "expression_check": true,
 "verification_expression": "factorial(4)*factorial(3)*factorial(4)",
 "evaluated_value": "3456"}

보너스: 게이트가 실판정을 내는 것도 확인 — 한 attempt에서 expected_answer: "3456가지"(단위 포함)를 failed로 잡아 재시도시켰고, 재시도 후보가 통과 (attempt_count: 2).

🤖 Generated with Claude Code


Summary by cubic

Preserves and enforces verification_expression in the refiner so refined candidates stay symbolically verifiable. The prior expression is surfaced to the model, and an updated SymPy expression is now required in the output; live gen improved from 1/3 to 3/3 verified.

  • Bug Fixes
    • In packages/agent/prompts/refiner.md, include prior.verification_expression in the Prior section when available.
    • Require verification_expression in Output with the same SymPy rules as the generator; update it when the problem changes and omit only when the answer isn’t a single value/expression.
    • Bump prompt version to 0.3.0.

Written for commit 765af52. Summary will update on new commits.

Review in cubic

refiner.md had no instruction to carry or regenerate verification_expression,
so any candidate routed through the refiner lost the field and fell back to
"기호 검증 불가" (sympy unverified). Add the prior expression to the Prior
section and require an updated expression in the output, with the same
SymPy notation rules as problem-generator.md.

Verified live: 3-problem real generation at topic=10공수05-02 went from
1/3 verified (refiner-path candidates unverified) to 3/3 verified, with
refined candidates carrying expressions that evaluate to the declared answer.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 1 file

Re-trigger cubic

@smilebank7 smilebank7 merged commit 0a7e900 into main Jun 11, 2026
5 checks passed
@smilebank7 smilebank7 deleted the fix/refiner-verification-expression branch June 11, 2026 05:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant