fix(prompts): preserve verification_expression through refiner#22
Merged
Conversation
refiner.md had no instruction to carry or regenerate verification_expression, so any candidate routed through the refiner lost the field and fell back to "기호 검증 불가" (sympy unverified). Add the prior expression to the Prior section and require an updated expression in the output, with the same SymPy notation rules as problem-generator.md. Verified live: 3-problem real generation at topic=10공수05-02 went from 1/3 verified (refiner-path candidates unverified) to 3/3 verified, with refined candidates carrying expressions that evaluate to the declared answer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
문제
PR #21의 검증식 평가 기능이 머지된 뒤 실생성으로 재검증한 결과, **refiner를 거친 후보는 여전히 "기호 검증 불가"**가 떴습니다 (1차 라운드: 1/3 verified, 2/3 warning — warning 2건 모두
refined_by에 refiner 포함).원인:
refiner.md프롬프트에verification_expression관련 지시가 전혀 없어, refiner가 문제를 고쳐 쓰면서 검증식 필드를 출력하지 않음 → sympy_verify 게이트가 평가할 식이 없어 unverified로 폴백.수정 (refiner.md v0.2.0 → v0.3.0)
{{#if prior.verification_expression}})verification_expression필드 요구 + 문제 수정 시 검증식도 함께 갱신하라는 지시 (problem-generator.md와 동일한 SymPy 표기 규칙)검증 (실LLM 라이브 생성, topic=10공수05-02 순열과 조합, mode=structural)
expression_check: true수정 후 게이트 evidence 예시 (refiner 경유 후보):
{"engine": "sympy", "status": "passed", "expected_answer": "3456", "expression_check": true, "verification_expression": "factorial(4)*factorial(3)*factorial(4)", "evaluated_value": "3456"}보너스: 게이트가 실판정을 내는 것도 확인 — 한 attempt에서
expected_answer: "3456가지"(단위 포함)를 failed로 잡아 재시도시켰고, 재시도 후보가 통과 (attempt_count: 2).🤖 Generated with Claude Code
Summary by cubic
Preserves and enforces
verification_expressionin the refiner so refined candidates stay symbolically verifiable. The prior expression is surfaced to the model, and an updated SymPy expression is now required in the output; live gen improved from 1/3 to 3/3 verified.packages/agent/prompts/refiner.md, includeprior.verification_expressionin the Prior section when available.verification_expressionin Output with the same SymPy rules as the generator; update it when the problem changes and omit only when the answer isn’t a single value/expression.0.3.0.Written for commit 765af52. Summary will update on new commits.