fix(rlm): eliminate empty-answer failure; add extended_reasoning + self-consistency#67
Merged
Conversation
When the CodeAct loop exhausts rlm_max_iters without ever calling final(), the engine hands back an empty string with found defaulted to True, so RLMAnswerService reported answer="" with no_answer=False -- a silent degenerate non-answer a caller cannot distinguish from a real answer (the AstraZeneca Q4 case: 251s of compute, empty body, no_answer False). Treat a blank (empty or whitespace-only) answer as no_answer=True and substitute an explanatory note, dropping orphaned citations. This makes the failure detectable so callers can fall back or retry. Adds unit tests for the empty and whitespace-only cases.
Adds DEBUG turn-level tracing to the RLM CodeAct loop (per-turn reasoning, the model-written code block, and the REPL stdout) plus two WARNING signals for the empty-answer failure mode: the session warns when it exhausts rlm_max_iters without calling final(), and the answer service warns when it falls back to no_answer for a blank result. Uses canon's stdlib logging so the lines flow through pyfly's structured log pipeline; DEBUG tracing is opt-in and silent in production.
…swer Q4-class questions diverged into an empty answer: the model synthesised a complete answer into a variable, but its sub-llm output exceeded the 4000-char REPL stdout cap, so printing it looked "cut off". The model then spent every remaining turn re-asking the sub-llm for "the remaining parts" and never called final(), so the loop exhausted and returned empty. Two fixes: - System prompt: state that print() stdout is truncated for display only, that a long answer in a variable is held in full even when its printout looks cut off, and to call final(variable) immediately without printing to verify. - Out-of-iters fallback: re-ask with NO tools (the model can no longer run code and must emit text) and a larger token budget, so even a diverged run reliably produces a substantive answer instead of an empty string. Verified on the AstraZeneca Q4 case: previously empty after 16 iterations (251s); now converges via final() at turn 12 with a grounded 3116-char answer. Adds a unit test asserting the forced-final turn is tool-less.
The convergence fix told the model to call final() immediately and not print to verify. That was too broad: on a benchmark-computation question it biased the model to submit an unverified figure, and in the AstraZeneca Q1 re-run RLM fabricated benchmarks equal to the team's own input numbers and reversed the conclusion (scored 1.2/5). Scope the guidance: keep "don't re-print/reassemble an answer that only LOOKS cut off" (the truncation fix that makes Q4 converge), but replace the blanket "call final() immediately, don't verify" with a REQUIREMENT to verify every benchmark/threshold/figure against the document text and to never state a benchmark not found in the docs (and never assume the benchmark equals the figure under evaluation). Verified on the host harness: Q1 now uses the real role-specific benchmarks (Reps daily >=6, MSL panel >=60, coverage 95/95/75, DSL=TBC) and computes the correct pass/fail; Q4 still converges with the grounded 90/50-60/200/300 curve and drops the prior "Gates/Compuertas" fabrication.
Adds an optional `extended_reasoning` flag to AnswerRequest. When true, the RLM engine runs at twice the configured budget (rlm_max_iters * 2) for that one request -- a per-request lever for hard multi-fact questions that need more turns to gather and verify evidence, instead of a blunt global bump of the default for all traffic. The budget is derived from the configured value (x2), not hardcoded, so it tracks FLYCANON_RLM_MAX_ITERS. RAG ignores the flag (single-shot). The dispatcher already forwards the request verbatim, so no dispatcher signature change is needed.
RLM is unstable run-to-run on hard analytical questions (e.g. the AstraZeneca benchmark question scored 2.0/3.3/4.5 across three identical-config runs). Its best-of ceiling clearly beats RAG, but blind you get the variable expected quality, and a single bad run can fall below RAG. Adds rlm_self_consistency (env FLYCANON_RLM_SELF_CONSISTENCY, default 1 = off). When >1, the non-streaming answer path runs the engine N times in parallel and an LLM selector picks the most-grounded / most-consistent answer, clawing back the variance at N times the compute. Token usage of every run plus the selector is merged so cost reflects the full spend. Streaming keeps a single run so live turns map to one trajectory. Composes with extended_reasoning.
…n in CI The S3 object-store backend (storage/s3.py) is an optional runtime extra, but its unit tests and the pyright type-check of that module run unconditionally in CI. boto3 was previously only present transitively; that transitive path went away, breaking the Unit tests and Typecheck gates with unresolved-import / None-client errors unrelated to any route change. Declaring boto3 in the dev toolchain makes the dev/CI environment able to test and type-check the S3 backend deterministically.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On harder questions the RLM answer path could return an empty answer flagged as valid (
no_answer=False, blank body). Root cause (found from the CodeAct transcript): the engine builds a complete answer in a Python variable, but when a sub-llm()output exceeds the REPL's 4000-char stdout cap the printout looks "cut off"; the model then spends its remaining turns re-asking for "the remaining parts," never callsfinal(), and the loop exhausts and returns nothing — undetectable by the caller. Raisingrlm_max_itersdid not reliably fix it (some questions diverged even at double the budget).Changes
07a2830) — a blank RLM result is surfaced asno_answer=truewith an explanatory note, never an empty string mislabeled as a valid answer.b275480) — per-turn DEBUG trace, plus a WARNING when the loop exhausts its budget without callingfinal().20878d6) — the system prompt now states stdout truncation is display-only (callfinal(variable)directly, don't reassemble), and the out-of-iterations fallback re-asks with no tools so a diverged run must emit text instead of an empty string.e7e117b) — requires verifying every benchmark/threshold/figure against the documents and forbids stating a value not found in the source; scopes the earlier "commit immediately" guidance so it no longer suppresses verification.extended_reasoning(b556ffa) — optionalAnswerRequest.extended_reasoningruns the engine at twice the configuredrlm_max_itersfor a single hard request, without a global default bump. RAG ignores it.476f16b,923a11d) —FLYCANON_RLM_SELF_CONSISTENCY(default1= off); when>1, the non-streaming answer path runs the engine N times in parallel and an LLM selector returns the most-grounded / most-consistent answer. Token usage of all runs + the selector is merged. Documented inenv_template.Config / API surface
FLYCANON_RLM_SELF_CONSISTENCY(int, default1, range 1–8).AnswerRequest.extended_reasoning(bool, defaultfalse; RLM-only — RAG ignores it).Tests
171 RLM-area unit tests pass, including red-verified tests for the tool-less out-of-iterations fallback, the
extended_reasoningbudget doubling, and the self-consistency N-runs-and-select.