🎯 Fix ReAct agent bypassing document_qa for legal questions - #7
Merged
Conversation
The chat agent's TOOL SELECTION RULES only forced document_qa for prices/contracts, so questions about N.4412/2016 thresholds, award procedures, and articles were answered from model knowledge instead of the RAG-indexed law text — no article citations, no sources, and tool_used came back as "unknown" with an empty trace. Add legal topics as the first tool-selection rule, and make it apply even to bare topic/keyword input (e.g. "Συνοπτικός διαγωνισμός") rather than only full questions, since the model was otherwise treating unphrased topics as not covered by the rule.
docker compose restart backend reuses the existing image since the backend service has no source bind-mount, so edits to backend code or prompts silently no-op until the image is rebuilt. Cost two full verification cycles while fixing the legal-routing bug.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug
The chat agent was answering Greek procurement-law questions from model knowledge instead of routing to
document_qa, bypassing the N.4412/2016 RAG layer entirely — so answers came back with no article citations and no sources.Repro:
"Ποια είναι τα όρια απευθείας ανάθεσης;"through/chatreturnedintermediate_steps=[]andtool_used="unknown". The same question through/doc_qacorrectly cited Άρθρο 118. Retrieval was healthy — only tool selection was wrong.Fix
Added a new first rule to the TOOL SELECTION RULES block in
backend/prompts/chat/v1.txt, forcingdocument_qafor any message about N.4412/2016, legal thresholds, award/tender procedures, guarantees, deadlines, or specific articles.The rule had to be worded for bare topic phrases, not just questions: the suggestion chips in the UI are noun-phrase labels (e.g. "Συνοπτικός διαγωνισμός"), not interrogative sentences. A rule written as "for any question about X" didn't match them — the agent's own reasoning trace showed it explicitly deciding no tool was needed because the user hadn't asked a specific question. The rule now covers bare topic/keyword input too.
Verification
Logged in, sent all six Greek suggestion chips to
/chat, and checked the resulting tool call via:document_qadocument_qadocument_qabid_comparisonsupplier_lookupreport_generationAll three legal chips now route to
document_qa; the other three still route correctly. Repeated the previously-flaky chip multiple times to confirm the fix is stable rather than a lucky sample. Full backend test suite passes (160 passed).Also
Added a README note:
docker compose restart backenddoes not pick up backend code or prompt changes since there is no source bind-mount — it silently reuses the old image. Usedocker compose build backend && docker compose up -d backendinstead. This cost two silent no-op verification cycles while debugging the issue above.