fix(kb): stop silently truncating long web pages on ingest - #579
Open
ViaJables wants to merge 2 commits into
Open
fix(kb): stop silently truncating long web pages on ingest#579ViaJables wants to merge 2 commits into
ViaJables wants to merge 2 commits into
Conversation
… submit The governance module only passed once an admin/examiner set wf.verified, which nothing but queue approval does. Every certifying user was blocked on another person, and submit_for_verification notified nobody, so requests sat unreviewed. Module content compounded it by instructing users to "mark the workflow as verified in the workflow settings" — not a thing that exists. - _validate_governance now passes on >=1 workflow submitted for verification (any status). Approval earns stars 2 and 3, so the credential keeps its teeth. Verified count takes max(approved requests, own verified workflows) so pre-queue verifications don't regress existing stars. - _notify_examiners sends an in-app notification + email to every admin and examiner when a request is created; skips the submitter, best-effort so a mail failure can't fail the submission. - New verification_submitted_email template and NotificationBell mapping. - Corrected module content in Certification.tsx and exercises.json. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Long web pages added to a knowledge base were silently cut off partway through, poisoning retrieval: the source showed "ready" with a green check, but everything past the cut was never chunked, so the KB gave wrong/incomplete answers about later sections. Root cause: the raw HTML was sliced at `web_fetcher_max_chars` (500k) *before* text extraction. HTML markup dwarfs the readable text, so a long .gov/.edu page exceeds 500k of HTML well before its body ends — trafilatura only ever saw the head of the document. Fix: - Cap raw HTML at a separate, much larger `web_fetcher_max_html_chars` (8M), and apply `web_fetcher_max_chars` only to the *extracted text*. Real documents now reach trafilatura in full. - Add a `truncated` flag to WebFetchResult, threaded through both URL ingestion paths (async service + Celery) onto KnowledgeBaseSource and the source API. When text genuinely hits the cap it's logged and flagged instead of passing silently. The text-ingest repair path clears the flag (bundled text is chunked whole). - Surface truncation in the UI: a ready-but-truncated source shows an amber warning icon + note in the source list and a banner in the inspector modal, instead of a clean check. - Tests: a long page whose HTML exceeds the text cap now keeps its tail; genuine text-over-cap and html-over-cap cases set the flag; a normal page stays unflagged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Support ticket: "Long web pages are silently truncated when added to a KB, causing wrong answers." A 2024 federalregister.gov guidance page was added to a KB — the source showed Status: ready, 378 chunks, green check, no warning — but the stored text stopped at "§200.331 — Subrecipient and Contractor Determinations" and everything after (including Subpart E) was never ingested.
Root cause
web_fetcher.pysliced the raw HTML atweb_fetcher_max_chars(500k) before text extraction:That limit is meant to bound extracted text. HTML markup (nav, inline styles, nested tables, scripts) is many times larger than the readable text, so a long .gov reg page blows past 500k of HTML well before its body ends. trafilatura only ever saw the head of the document; the extracted text then came in under 500k, so every later cap was a no-op — and status was set to
readywith no flag.Fix
1. Extract from the full page. New
web_fetcher_max_html_chars = 8_000_000bounds raw HTML separately and generously;web_fetcher_max_charsnow applies only to extracted text. Real documents never hit the HTML cap, so the whole body reaches trafilatura.2. Make genuine truncation non-silent.
WebFetchResult.truncatedis set whenever text or HTML actually hits a cap (with alogger.warning), threaded through both URL-ingestion paths (async service + Celery) ontoKnowledgeBaseSource.truncatedand the source API. The text-ingest repair path clears it (bundled text is chunked whole).3. Surface it in the UI. A ready-but-truncated source shows an amber ⚠ (instead of the green check) with a note in the source list, and a warning banner above the Extracted-text panel in the inspector modal.
Tests
New cases in
test_web_fetcher.py:Section 299present,truncated=False) — the regression guard.truncated=True.Backend compiles; frontend typechecks; new fetcher tests pass.
Note for reviewers
Built in an isolated worktree from
mainso this branch contains only the truncation fix — an unrelatedcan_managefeature was in progress in the working tree and is deliberately excluded.Follow-up (not in this PR)
This fixes ingestion going forward. The already-ingested federalregister source is still truncated in ChromaDB and needs a re-sync to pull Subpart E in. Happy to wire a one-shot backfill if wanted.
🤖 Generated with Claude Code