feat(grouper): strip noise + deterministic dedup before classification#31
Merged
Conversation
Reduces digest duplication and noise by addressing three sources of low-signal output observed in production: 1. Channel summary header line (🚀 ...) was being extracted as a separate bullet by the grouper, duplicating Section 1 key points. Strip it before feeding to classification. 2. Section 2 (📎 Also/Также) low-priority items were promoted to cross-channel grouping where they competed with high-signal bullets. Strip the section before classification — Also is per-channel by design. 3. The grouper occasionally emitted the same (group, source, point) twice. Add deterministic dedup in _parse_grouped_response keyed on (group, source, normalized_point) so prompt flakes can no longer produce verbatim duplicates. Also tightens the per-channel summarizer prompt with an explicit QUALITY GATE that drops low-signal posts (photo-only, meta-empty, expired local invites, internal admin, AI speculation) and caps Section 1 at 5 bullets. Measured impact (Approach A config-only baseline already cut items 80→44): these changes target the remaining 5 issue categories identified after that baseline.
Adding deterministic dedup pushed _parse_grouped_response complexity to 12, over the project's flake8 max-complexity=10. Extract the per-group inner loop (validation + dedup + GroupedPoint construction) into a helper, restoring complexity under threshold. No behavior change.
8 tasks
belaytzev
added a commit
that referenced
this pull request
May 9, 2026
… chokepoint (#32) * feat(grouper): strip noise + deterministic dedup before classification Reduces digest duplication and noise by addressing three sources of low-signal output observed in production: 1. Channel summary header line (🚀 ...) was being extracted as a separate bullet by the grouper, duplicating Section 1 key points. Strip it before feeding to classification. 2. Section 2 (📎 Also/Также) low-priority items were promoted to cross-channel grouping where they competed with high-signal bullets. Strip the section before classification — Also is per-channel by design. 3. The grouper occasionally emitted the same (group, source, point) twice. Add deterministic dedup in _parse_grouped_response keyed on (group, source, normalized_point) so prompt flakes can no longer produce verbatim duplicates. Also tightens the per-channel summarizer prompt with an explicit QUALITY GATE that drops low-signal posts (photo-only, meta-empty, expired local invites, internal admin, AI speculation) and caps Section 1 at 5 bullets. Measured impact (Approach A config-only baseline already cut items 80→44): these changes target the remaining 5 issue categories identified after that baseline. * refactor(grouper): extract _collect_group_points to satisfy C901 Adding deterministic dedup pushed _parse_grouped_response complexity to 12, over the project's flake8 max-complexity=10. Extract the per-group inner loop (validation + dedup + GroupedPoint construction) into a helper, restoring complexity under threshold. No behavior change. * refactor(grouper): split into 2a extractor + 2b classifier with dedup chokepoint Architecturally rewires the grouper from a single combined AI call (extract + classify + dedup at once) into a two-pass pipeline with a deterministic chokepoint between, addressing the architect's review of PR #31. Why: - Single combined call asks an LLM to do three jobs at once on input from 30+ channels. At low temperature this still produces flakes: same-topic bullets from different channels stay separate, cross-group duplicates leak, and the per-channel header line gets echoed as a bullet. - The grouper prompt was getting heavier and heavier trying to compensate. PR #31 added a deterministic post-dedup; this PR splits the pipeline so the LLM can focus on one task at a time and the deterministic step has more leverage. New flow: Pass 2a — `_extract_all_bullets`: Per-channel parallel AI calls (Semaphore-bounded at concurrency 10), each turning one channel summary into a flat list of ExtractedBullet. Each extractor sees only its own channel — no cross-channel context to confuse classification with. Chokepoint — `_dedup_extracted` (only when `dedup_topics: true`): Cross-channel deterministic merge by normalized point text. Same story from N channels → 1 bullet with sources joined as comma-separated list, longest text variant kept. The LLM never sees the duplicates. Pass 2b — `_classify_bullets`: Single AI call over the dedup'd flat list, classifies each bullet into a user-defined topic group. Smaller, focused prompt — much shorter than the old combined prompt. Public API unchanged: `group_summaries(channel_summaries, channel_urls)` still returns `Dict[str, List[GroupedPoint]]`. Also fixes core.py double-print bug: When a group's formatted message exceeds split_message threshold (4000 chars), it produces multiple chunks. The old code passed every chunk's group name into the summary header, producing duplicates like "📰 News, 📰 News" in the stats line. Dedupe with `dict.fromkeys` to preserve order. Tests: 33 grouper tests (all passing), full suite 400/0 with 79% coverage.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Cuts digest duplication and noise by fixing three sources of low-signal output observed in production digests.
After Approach A (config-only: model upgrade + lower temp + tighter token budget) cut items 80→44, five problem categories remained. This PR targets all five:
🚀 ...line before grouping📎 Also/Такжеsection before grouping(group, source, normalized_point)in_parse_grouped_responseWhy this design
Backend Architect review flagged that the grouper's single-pass design (extract + classify + dedup) is asking too much of the LLM at temperature 0.1 with 33 channels — it produces flakes. Prompt-only
dedup_rulewill not reliably catch verbatim duplicates. The right fix for that class is a deterministic chokepoint in_parse_grouped_response, which this PR adds.A future PR can split the grouper into 2a (extractor) + 2b (classifier+dedup) for stronger semantic merging; this PR deliberately stays small and ships the highest-leverage deterministic fixes first.
Test plan
tests/test_grouper.pycovering header strip, Section 2 strip (EN + RU), verbatim dedup, whitespace/case normalization, same-point-different-source preserveduv run pytest tests/— 389 passed, 7 skipped, coverage 77.85%uv run mypy src/— no issuesuv tool run ruff check src/ tests/— cleanuv run black --check src/ tests/— clean/digeston real channels and verify item count drop + noise qualityFiles
src/grouper.py— strip helpers, dedup in parsersrc/prompts/base_summary.txt— QUALITY GATE block, hard cap 5src/summarizer.py— inline prompt aligned with QUALITY GATEtests/test_grouper.py— 7 new test cases