Skip to content

feat(grouper): deterministic QUALITY GATE filter + broader noise strip#33

Merged
belaytzev merged 2 commits into
mainfrom
feat/deterministic-quality-gate
May 9, 2026
Merged

feat(grouper): deterministic QUALITY GATE filter + broader noise strip#33
belaytzev merged 2 commits into
mainfrom
feat/deterministic-quality-gate

Conversation

@belaytzev

Copy link
Copy Markdown
Owner

Summary

Architect's call after PR #32 merge analysis: stop trusting the LLM to follow negative drop rules. Add a deterministic regex/heuristic filter pass between extraction and dedup. Move QUALITY GATE prompt rules into the extractor as belt-and-suspenders.

This is the next iteration on top of PR #32. After running `/digest` on real bot, five categories of noise survived all the prompt-level rules:

Surviving noise Example Root cause Fix here
Admin chatter `🆕 В чате появился новый участник Denis Nogtev` gpt-5.4-mini ignored prompt Deterministic regex
Meta-empty `🛸 В подборке упомянуты темы, но без дополнительных деталей` gpt-5.4-mini ignored prompt Deterministic regex
Speculation `📊 Парк выглядит как сильный фотоспот` gpt-5.4-mini ignored prompt Hedging+entity check
Section header leak `📌 Key points: (Вастрик Селфхостеры🔧, kyrillic🌍, ...)` Strip regex didn't cover Broader noise strip
Numbered emoji prefix `1️⃣ 🍏 Intel и Apple...` Cosmetic clutter Strip in normalizer

Architecture change

Pass 2a extract → [NEW] QUALITY GATE filter → dedup chokepoint → Pass 2b classify
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
                  Deterministic. No AI dependency.

Same pattern as the dedup chokepoint introduced in PR #32: pull noise removal out of prompts and into code. Prompt rules stay as belt-and-suspenders but the actual safety comes from `_quality_gate_filter`.

QUALITY GATE rules (deterministic)

  • Admin chatter: `r"новый участник|joined the chat|появил.{0,30}участник"`
  • Meta-empty: `r"без (?:дополнительных )?(?:деталей|подробностей)|no details?|just a poll"`
  • Too short, no facts: <30 chars AND no digits AND no @-handle AND no URL AND no proper noun pair
  • Speculation without entity: matches `r"probably|похоже|вероятно|выглядит как"` AND no concrete entity

Substantive bullets pass through untouched (validated by `test_keeps_substantive_bullets_unchanged`).

Broader noise strip in _strip_channel_summary_noise

Five new regex patterns added to the existing 🚀 + 📎 strip:

  • `📌 Key points:` / `Ключевые моменты:` / `Puntos clave:` / etc — multilingual
  • `1️⃣` ... `9️⃣` standalone numbering prefix
  • `[emoji]`, `[brief fact]`, `[brief subject]`, `[link]` template token placeholders

Why move QUALITY GATE into extractor prompt too

Architect: "will help marginally because the task is simpler, but you're still asking a weak model to follow negative rules at low temperature on Russian text. Don't bet on it."

So this is belt-and-suspenders only. The deterministic filter is the load-bearing fix. Prompt now explicitly says "DROP rules OVERRIDE extract-verbatim" to resolve the conflict the model previously had between rules.

What's next (out of scope for this PR)

Per architect, the bigger remaining wins:

  1. Embedding-based fuzzy dedup (text-embedding-3-small, ~$0.0001/day) — would catch paraphrased duplicates that exact-string dedup misses (e.g. "Apple договорились с Intel" vs "Intel будет производить часть чипов Apple"). Solves cross-group duplication root cause.
  2. Structured summarizer JSON output instead of prose — eliminates the entire class of template-leak bugs at the source.

Test plan

Files

  • `src/grouper.py` — `_quality_gate_filter`, broader noise strip patterns, extractor prompt update, wire into `group_summaries`
  • `tests/test_grouper.py` — `TestStripBroaderNoise` (4 tests), `TestQualityGateFilter` (6 tests), updated `test_classifier_error_propagates` to use substantive bullet

belaytzev added 2 commits May 9, 2026 19:00
Stops trusting the LLM to follow negative drop rules. Adds a regex-based
filter that runs on the extracted bullet list before dedup/classification,
catching the noise patterns the prompt-level QUALITY GATE was failing to
suppress in practice (gpt-5.4-mini ignores negative rules at low temp).

Architect's recommendation: "use grep for what grep can do."

Three changes:

1. Broader noise strip in `_strip_channel_summary_noise`:
   - 📌 Key points / Ключевые моменты / Puntos clave / etc — template
     header was leaking into bullets as `📌 Key points: (channel list)`
   - 1️⃣-9️⃣ section numbering prefixes — cosmetic clutter
   - [emoji], [brief fact] template token placeholders

2. New `_quality_gate_filter` deterministic pass between extract and dedup.
   Drops:
   - Admin chatter (new chat members, joins/leaves)
   - Meta-empty bullets ("без подробностей", "no details", "just a poll")
   - Short bullets (<30 chars) lacking concrete entity (digits/@/URL/proper)
   - Speculation/hedging without concrete entity ("вероятно", "probably")

3. QUALITY GATE moved into extractor system prompt as belt-and-suspenders.
   Framed as "DROP rules OVERRIDE extract-verbatim" so the model isn't torn
   between conflicting instructions. Real safety still comes from #2.

Architecture:
  Pass 2a extract → QUALITY GATE filter → cross-channel dedup → Pass 2b classify
                    ^^^^^^^^^^^^^^^^^^
                    NEW deterministic chokepoint

Tests: 410 passed, 7 skipped, coverage 79.18%.

Real-world digest motivation (from PR #32 post-merge run):
- Tech section had `🆕 В чате появился новый участник Denis Nogtev` (admin chatter)
- Другое had `📌 Key points: (Вастрик Селфхостеры🔧, kyrillic🌍, ...)` (template leak)
- Tech had `🛸 В подборке упомянуты темы, но без дополнительных деталей` (meta-empty)
- Другое had `📊 Парк выглядит как сильный фотоспот` (speculation, no entity)
CI pins black 24.10.0 (per CLAUDE.md note); local newer black accepted
the multi-line form. CI black wants the call on one line.

Verified locally with: uvx --from "black==24.10.0" black --check src/ tests/
@belaytzev belaytzev merged commit 37202b9 into main May 9, 2026
5 checks passed
@belaytzev belaytzev deleted the feat/deterministic-quality-gate branch May 9, 2026 17:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant