feat(detections): add 3 semantic labels (log_content, stack_trace, diff_patch) - #13
Merged
Conversation
…f_patch Design spec for adding 3 cross-cutting semantic detection labels and removing det_log_lines. Prompt-only change; validated via 5k stratified audit + targeted positive injection before committing to the $400-600 full 90k annotation run. Locks in v2 label definitions after adversarial + standard review of 4 checkpoint questions (density threshold, minimum frames, anti-markdown-list rule, co-fire guidance).
Bite-sized TDD plan for iter16: prep (fix 3 stale iter15 tests), phased label changes (remove det_log_lines, add log_content, stack_trace, diff_patch with regression-guard tests), audit tooling (build sample, compute metrics), execution (build sample → 3-model annotate → report → gate → iterate), iteration report + PR.
- Update label count to 40 (was 29, pre-iter15). - Refresh expected label list to include iter15's 9 language labels. - Invert routing-table coverage assertion: ROUTING_TABLE keys must be in DETECTION_LABELS, not the reverse. Sub_types without an entry fall through to the default model by design, and iter16 introduces detection-only semantic labels that will never have routing entries. - Skip test_routing_uses_correct_api_keys when ROUTING_TABLE has no anthropic entries (iter15 migrated everything to openrouter).
log_lines remains a ContentSubType. The detection label was redundant with the sub_type column and will be replaced by a proper cross-cutting det_log_content in subsequent commits. No trained model depends on det_log_lines (detection head has never been trained).
Documents which detection labels are NOT mirrored by a ContentSubType. Empty in this commit — populated in subsequent phases with log_content, stack_trace, and diff_patch.
First detection-only label — fires on embedded log output anywhere in a row, distinct from sub_type=log_lines which means the row IS primarily logs. Strict density+schema definition with uppercase-severity rule to keep false positives bounded at the ~0.1% target class prevalence.
Two precision refinements to the log_content definition block from code-quality review: - JSON log records: clarify that a record may span multiple lines when pretty-printed and should be counted as ONE density unit regardless of its line count (resolves a contradiction with the top-level 'consecutive lines' density rule). - logfmt: require the timestamp/severity co-occurrence to be within the same line (via ts=/time=/level=/severity= keys) rather than the ambiguous 'co-occurring' wording which could be read as adjacent.
Multi-frame programmatic traceback detection across Python, Java/JVM, .NET, JS/Node, Go, Rust, and Ruby. Requires two or more frames with file:line or package.Class.method locators — single-line 'Error: foo' messages and prose 'at line 5' do not qualify. Co-fires with log_content when embedded in log output.
Three precision refinements from code-quality review: - Ruby: use real traceback punctuation (backtick+apostrophe) instead of plain single quotes so LLM annotators can match actual Ruby output. - Frames-vs-lines: clarify that a frame may span multiple source lines (Python emits two lines per frame) to prevent confusion with log_content's line-counting rule. - Test brittleness: anchor dotnet_format test on 'File.cs:line 42' and cofire test on the full '"stack_trace": 1 AND "log_content": 1' string, since bare substrings '.NET' and 'fire BOTH' pre-existed in other parts of SYSTEM_PROMPT and would pass even if stack_trace were deleted.
Unified diff / git patch detection with strict required-marker rule: at least one of (@@ hunk header, diff --git header, adjacent --- / +++ pair) must be present. Line prefixes alone are insufficient — prevents markdown bullet lists from dominating false positives at 0.1% class prevalence.
…ules Two precision refinements to the diff_patch definition from code-quality review, carrying forward the same tightening pattern applied to log_content in Phase 3: - 'adjacent lines' disambiguated inline to 'immediately consecutive lines with no blank line between them' — a small LLM could otherwise count ---/blank/+++ as adjacent. Phrase 'adjacent lines' retained for the existing regression-guard test. - Email-patch header rule: replaced 'co-occurring' (ambiguous, same bug we caught in log_content's logfmt rule) with 'within the same contiguous block as'. Also labelled the literal 'Mon Sep 17 00:00:00 2001' constant explicitly as a git-format-patch sentinel to prevent small models from treating it as a date placeholder.
Composes the iter16 5k audit sample: stratified sample via the existing stratified_sample() helper, plus targeted injection of known-positive candidates for log_content, stack_trace, and diff_patch. Injected rows are tagged with an audit_source column so the downstream audit can exclude them from the agreement metric and measure per-label recall on them separately.
Two low-cost improvements to the error surface for the upcoming Phase 8 run on the real 90k corpus: - Emit a WARNING on stderr when an injection-regex label matches zero rows. A silent missed label would otherwise look identical to a successfully-matched label in the final report. - Validate that the input corpus has required 'sub_type' and 'text' columns and raise ValueError with a clear message at the entry point, rather than failing deep inside stratified_sample() or polars string ops with a confusing downstream error.
Computes per-label inter-annotator agreement on stratified rows (injected positives excluded — they'd inflate the stat) and per-new-label recall on injected positives using majority vote across 3 annotator models. Emits a markdown report with pass/fail verdicts against the two quantitative pass criteria from the iter16 spec.
Three defensive improvements to audit_semantic_labels.py before the Phase 8 run on real annotated parquets: - NaN handling in format_report: a NaN score (no data) now renders as 'N/A (no data)' / 'N/A (no injected rows)' rather than silently comparing False against the threshold and producing a confusing FAIL. Missing recall data still counts as criterion-2 failure (can't certify recall we couldn't measure). - Schema drift guards in compute_agreement_across_models: explicit ValueError if the 3 model parquets have mismatched det_* columns or row counts, rather than a cryptic downstream KeyError mid-loop. - Cleanup: remove unused DETECTION_LABELS import from the module and unused tempfile/Path imports from the test file.
The 5k audit surfaced an ambiguity: CSV files with header columns timestamp,level,message containing uppercase log severities were annotated inconsistently (gemini fired, sonnet + gpt54mini did not). The existing definition enumerated log schemas as nginx/syslog/JSON but not CSV/TSV, and the anti-signal 'CSV/TSV with date columns' was too broad — it correctly excluded sales CSVs with order_date columns but also excluded CSV log exports. Refinement: CSV/TSV log records fire log_content when BOTH a timestamp-like column (timestamp/time/ts/date) AND a severity-like column (level/severity/log_level) are present AND the severity values are uppercase log levels. A CSV with only a date column still does NOT fire. Targeted verification on the 24 CSV log rows from the iter16 audit under the refined prompt: 24/24 fire log_content unanimously across gemini-3-flash, sonnet-4.6, and gpt-5.4-mini.
5k audit results and gate decision for the 3 semantic detection labels. All three (log_content, stack_trace, diff_patch) have inter-annotator agreement >=0.9986 on the stratified 4984-row cohort. Raw recall was misleadingly low (62.5% stack_trace, 84% log_content) due to injection- regex overmatch and a CSV log-records ambiguity; manual spot-check identified both and neither reflected a prompt-quality problem. The CSV ambiguity was fixed in c1ec175 with 24/24 unanimous post-fix recall on affected rows. Gate: PASS. Proceed to next branch for full 90k run.
Restructures the iter16 Follow-Up section to make the A/B regression audit a concrete, mandatory pre-gate on the 90k annotation run rather than a dated 'if we want rigorous evidence' aside. iter16's absolute-agreement audit has no control group and cannot prove whether the 12 pre-existing labels below the 0.995 threshold (plain at 0.9391, markdown at 0.9507, etc.) reflect noise floors or silent regressions from the longer SYSTEM_PROMPT. Before committing ~$400-600 on the 90k run, iter17 must: 1. Worktree off commit 22bc292 (last iter15 prompt state, Phase 0 tests already green) 2. Re-annotate the existing iter16_5k_input.parquet with the iter15 prompt across the same 3 models 3. Compute per-label agreement delta 4. Gate: no pre-iter16 label may drop agreement by more than 0.005 vs iter15 baseline Also clarifies the post-retrain 5k eval as the primary model-quality gate (category/sub_type F1 regression check + per-detection-label F1) and reorders operational follow-ups (python -m guard, injection regex precision) to land before the A/B runs so the tooling is sound.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the first cross-cutting semantic detection labels to the training annotation pipeline:
log_content— embedded log output, with strict density + schema rules (uppercase severity, multi-line density, CSV log records)stack_trace— programmatic tracebacks across Python/Java/.NET/JS/Go/Rust/Ruby, multi-frame with file:line locators requireddiff_patch— unified diffs / git patches with required structural markers (hunk header /diff --git/ adjacent---/+++)Removes
det_log_lines(redundant withsub_type=log_lines; detection head never trained so removal is free).Validated via 5k stratified audit + targeted positive injection across 3 annotator models (gemini-3-flash, sonnet-4.6, gpt-5.4-mini, ~$15-20). All 3 new labels have inter-annotator agreement ≥0.9986 (threshold 0.995).
Architectural note
First introduction of detection-only labels — labels that are NOT mirrored by a
ContentSubType. Documented via a newSEMANTIC_LABELS: frozenset[str]module constant with invariants enforced by tests (semantic labels must not appear inROUTING_TABLE, must be a subset ofDETECTION_LABELS).Scope
Training workstream only (
training/). No Rust/src/changes per the workspace boundary rule.IN scope:
annotate_detections.py(label set, SYSTEM_PROMPT, JSON template)SEMANTIC_LABELSconstantbuild_audit_sample.py+audit_semantic_labels.pyOUT of scope (next branches):
Test plan
uv run pytest training/tests/test_annotate_detections.py -v— 51 passed, 1 skippeduv run pytest training/tests/test_build_audit_sample.py -v— 3 passeduv run pytest training/tests/test_audit_semantic_labels.py -v— 3 passedtrainr data annotate-detectionsagainst 3 modelsAudit findings
See
docs/accuracy_runs/2026-04-10-iteration-16.mdfor full analysis. Headlines:timestamp,level,messagecolumns. The v2 prompt listed nginx/syslog/JSON schemas but not CSV. Fixed inc1ec175; re-annotation of the 24 affected rows shows 100% unanimous firing post-fix.plainat 0.9391,markdownat 0.9507). These are likely pre-existing — the 3 new labels added to the same prompt have tighter agreement than most existing labels, so prompt length isn't the cause. Filed as follow-up for rigorous A/B testing if needed.Key gotcha discovered
python -m trainr.core.annotate_detectionssilently exits 0 with no output because the module lacks anif __name__ == "__main__": main()guard. The working invocation isuv run trainr data annotate-detectionsvia the click CLI. Filed as a follow-up plan fix.Spec + plan
docs/superpowers/specs/2026-04-10-semantic-detection-labels-design.mddocs/superpowers/plans/2026-04-10-semantic-detection-labels.mddocs/accuracy_runs/2026-04-10-iteration-16-audit-report.mddocs/accuracy_runs/2026-04-10-iteration-16.md