Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
eefde1f
docs(spec): semantic detection labels — log_content, stack_trace, dif…
bfirestone Apr 11, 2026
b2d0594
docs(plan): semantic detection labels implementation plan
bfirestone Apr 11, 2026
22bc292
test(detections): repair stale iter15 test baseline
bfirestone Apr 11, 2026
731275e
feat(detections): remove det_log_lines (sub_type-only now)
bfirestone Apr 11, 2026
c577020
feat(detections): introduce SEMANTIC_LABELS frozenset
bfirestone Apr 11, 2026
9c6181a
feat(detections): add log_content semantic label
bfirestone Apr 11, 2026
a153c9b
fix(detections): tighten log_content JSON + logfmt wording
bfirestone Apr 11, 2026
7a4d5b2
feat(detections): add stack_trace semantic label
bfirestone Apr 11, 2026
2da73a6
fix(detections): tighten stack_trace definition + regression tests
bfirestone Apr 11, 2026
bae9664
feat(detections): add diff_patch semantic label
bfirestone Apr 11, 2026
1009529
fix(detections): tighten diff_patch adjacent-lines and email-header r…
bfirestone Apr 11, 2026
9b959a3
feat(audit): build_audit_sample.py — stratified + injected audit input
bfirestone Apr 11, 2026
d0f6e26
fix(audit): defensive guards in build_audit_sample
bfirestone Apr 11, 2026
489e26f
feat(audit): audit_semantic_labels.py — metrics + report generator
bfirestone Apr 11, 2026
0d95a8a
fix(audit): nan handling + schema drift guards + unused imports
bfirestone Apr 11, 2026
c1ec175
fix(detections): log_content — recognize CSV/TSV log records
bfirestone Apr 11, 2026
5ab5ab7
docs(iter16): iteration report + audit artifacts
bfirestone Apr 11, 2026
ad7244f
docs(iter16): elevate A/B regression audit to pre-90k gate in next steps
bfirestone Apr 11, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 77 additions & 0 deletions docs/accuracy_runs/2026-04-10-iteration-16-audit-report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# Iter16 Semantic Label Audit Report

**Date:** 2026-04-10
**Models:** gemini3flash, sonnet, gpt54mini

## Pass Criteria

1. Inter-annotator agreement >=0.995 on stratified 5k for ALL labels.
2. Recall (majority >=2 of 3) >=0.9 on injected positives.
3. Zero obvious rule violations in spot-check (manual).

## Criterion 1: Inter-Annotator Agreement (stratified 5k)

| Label | Agreement | Pass |
|---|---:|:---:|
| c_cpp | 0.9940 | FAIL |
| csharp | 0.9998 | PASS |
| css | 0.9973 | PASS |
| csv | 0.9994 | PASS |
| diff_patch | 0.9999 | PASS |
| dockerfile | 0.9999 | PASS |
| fixed_width | 0.9922 | FAIL |
| go | 0.9994 | PASS |
| graphql | 0.9998 | PASS |
| html | 0.9910 | FAIL |
| ini | 0.9918 | FAIL |
| java | 0.9987 | PASS |
| javascript | 0.9923 | FAIL |
| json | 0.9947 | FAIL |
| jsonl | 0.9981 | PASS |
| key_value | 0.9775 | FAIL |
| kotlin | 0.9998 | PASS |
| latex | 0.9922 | FAIL |
| log_content | 0.9986 | PASS |
| lua | 0.9997 | PASS |
| makefile | 0.9965 | PASS |
| markdown | 0.9507 | FAIL |
| objc | 0.9999 | PASS |
| php | 0.9995 | PASS |
| pipe_table | 0.9971 | PASS |
| plain | 0.9391 | FAIL |
| powershell | 0.9997 | PASS |
| python | 0.9981 | PASS |
| r | 0.9998 | PASS |
| rst | 0.9975 | PASS |
| ruby | 0.9999 | PASS |
| rust | 0.9999 | PASS |
| sgml | 0.9976 | PASS |
| shell | 0.9850 | FAIL |
| sql | 0.9990 | PASS |
| stack_trace | 0.9997 | PASS |
| swift | 0.9999 | PASS |
| toml | 0.9987 | PASS |
| tsv | 0.9988 | PASS |
| typescript | 0.9978 | PASS |
| xml | 0.9975 | PASS |
| yaml | 0.9949 | FAIL |

**Criterion 1 verdict: FAIL**

## Criterion 2: Recall on Injected Positives

| Label | Recall (majority) | Pass |
|---|---:|:---:|
| diff_patch | 1.0000 | PASS |
| log_content | 0.8400 | FAIL |
| stack_trace | 0.6250 | FAIL |

**Criterion 2 verdict: FAIL**

## Criterion 3: Spot-Check Rule Violations

_Manual review required -- scan disagreement rows and injected positives for obvious rule violations (e.g., log_content firing on lowercase error in prose)._

## Overall Decision

**Gate: FAIL.** Iterate on label definitions; re-run audit.
244 changes: 244 additions & 0 deletions docs/accuracy_runs/2026-04-10-iteration-16.md

Large diffs are not rendered by default.

Loading
Loading