perf: reduce LLM cost via span deduplication and two-pass mapping

[Blevene]

Context

Processing a 1.2MB Unit42 PDF costs ~$0.37 across 28 LLM calls (204K tokens). The BatchMapper accounts for 84% of cost (23 calls at 10 spans/batch for 201 spans). Increasing batch size to 25 (done in commit 49fb67e) helps, but further reductions are possible.

Proposed Optimizations

1. Span deduplication before mapping (Medium effort, ~30% cost reduction)

The SpanFinderAgent detects 201 spans from a large report, but many overlap — the same text region matched by multiple tactic patterns. Deduplicating spans before they reach the BatchMapper would reduce the total span count significantly.

Approach:

• After SpanFinder, group spans by overlapping character ranges
• Merge overlapping spans into a single representative span, preserving all matched pattern metadata
• Only send unique text regions to the mapper

Expected impact: 201 spans → ~80-120 unique spans → fewer mapper batches

2. Two-pass mapping with cheap pre-filter (Higher effort, ~50% cost reduction)

Most spans from pattern matching are noise (generic language that happens to match tactic keywords). A cheap first pass could filter these before the expensive LLM mapper.

Approach options:

• Heuristic scoring: Use keyword density, technique ID presence, and vector similarity scores to classify spans as "likely technique" vs "noise" without an LLM call
• Small model pre-filter: Use a cheaper/faster model (e.g., Gemini Flash Lite or a local classifier) for binary "contains TTP?" classification, then only send positive spans to the full mapper
• Confidence threshold on retriever: The BatchRetrieverAgent already scores candidates — raise the threshold to filter low-confidence spans before mapping

Expected impact: 201 spans → 50-80 high-confidence spans → 2-4 mapper batches instead of 8-10

Cost projections

Optimization	Mapper calls (201 spans)	Est. cost	Savings
Current (batch=25)	~8	~$0.15	baseline
+ Span dedup	~4-5	~$0.08	~47%
+ Two-pass filter	~2-3	~$0.05	~67%
Both combined	~2	~$0.04	~73%

References

• Cost tracking PR: feat: wire LLM cost tracking with litellm.completion_cost() #19
• Batch size increase: commit 49fb67e
• BatchMapper: bandjacks/llm/mapper_optimized.py
• SpanFinder: bandjacks/llm/agents_v2.py (SpanFinderAgent)

[Blevene]

Implementation Update

Span deduplication — implemented, opt-in

Span dedup is now available via enable_span_dedup: true in the extraction config. Disabled by default to preserve extraction quality.

Benchmark results (same 1.2MB Unit42 PDF, 3 runs):

Configuration	LLM Calls	Cost	Techniques	Flow Coherence
Batch=10, no dedup (old default)	28	$0.367	61	100%
Batch=25, no dedup (new default)	17	$0.274	57	100%
Batch=25 + span dedup	13	$0.210	47	100%

Quality impact of dedup:

• Reduces LLM calls from 17 to 13 (24% fewer), saving ~$0.06/report
• But drops technique count from 57 to 47 (~15% fewer techniques)
• Flow kill chain coherence is unaffected (100% in all runs)
• Average edge probability slightly improved (0.48 → 0.50)
• 36 techniques are stable across all 3 runs (the "core set")

Usage:

# Via API
POST /v1/reports/ingest_async
{"text": "...", "config": {"enable_span_dedup": true}}

# Via CLI
python -m bandjacks.cli.batch_extract --config '{"enable_span_dedup": true}'

Commits: 821b201, 42e6401, 489bd5e

Two-pass mapping — still open

The pre-filter idea (heuristic or cheap model to classify spans before expensive mapper) remains the highest-impact optimization. Estimated 50%+ cost reduction with minimal quality loss. This requires more design work.

[Blevene]

Pre-filter Implementation — Shipped as Default

What it does

After the retriever assigns candidate techniques to spans, a pre-filter limits how many spans can proceed to the expensive LLM mapper per candidate technique. Default: max 2 spans per technique (max_spans_per_technique=2).

Example: if 24 spans all have T1588.001 as their top retriever candidate, only the 2 with the highest retriever scores go to the mapper. The rest are redundant — the consolidator would merge their claims anyway.

Benchmark (same 1.2MB Unit42 PDF, 4 runs)

Config	Cost	LLM Calls	Techniques	Step Confidence	Coherence
Batch=10 (old default)	$0.367	28	61	90.6	100%
Batch=25 (no filter)	$0.274	17	57	91.0	100%
B25 + span dedup (opt-in)	$0.210	13	47	88.2	100%
B25 + pre-filter (new default)	$0.198	13	46	91.7	100%

Why pre-filter > span dedup

Both produce similar technique counts (~46-47), but the pre-filter has 3.5 points higher step confidence (91.7 vs 88.2). It selects better evidence per technique rather than just removing duplicate text. The mapper gets the highest-scoring retriever matches, producing more confident technique assignments.

Quality tradeoff

• 46 techniques vs 57 without filtering (19% fewer)
• But the 46 are the best-evidenced — higher confidence, same kill chain coherence
• 29 techniques are stable across all 4 configurations (the "core set")
• The ~11 "lost" techniques are marginal matches that vary between runs due to LLM non-determinism

Configuration

# Default (on): keep best 2 spans per candidate technique
config = {}  # max_spans_per_technique defaults to 2

# Disable pre-filter for exhaustive extraction
config = {"max_spans_per_technique": 0}

# More aggressive filtering
config = {"max_spans_per_technique": 1}

# Combined with span dedup for maximum cost reduction
config = {"enable_span_dedup": True, "max_spans_per_technique": 2}

Cost summary (per 1.2MB report)

Optimization	Cost	Savings vs old default
Old default (batch=10)	$0.367	—
New default (batch=25 + pre-filter)	$0.198	46%
+ span dedup (opt-in)	~$0.15	~59%

Commits: c5ffa2e (extraction_pipeline.py + optimized_chunked_extractor.py)