An empirical search for the best context-management setup for a local coding/general agent. Treating the scaffold around the LLM as an ML-research problem: hypothesize → run a controlled A/B → measure → keep or kill. Kept as a running log; findings that generalize are flagged 📌 FINDING so they're easy to lift out later.
- Agent: pit — minimal owned harness (~800 lines, httpx-only). Tools, ReAct loop, skills.
- Model: Qwen 3.6-35B-A3B Uncensored Q4_K_P on llama.cpp (llama-server, :8080).
- Hardware: RTX 5060 Ti 16GB + Ryzen 9 9950X + 128GB DDR5.
- Measurement rig:
cache_probe.py(KV-cache behavior via servertimings.prompt_n/cache_n);metrics.py(per-session FLUFF / MISSING / COST / tool-share from the jsonl trace); the_gen/_drivetask suite with hard graders for the quality gate.
An LLM harness is a stochastic, emergent system — you can't derive the right context policy a priori, only measure it. Proof from our own logs: the "obvious" hardcoded fix (stub stale tool results in place) looked correct and was wrong — it silently torches the KV cache. Only measurement caught it. So: every context-management idea is an experiment, not a decision.
📌 The seam we're working in: the agent-memory literature ignores KV-cache economics; the inference-cache literature ignores memory curation. A cache-aware memory manager — curation whose every move respects prefix-stability — is largely unexplored. That's where any real breakthrough here would live.
Constrained optimization: minimize cost, subject to quality ≥ baseline (no regression).
- Quality gate (hard constraint): every grader gate the baseline passed must still pass. A token/time win that drops any correctness gate is DISQUALIFIED. Also guards against the model getting lazy to save tokens.
- Cost (objective) = total MODEL WALL-CLOCK to solve the task, not raw tokens. Raw tokens
overcount because (a) cached re-sent tokens (~99% prefix reuse) skip prefill and cost ~nothing,
and (b) tokens aren't uniform — decode slows with depth, prefill ≫ decode. Wall-clock captures
all of it and is what the user feels. Instrumented via per-turn
gen_secondsmeta records (contention-free while the GPU is dedicated). Tokens-computed logged as a hardware-agnostic backup. - Distributions, not points: each config runs N=3–5×; compare median cost + quality pass-rate (the model is nondeterministic — single runs mislead, cf. Exp 2).
- "Best" = the Pareto knee: lowest median model-time that holds 100% baseline quality; ties break toward SIMPLICITY (fewer moving parts). One line: the simplest config with the lowest median model-time that never drops a correctness gate, over repeated runs.
Q: what context moves are cheap vs catastrophic on THIS model?
| move | KV reuse | prefill |
|---|---|---|
| append at end | 95.4% | 716ms |
| edit an EARLY token | 0% | 4200ms |
| edit near the END | 80.6% | 1014ms |
| delete a MID block | 0% | 4146ms |
📌 FINDING: on Qwen 3.6, KV cache reuses the prompt only up to the first changed token. Append is ~free; touching early tokens or deleting mid-context = full re-prefill (~25-75× costlier per turn than an append). KILLED: in-place tool-result stubbing, mid-context eviction. VIABLE: spill-to-file, suffix-only curation, epoch-rebuild.
Big tool dumps (90% of context) → keep a 4k preview in-context, spill the full body to a scratch file, return a re-fetchable handle. Append-only ⇒ cache-safe. A/B on log-diagnosis:
| baseline | spill on | |
|---|---|---|
| context | 30,097 tok | 11,431 tok (−62%) |
| fluff | 25,003 tok | 6,670 tok (−73%) |
| grade | PASS | PASS (held) |
| missing | 0 | 0 (re-fetchable) |
📌 FINDING: spill-to-file cuts context 60%+ in the giant-dump regime with zero quality loss, because the body stays re-fetchable (anti-fluff AND anti-missing). Regime-specific: in the multi-turn/small-result regime the model self-limits tool output, so spill is a mild win (−26%), never a loss.
Revealed the multi-turn cost isn't fluff — it's the model running the same expensive grep 6× across turns (cross-turn re-fetch). 📌 FINDING: the per-turn loop-breaker resets each turn, so cross-turn repetition slips past it. Two failure modes have different profiles: one-shot deep reads = FLUFF (giant dumps); multi-turn = MISSING (re-fetch).
Exact-match (name+args) cache to stop re-fetch. Result: 0 cache hits. Two causes:
(1) BUG — mutation-detector matched bare > i.e. comparison operators in analysis code, wiping the
cache every call; (2) DESIGN FLAW — real re-fetches VARY in exact args (same target, different
command), so exact-match structurally can't catch them.
📌 FINDING: you don't fix re-fetch by caching source output. Real re-fetches vary, and
serving stale source is risky. Fix it on the RETRIEVAL side — let the model recall its earlier
conclusion. → motivates Exp 3. Cache demoted to opt-in.
A/B recall off vs on, same 4-turn task. Result: the model called recall 0 times even when the tool was available + a system-prompt hint told it to. Re-touch 2 vs 1 (noise; recall never fired). Quality held (both diagnosed correctly). 📌 FINDING: a small local model does NOT adopt a new retrieval tool just because it exists + a prompt nudge — it defaults to its habit (re-grep). So the fix for re-fetch, IF we want one, must be HARNESS-driven (auto-detect a cross-turn re-fetch and inject the prior finding), not model-driven. But: re-fetch is intermittent (0/1/2/5/6 across runs) and small (1-2 medium results), so its cost is minor. DECISION: deprioritize re-fetch. Both model-facing fixes (Exp 2 cache, Exp 3 recall) failed to engage; the juice isn't worth it. recall stays available (harmless) but off the critical path. The big lever remains SPILL (Exp 1).
Agentic retrieval: a recall(query) tool that keyword-searches the model's OWN prior findings
this session (assistant conclusions weighted 2× over raw tool output) so it retrieves what it
already learned instead of re-querying. Honest on gaps ("nothing matches — go fetch it"), so it
can't serve stale data. A/B: recall off vs on, same 4-turn re-grep task. Watching: does the model
use it, does re-touch drop, does grade hold. [result pending]
Static audit of what pit actually sends: system prompt byte-stable across builds (no
timestamps/UUIDs/volatile tokens), tools schema byte-stable + deterministic order, messages
append-only with system first. Empirical confirmation from live server logs: successive turns
show selected slot by LCP similarity, sim_best = 0.994–0.999 and graphs reused = 68k–71k —
i.e. ~99% cross-turn prefix reuse in a real session.
📌 FINDING: pit is prefix-stable by construction (append-only ReAct loop). The only
cache-breaking events are deliberate epoch-rebuilds (batch consolidation) and /model switches. The
naive in-place pruner (Exp 0, killed) was the one thing that violated this. So the design is
already right — the discipline is "never edit early tokens," which the loop already honors.
Pure measurement, exp4_decode_curve.py, deterministic:
| ctx tok | decode t/s | % of peak | prefill t/s |
|---|---|---|---|
| 1.7k | 56.5 | 100% | 626 |
| 17k | 54.6 | 97% | 764 |
| 35k | 52.2 | 92% | 777 |
| 52k | 49.3 | 87% | 779 |
| 69k | 44.9 | 80% | 770 |
| 87k | 41.9 | 74% | 761 |
📌 FINDING: decode falls ~LINEARLY with context depth, ≈ −1.7%/10k tokens (56.5→41.9 t/s, 0→87k). Prefill throughput stays flat ~760-780 t/s (compute-bound per token). The old "SWA makes long context flat/free" claim is WRONG for Qwen 3.6. Consequence: context management buys BOTH prefill savings AND decode speed — every 10k of context you don't carry is ~1.7% faster generation on every token thereafter. (TODO: fix the stale claim in the older notes.)
gen5 long-doc extraction (guarantees a big read). Objective: min gen-seconds s.t. quality holds.
| arm | median gen | median ctx | quality-hold (7figs+3acts) |
|---|---|---|---|
| spill OFF | 57.0s | 22,346 tok | 3/3 |
| spill ON | 52.3s | 10,941 tok (−51%) | 2/3 ❌ |
📌 FINDING (important, only N-reps + a completeness task revealed it): spill is NOT an unconditional win. On this extraction task it cut context 51% and gen-time ~8%, but dropped quality on 1 of 3 reps — so by our rule (zero quality regression) it FAILS the gate here. Root cause = a spill cascade bug: reading a spilled file re-spills it (>4k preview again), so the model chases 4k fragments through .scratch/ and never sees the whole doc → misses 2 figures + 1 action. Passing reps got lucky going back to the original file. Refined model of spill: great on REDUNDANT output (diagnosis — you want the conclusion, Exp 1 −62% quality-held), risky on COMPLETENESS output (extraction — you need every item) BECAUSE of the cascade. Fix shipped: never re-spill a read of a .scratch/ file. → Exp 8 re-tests. LESSON: a "confirmed win" (Exp 1, N=1) became regime-dependent under N=3 + a strict gate. This is why the methodology (distributions + hard quality gate) exists.
Cascade fix (don't re-spill .scratch reads): quality restored to 3/3, BUT context savings evaporated (20,141 tok ≈ spill-off's 22,346) and gen-time got WORSE (82.8s vs off's 57.0s) — because recovering the full doc pulls it all back in plus extra round-trips. 📌 FINDING: for a completeness task, spill can only add overhead — its premise (body is redundant) is false. The real discriminator is TOOL TYPE: spill helps unbounded redundant shell dumps (bash/grep, Exp 1), hurts document reads (read_file, which already has cap+pagination). → tool-aware spill (Exp 9).
N=3 on the task that broke: quality 3/3 AND fastest of all configs (median 46.1s vs spill-off 57.0s). Context ~same as off (22k — correct: the document read stays in-context) but intermediate grep dumps still spill, so it's actually faster than off with no quality risk. 📌 FINDING: tool-aware spill (spill unbounded shell output, never read_file/grep/glob which are already capped) resolves the regime problem — extraction 3/3 @ 46s, and (Exp 1 regime) diagnosis still spills its big bash-grep dumps. → Exp 10 confirms the diagnosis win persists.
| config | median gen | median ctx | quality |
|---|---|---|---|
| spill OFF | 153.2s | 10,117 tok | 3/3 |
| spill TOOL-AWARE | 103.9s | 9,803 tok | 3/3 |
| Both 3/3. 📌 HONEST CAVEAT: context ~tied this run — the model self-limited to small greps (no | |||
| giant dump to spill), so Exp 1's −62% is CONDITIONAL on a giant dump occurring. Timing variance is | |||
| large at N=3 (off 70–215s, tool-aware 92–370s), so the time delta isn't statistically firm — treat | |||
| tool-aware as "quality-safe + trends leaner/faster, big win when a giant dump happens, harmless | |||
| otherwise" = a free option worth defaulting on. |
The post claimed F16 KV beats quantized KV by +24% prefill for our model class at 65k (on AMD/ Vulkan). Tested F16 vs our q4_0 on Nvidia 5060 Ti + --fit, both arms measured back-to-back same session (cache_probe + decode curve). RESULT — it REVERSES on our hardware:
- Prefill: F16 ~17% SLOWER at every context (opposite of his +24%). Cause: --fit reserves KV for the full 128k at load, so F16's 4× cache forces MORE experts onto CPU; that offload penalty swamps the dequant savings. His AMD setup had different VRAM/offload dynamics.
- Decode: crossover — q4_0 wins below ~52k (62→54 vs 56→51 t/s), F16 wins above (87k: 44.8 vs 41.9, +7%) because F16 skips per-token dequant so it retains decode better at depth. 📌 FINDING: on Nvidia + --fit + 128k reservation, keep q4_0 KV — it wins prefill everywhere AND decode in the low-moderate range where our lean-context strategy keeps us. F16's only edge is decode >52k, the regime we avoid. Craft point: the viral post's headline finding literally reversed on different hardware (exactly as its own caveat warned) — first-party measurement > viral tables. Our existing config was already right.
After 10 experiments (1 headline win, several nulls, one win that survived only after being refined). The rule that defines "best": simplest config with lowest median model-time that never drops a correctness gate, over repeated runs.
- Tool-aware spill — ON (default). Spill UNBOUNDED shell output (bash/powershell/python_eval) to a scratch file + a re-fetchable handle; NEVER spill read_file/grep/glob (already capped) or a .scratch read (cascade). Quality-safe in both regimes (extraction 3/3, diagnosis 3/3); −62% context when a giant redundant dump occurs; harmless otherwise. A free option.
- Append-only, prefix-stable layout. Stable prefix (tools → system → memory), volatile last, never edit early tokens. Already true by construction (~99% cross-turn cache reuse, Exp 6). The discipline: no mid-context edits/eviction (Exp 0 killed in-place stubbing: 0% reuse, 4s hit).
- Re-fetch tooling — NOT on the critical path. Content cache (Exp 2) and recall() tool (Exp 3) both NULL: the model won't adopt recall, exact-match cache can't catch varying re-fetches. Re-fetch is intermittent + small. recall() stays available (harmless, opt-in) but we don't rely on it.
- Consolidation — lazy/idle batch (research-backed; the batch-memory exists but was OFF during these experiments; not yet A/B'd — the one open item worth a rigorous test next).
- Retrieval philosophy — keyword + agentic, no vectors/graph (retrieval research pass; no infra earned).
- 📌 The naive fix was a trap. Stubbing stale tool results in place looks obviously right and is wrong — mid-context edits get 0% KV-cache reuse (4s re-prefill). Only measurement caught it.
- 📌 Spill-to-file works, but only tool-aware. Spilling unbounded redundant shell dumps is a big win (−62% ctx, quality held); spilling document reads breaks completeness tasks (a spill CASCADE traps the model chasing 4k fragments). The discriminator is tool type, not size.
- 📌 A small local model won't adopt a new retrieval tool just because it exists + a prompt nudge (recall: 0 uses). Behavioral fixes for local models must be harness-driven, not model-optional.
- 📌 Decode is NOT context-flat (Qwen 3.6): −1.7%/10k tokens (56→42 t/s by 87k). Lean context buys decode speed, not just prefill.
- 📌 Methodology mattered more than any single result. A "confirmed" N=1 win (spill −62%) became a quality REGRESSION under N=3 + a strict gate. Distributions + a hard quality gate are load-bearing.
"SWA makes long-context generation flat/free" — WRONG on Qwen 3.6. Server logs: ~56 t/s at ~0 ctx vs ~40 t/s at ~58k ctx (pure decode). 📌 FINDING: decode has a real ~28% tax by 58k, climbing with depth → context management buys DECODE speed too, not just prefill. (Full curve: Exp 4.)
- Exp 4 — map decode-vs-context curve (0/20/40/60/80/100k) → headline chart, fix the "flat" claim.
- Exp 5 — SPILL_CHARS sweep: what preview size best trades fluff vs grade?
- Exp 6 — cache-friendly-layout audit: is pit's actual prompt prefix-stable (deterministic tool order, no timestamps/volatile data early)? Possible big win (Exp 0 says 25-75×).
- Exp 7 — full stack (spill+recall+idle-consolidation) vs bare baseline across the whole _gen suite: the convergence test for "best setup" — max cost cut at zero grade loss.
Trigger (real interactive run, 2026-07-06): the aggressive-uncensored finetune got a plain
everyday Windows-settings question and, instead of answering, confabulated a nonexistent
training-env path (C:/Users/ghs12/.gnutools/skills/vibe-coding/skills/skill-english/1000 -
clearly a leaked SFT artifact, not a path from this machine or the request) and
burned all 15 tool calls hunting for it. Never answered. Re-run answered fine → INTERMITTENT.
Even the good run hallucinated registry keys + a fake Restart-Process cmdlet.
Root cause: model, not harness. Leaked SFT-environment artifacts + degraded instruction-following from the "Aggressive" uncensoring. The exact-repeat loop-breaker missed it because every call had DIFFERENT args (different path / maxdepth) — it only catches byte-identical repeats.
📌 FINDING: the failure CLASS is "no progress," not "bad path." Guardrail must detect lack of progress (not specific strings), enforce in the harness (Exp 3: prompt nudges are ignored), and soft-correct before hard-stop (legit debugging also fails commands → blunt kills = false positives).
Built (guardrails.py, env-gated PIT_GUARDRAILS, default ON): a ProgressMonitor with 3 signals
over a rolling window — (a) unproductive-step streak, (b) near-repeat on normalized target (catches
reworded dead ends), (c) PROVENANCE guard: a failed op on a filesystem path with no provenance
(not in the request, not in prior successful output, doesn't exist) is weighted heavily = the sharp
instrument for confabulation. Tiered intervention: 1st trip → inject a re-ground correction + reset;
2nd trip → hard stop with a "likely model confabulation, try /model qwen" message.
Verified: (1) offline replay of the real confab trace → reground→stop deterministically,
offender correctly identified; legit debug sequence (read→failing test→edit→passing test) → 0 trips.
(2) LIVE run of the original prompt → guardrail caught a DIFFERENT stall (a command failing twice),
re-grounded, and the model RECOVERED and answered in 5 steps instead of spiralling. Generalized to an
unforeseen failure. (3) Path-extractor bug found in the live run (grabbed the /nologo flag as a
path) → fixed: driveless tokens need ≥2 separators or a file extension to count as a path.
📌 FINDING: guardrails = containment + graceful recovery (worth having under ANY model); they do
NOT cure the confabulation. Root-cause fix is a steadier model (censored qwen Q4_K_S). Do both.
Open: A/B PIT_GUARDRAILS on/off across the _gen suite — confirm zero false-trips on legit runs
(the false-positive risk is the only real cost). And still owe the /model qwen vs qwen-unc-q4
head-to-head on this exact prompt (needs a server reload).