pit

A minimal coding/general agent you fully own, plus an empirical lab for the harness itself.

The agent is one ReAct loop, a small set of plain-function tools, and raw OpenAI chat-completions over httpx. No SDKs, no framework, no plugin system: roughly 800 lines of Python you can read in an afternoon. It runs on Windows against a local llama-server or any OpenAI-compatible endpoint.

Why own the loop

Every behavior of the agent is a line of code you can point to, measure, and change. That mattered twice here:

1. The eval forced harness fixes a framework would have hidden. A python_eval tool (no shell, no quoting hell) turned one model's 40-step failure loop into a 9-step pass. A loop-breaker stops a model from burning its step budget repeating one dead search. Deletion commands are hard-blocked in the tool layer (three layers deep, including a runtime hook injected into any Python the agent spawns), not discouraged in the prompt.
2. The harness becomes a research instrument. Because pit logs every message, tool call, and per-turn generation time to a session jsonl, context-management ideas stop being opinions and become controlled experiments.

The research layer

pit has two modes, switchable at runtime with /mode:

• managed (default): the researched setup. Tool-aware spill (big unbounded shell dumps go to a scratch file with a re-fetchable handle) plus lazy batch memory (at 24k chars, one model pass files facts/tasks/decisions into a typed store and resets the buffer).
• regular: the vanilla baseline. No spill, plain auto-compaction near the window limit, like a stock agent.

The same task can be run in both arms and graded by hard mechanical gates (grade_gen.py), with cost measured as model wall-clock, not raw tokens. The full story is in LAB_NOTEBOOK.md; headline findings, all first-party measurements on a local Qwen 3.6-35B over llama.cpp:

• The "obvious" fix was a trap. Stubbing stale tool results in place looks right and is wrong: any mid-context edit gets 0% KV-cache reuse (~4s re-prefill), while appending reuses ~95%. Only measurement caught it.
• Spill-to-file works, but only tool-aware. Spilling a redundant shell dump cut context 62% with quality held; spilling document reads broke completeness tasks via a spill cascade. The discriminator is tool type, not size.
• A small local model will not adopt a new retrieval tool just because it exists and the prompt suggests it (a recall tool: 0 uses). Behavioral fixes must live in the harness.
• Decode is not context-flat: roughly -1.7% per 10k tokens of depth (56.5 to 41.9 t/s from ~2k to ~87k context). Lean context buys decode speed, not just prefill.
• Methodology mattered most. A "confirmed" single-run win became a quality regression under N=3 reps and a strict gate. Distributions plus a hard correctness gate are load-bearing.

The model itself was picked by eval, not vibes (EVAL_AND_SETUP.md): across ~25 model/quant configs at 26-35B, quants tied on quality, so the tiebreak is speed. The general-agent task suite (GEN_SUITE.md) regenerates deterministically via python _gen_build/build_all.py.

Quickstart

pip install httpx        # the only dependency (Python 3.11+)
python pit.py            # REPL, managed mode; attaches to :8080 if a server is up
python pit.py --mode regular           # vanilla baseline
python pit.py -p "list the .py files"  # one-shot: run one turn, print, exit

Point it at any OpenAI-compatible endpoint by editing models.json. Each entry: base_url, model, optional api_key_env (bearer token from that env var), optional extra_body (merged into the request), optional bat (a Windows launcher run on /model; %VARS% are expanded). In the REPL: /mode, /model, /models, /tools, /memory, /compact, /clear, /quit. Any other /word injects commands/word.md as a prompt: dropping a markdown file in commands/ is the whole skills system. If the current directory has a CLAUDE.md, it is appended to the system prompt. Sessions log to sessions/*.jsonl; run python metrics.py on one to get the cost/fluff report.

Safety

Deletion (rm, Remove-Item, git reset --hard, DROP TABLE, os.remove, ...) is hard-blocked at the tool layer, at script-write time, and by a runtime guard imported into every spawned Python process. The agent moves unwanted files to junk/ instead.

Status and limitations

Working daily driver, honest caveats: the model is the ceiling on hard multi-file work (a frontier agent is not close to threatened); most experiments are N=3, real signal but not statistical proof; the batch-memory consolidation exists but has not been rigorously A/B'd yet. Raw experiment logs and session transcripts are local-only and not part of the repo; the curated results live in LAB_NOTEBOOK.md and EVAL_AND_SETUP.md.

License

MIT.