HarnessLab

Lab → Factory for agent harness algorithms and multi-agent systems.

HarnessLab is a Python platform to experiment with pluggable agent harness algorithms and multi-agent coordination, and then generate production-ready multi-agent codebases (Python / TypeScript / Rust) from the configurations that worked. It turns an experiment platform into a development platform.

Because the engine is pure Python, algorithm updates need no recompilation — components are hot-loaded at runtime.

Capabilities

1. Algorithm library — common harness components organized by category (loop, compaction, memory, context, coordinator, scorer). Develop new ones or pick existing ones, then assemble an experiment agent in minutes.
2. Projects + multi-agent coordination — create a project, design several agents and a coordination topology, and run experiments.
3. Factory codegen — generate a complete, runnable multi-agent codebase in your chosen language from the experimented configuration.
4. Community market — a decentralized protocol to publish/consume algorithms and evaluation datasets (git / URL / local), with integrity, optional signing and capability consent. See docs/spec/package-manifest.md.

Architecture

harnesslab/
  engine/      # the Python agent runtime (loop, harness, LLM clients, tools, events)
  library/     # the categorized, hot-reloadable algorithm library + registry
  project/     # Project / AgentSpec / Topology model + SQLite store
  experiment/  # datasets, multi-agent runner, evaluator, self-improving autoloop
  factory/     # language-neutral IR + python/ts/rust emitters (Jinja2 templates)
  design/      # the Architect design assistant + deterministic generator
  market/      # decentralized package protocol (manifest) + client (install/pack)
  api/         # FastAPI app exposing everything under /api/v1
frontend/      # React + TypeScript + Vite UI (projects, library, design, experiments, factory, market)

The engine runs fully offline: when no LLM API key is set it uses a deterministic mock client, so the whole pipeline (loops, coordination, codegen, market) is reproducible in CI.

Quick start

python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

# run the API + open the UI (in another shell: cd frontend && npm install && npm run dev)
harnesslab serve

# or drive it from Python
python - <<'PY'
import asyncio
from harnesslab.project import Project, AgentSpec, Topology, ComponentRef
from harnesslab.experiment import evaluate
from harnesslab.factory import write_to_dir

proj = Project(name="Research Team", agents=[
    AgentSpec(name="planner", loop=ComponentRef(name="plan_and_solve")),
    AgentSpec(name="solver",  loop=ComponentRef(name="react")),
], topology=Topology(mode="coordinated",
                     coordinator=ComponentRef(name="sequential_pipeline"),
                     agent_order=["planner", "solver"]))

run = asyncio.run(evaluate(proj, "smoke_qa"))
print("accuracy:", run["accuracy"])
write_to_dir(proj, "python", "out")   # generate a runnable multi-agent codebase
PY

CLI

harnesslab serve                              # run the API server
harnesslab library --kind coordinator         # list components
harnesslab generate project.json rust out/    # codegen
harnesslab install file:./examples/packages/cool-loop   # install a market package
harnesslab trust acme s3cret                  # add a trusted signing key (HMAC)
harnesslab pack ./my-pkg --sign-key acme --sign-secret s3cret   # finalize + sign (HMAC)
harnesslab keygen                             # generate an ed25519 keypair
harnesslab pack ./my-pkg --sign-algo ed25519 --sign-secret <seed_hex>  # sign (ed25519)
harnesslab trust mypub <public_key_hex>       # trust an ed25519 public key
harnesslab install <source> --verify-signature # require a valid trusted signature
harnesslab generate project.json rust out/ --rust-client reqwest  # real Rust client

Market sources & trust

Packages are fetched from decentralized source specs — git+…, https://….tar.gz, file:… — with no central server required. An optional central index is supported via registry:name[@version] when HARNESSLAB_REGISTRY points at one or more index files/URLs (see examples/registry/index.json); decentralized sources remain the default. Installs always verify artifact integrity (sha256); detached signatures are verified against a trusted-keys store when present, and required when --verify-signature is passed. Two interoperable schemes are supported over one canonical content digest: HMAC-SHA256 (shared key) and ed25519 (asymmetric, pure-Python — publishers sign with a private seed and consumers trust only the public key, so no secret is ever shared).

Built-in algorithms

• loop: react, plan_and_solve, reflexion, tree_of_thought, rewoo, tool_calling, budgeted, reflect_retry (engine also enforces an optional max_total_tokens cost ceiling)
• compaction: none, summary, sliding_window, vector_retrieval
• memory: noop, in_memory_vector, sqlite, persistent_vector (on-disk, pluggable hashing/bow/openai embeddings)
• context: none, workspace_root, project_context, rag (retrieval-augmented over a corpus dir; ranks by bag-of-words or, with embedding=openai, a real model)
• coordinator: sequential_pipeline, supervisor, group_chat, debate, majority_vote, auction, blackboard, map_reduce, hierarchical, tournament, coscientist
• scorer: exact_match, f1, numeric, contains, mcq_letter, rubric_overlap, llm_judge, weighted_rubric
• tools: file_read, file_write, file_edit, shell, python, finish (a tool may set terminate to end the loop; loops also support steering / follow-up message injection). PolicyGate enforces sandbox profiles: path-escape denial, workspace-relative allowlists, and a network flag.

tournament (configurable judge + rounds) and coscientist (persistent research board with a proximity/dedup agent, optionally iterated for N rounds) are parameterizable via ComponentRef.params.

Embeddings (real model backend)

Vector memory and RAG use a pluggable embedding backend. hashing/bow are deterministic and offline; openai calls any OpenAI-compatible /embeddings endpoint over the stdlib (no extra dependency) and falls back to the local embedding if unreachable. Point it at a local model such as BGE-M3:

export HARNESSLAB_EMBED_BASE_URL=http://192.168.31.77:3333/v1
export HARNESSLAB_EMBED_MODEL=BGE-M3
# optional: export HARNESSLAB_EMBED_API_KEY=...

Then use persistent_vector / rag with embedding: openai. Verify the connection with GET /api/v1/registry/embeddings/status?probe=1.

Multi-language reference code

Every common loop, coordinator and scorer ships a curated, self-contained reference implementation in Python, TypeScript and Rust so a developer (or a code-generating agent) can copy/paste an algorithm directly:

GET /api/v1/registry/{kind}/{name}/reference   # -> { languages, snippets }

The Algorithm Library tab in the UI is a two-pane browser: search/filter the catalog on the left, and on the right see each component's description, parameter schema and a language-tabbed reference snippet with a one-click copy button. The catalog also exposes reference_langs per component so availability is visible at a glance. The same snippets back the Factory codegen, so the reference you copy matches what the generator emits.

UI

The desktop layout follows an experiment-platform/dev-tool convention: a fixed left sidebar groups the workflow into Build (Projects, Visual Designer, Design Assistant), Library (Algorithm Library), Evaluate (Experiments) and Ship (Factory, Market); a top bar carries the active-project context and engine status. The Experiments tab includes a cost/latency leaderboard.

Visual Designer (RAD + AI)

A live node canvas (@xyflow/react) that renders a project as agents, a coordinator and a scorer, auto-laid-out with dagre. It blends classic RAD visual building with structured editing:

• Palette → canvas drag-and-drop: drop a loop/compaction/memory/context component onto an agent slot, a tool onto an agent, or a coordinator/scorer onto the canvas.
• Inspector: schema-aware property editing (rename, model, prompts, token ceilings, slots, tools, topology, scorer).
• Every edit is expressed as a declarative Edit Op validated against the component + tool registries and applied transactionally (all-or-nothing) on the backend, so the model always stays runnable. Invalid sets are rejected with precise problems and never persisted.
• Live code preview regenerates the runnable Python codebase from the current project — "what you see is what you run." Canvas positions persist via a layout store kept separate from the runnable model.
• Run console + observability: run a task straight from the canvas, stream the event log, and see per-agent metrics (turns / tool calls / tokens / ms) overlaid on the nodes.
• AI Copilot: describe a change in plain language ("add a critic agent and switch to coordinated with debate") and the copilot proposes validated edit ops with a dry-run preview (diff/problems). Nothing is applied until you confirm, so the human stays in the loop and the project always stays runnable.
• Collaboration: realtime presence over SSE (one EventSource, no polling) shows who else is on a project; soft locks (auto-expiry) show who is editing which agent; an optimistic rev check rejects stale edits (409 → auto-refresh). For stricter control a client can take an exclusive edit lease — others go read-only (ops return 423) until it's released or expires.
• Run history & comparison: every run is recorded; the History tab lets you overlay a past run's per-agent metrics on the canvas or tick two runs for a side-by-side turns/tools/tokens/ms delta table and result diff.

See docs/visual-ide-design.md for the architecture and the Edit Ops protocol.

Examples (also run as e2e tests)

Two examples are modeled directly from real upstream systems (cloned, not reconstructed) and double as end-to-end tests in tests/e2e/:

• pi-agent — a single tool-calling coding agent modeled on badlogic/pi-mono's @earendil-works/pi-agent-core. It experiments through the engine and regenerates equivalent runnable code in Python/TypeScript/Rust. See examples/pi_agent/MAPPING.md for the line-by-line fidelity mapping.
• AI co-scientist — Google's multi-agent design (Generation → Reflection → Ranking → Evolution → Meta-review) via the coscientist coordinator, benchmarked on the real sc-HeurekaBench (mlbio-epfl/HeurekaBench) MCQ (50 Qs, mcq_letter) and OEQ (41 Qs, llm_judge) sets, each shipped as a market dataset package.

python -m examples.build                                    # write example project configs
harnesslab install file:examples/packages/sc-heureka-mcq    # install the MCQ test set
harnesslab install file:examples/packages/sc-heureka-oeq    # install the OEQ test set
pytest tests/e2e -q                                         # run both examples end-to-end

See docs/ROADMAP.md for the ongoing plan to enrich the component libraries and platform.

Tests

pytest -q

License

MIT