Field Fleet / Field Canary

Field Fleet runs standing OpenClaw canaries on stable, beta, and dev tracks. A member is enrolled once, retains its state indefinitely, upgrades in place, and accumulates migration history across daily cycles. It is Kova's long-lived field lane—not a replacement for Kova's isolated scenario and release-gate coverage.

The design builds on work by Shakker and Eisenh0rn40. Kova report compatibility and Crabline deterministic channel fixtures make the evidence usable by the existing ecosystem.

MVP behavior

• Fixed execution order: upgrade → smoke → channel → provider → soak.
• Each member is bounded to eight channel and eight provider lanes so all contract-valid retained evidence fits the crash-safe manifest envelope.
• run is a dry plan unless --execute is explicitly present.
• Commands are argv arrays. There is no shell parsing or shell-string mode.
• Child processes receive a small constructed environment; ambient credentials are not inherited. Execution is Linux-only; timeouts terminate the adapter process group. Trusted adapters must never daemonize, call setsid, or leave background processes behind.
• State and evidence roots must be absolute, non-overlapping, sufficiently narrow, and free of symlink components. State trees containing symlinks or special files fail closed.
• Enrollment creates an append-only identity marker and a read-only canary. Linux unnamed temporary files make canary, marker, and cycle-anchor publication atomic and no-replace; a fully published canary can recover an interrupted initial marker publication only while state and cycle history remain pristine. Enrollment pins the initial state fingerprint. Every run adds an immutable state-side cycle anchor carrying the pending intent, compares the prior after fingerprint to the new before fingerprint, and revalidates the complete anchor chain after adapters exit. Either side of the anchor/ledger crash window is recoverable; moving or losing evidence history cannot silently restart an established member.
• There is deliberately no delete, prune, reset, restore, rollback, or cleanup operation. Pending/final hash-chained ledger events are immutable, atomically published files, so interrupted cycles recover deterministically as failure evidence on the next invocation.
• Public kova.report.v1 JSON, Markdown, compact summary JSON, and dark HTML contain statuses, fingerprints, matrix identity, byte counts, and SHA-256 digests. A bounded Kova-style artifact index accompanies them. Command arguments, environment, diagnostics, and raw output remain in 0600 bounded artifacts. A digest-bound private manifest is reverified before every later cycle; invalid or over-envelope adapter artifacts produce degraded, tree-hashed failure evidence instead of wedging the pending cycle.

Build and use

Install the tagged release with:

go install github.com/VACInc/field-fleet/cmd/field-fleet@v1.1.0

Or build both binaries from a clone:

go build -o bin/field-fleet ./cmd/field-fleet
go build -o bin/fixture-driver ./cmd/fixture-driver

bin/field-fleet validate --config configs/example.json
bin/field-fleet plan --config configs/example.json
bin/field-fleet enroll --config configs/example.json --member stable-canary
bin/field-fleet enroll --config configs/example.json --member stable-canary --execute
bin/field-fleet run --config configs/example.json --member stable-canary
bin/field-fleet run --config configs/example.json --member stable-canary --execute
bin/field-fleet render --input /path/report.json --output /path/report.html --execute

configs/example.json is an operator-facing stable/beta/dev fleet. Its adapter path is intentionally illustrative and is never invoked by tests. Replace it with an owned executable that implements the documented receipt contracts. fixtures/e2e.config.json and cmd/fixture-driver are hermetic test assets; they never touch a real OpenClaw runtime, channel, service, container, port, or scheduler.

Enrollment is stateful. Review paths before running it. Re-running enrollment only validates the existing immutable identity; it does not recreate or wipe state.

Adapter contracts

Each command must create {contractPath} as a regular 0600 file before it exits. Contract paths are opened no-follow and nonblocking so a FIFO or other special file cannot strand the member lock outside the adapter timeout:

• upgrade: current kova.run.receipt.v1; its jsonPath must remain under the private phase artifact directory and the referenced report is fully checked;
• smoke/soak: current kova.report.v1 identity, summary, and records;
• mock/live channel: current Crabline single-result or suite JSON plus a fieldClass assertion (mock or live) bound by the Field Fleet adapter;
• provider: fieldfleet.provider.receipt.v1 with exact engine, model, quantization, tool-calling mode, and boolean ok.

Supported fixed placeholders are {stateDir}, {artifactDir}, {contractPath}, {memberId}, {track}, {cycle}, {laneId}, {channelMode}, {engine}, {model}, {quantization}, and {toolCalling}. Unknown placeholders fail before execution.

Each channel lane pins its Crabline fixture id, provider id, command mode, and mock/live field class. A receipt for another lane cannot satisfy it. Any failed mandatory phase blocks every later phase without invoking its adapter. Missing or malformed required receipts are INCOMPLETE; an adapter or validated assertion that actually runs and fails is FAIL; dependent work is BLOCKED.

Adapters are trusted operator-supplied integration code. Field Fleet constrains their interface and evidence, but it is not an untrusted-code sandbox; use the Observatory/Crabbox VM boundary for hostile skills and plugins.

The generated report's fieldFleet extension carries channel mock/live identity and Ollama/vLLM/llama.cpp/hosted matrix receipts while remaining a valid kova.report.v1 document.

Operations

Scheduling is intentionally outside this binary. After validating real adapter commands manually, use an ordinary systemd timer or equivalent deterministic scheduler for daily upgrades and nightly checks. Never point test fixtures at a real member state directory.

Field Fleet has no deployment-provider or Crabbox dependency. A standing member is expected to live on an ordinary persistent VM or equivalent durable host.

Run verification with:

gofmt -w cmd internal
go test -race ./...
go vet ./...