Skip to content

AGENT_E2E.md

Latest commit

 

History

History
139 lines (107 loc) · 8.21 KB

AGENT_E2E.md

File metadata and controls

139 lines (107 loc) · 8.21 KB
Raw

Real coding-agent E2E

Plain API chat does not prove coding-agent compatibility. FreeNIMAPI's pass gate runs installed client binaries against a deliberately broken repository.

This behavioral flow became the reference for the FreeAPI Agent Acceptance Standard (FAAS) v1.0 on 2026-07-16. Historical FreeNIMAPI runs predate the complete FAAS provenance schema and are not retroactively presented as current N4 evidence. The public sanitized status is available in evidence/agent-qualification-2026-07-17.json.

Fixture task

Each client receives a fresh copy of fixtures/buggy-invoice and this user-level task:

  1. state one short high-level plan without private chain-of-thought;
  2. inspect the repository;
  3. run npm test and observe the initial failure;
  4. diagnose and fix production code without editing tests;
  5. rerun tests until green;
  6. create PROOF.txt with an exact sentinel;
  7. complete the final agent turn.

The visible plan helps a screen recording explain the workflow, but it is not accepted as evidence. Only completed tool callbacks and independent checks can satisfy the pass gate.

The runner then checks all of the following independently:

  • a cold-start compatibility preflight reproduces Hermes local-server discovery, lists the model catalog, and retrieves the selected model through GET /v1/models/{model} before the first generation;
  • a runner-owned proxy emits zero unexpected WARN or ERROR entries during startup, model discovery, tool preflight, and the complete agent turn;
  • client process exit code is zero;
  • a real command/edit tool appears in the client trace;
  • production source changed;
  • the complete test/ tree and package.json hash are unchanged;
  • the proof file is exact;
  • an external immutable node --test test/invoice.test.js run passes in a separate no-network verifier sandbox;
  • the trace proves observed failure → edit → observed green order from completed command/tool results with real exit codes, never from agent narration;
  • the client's final-turn event exists.

Isolation

scripts/agent_e2e.js runs one case at a time with:

  • fresh HOME, TMPDIR, XDG directories, and workspace;
  • explicit environment maps instead of inheriting the parent environment;
  • absolute client binary paths;
  • a dummy/separate local proxy key only;
  • a deny-default macOS Seatbelt wrapper that limits reads/execution to required system/Homebrew runtimes, exact client resources, and the run root; limits writes to the run root; denies Keychain services and cross-process inspection; and allows network only to the exact ephemeral FreeNIMAPI loopback port;
  • a separate verifier profile with no network access and no local proxy credential for both the initial RED and final independent test;
  • confined evidence reads that reject symlinks, hard links, special files, and path escapes;
  • disabled browser, web, sharing, plugins, project config, and auto-update paths where supported;
  • no NVIDIA_API_KEY in any client process.
  • process-group termination before evidence is read, followed by another protected-contract hash after the independent verifier run.

The current runner deliberately refuses non-macOS real-client execution until a platform-equivalent OS sandbox is implemented. Client paths can be overridden with CODEX_BIN, CLAUDE_BIN, OPENCODE_BIN, and OPENCLAW_BIN. The child PATH is fixed to the current Node/npm runtime, Homebrew tools required by installed clients, and system binaries; it is not inherited from the user shell.

Codex is invoked with its internal sandbox set to danger-full-access only because its entire process tree is already inside the runner's stricter Seatbelt profile. macOS rejects a nested sandbox-exec layer. This does not grant Codex access beyond the outer profile's temporary workspace and loopback-only network boundary.

Current installed clients targeted by the runner:

Client Version verified Wire API
Hermes Agent 0.16.0 Chat Completions
Codex 0.144.2 Responses
Claude Code 2.1.169 Anthropic Messages
OpenCode 1.17.18 Chat and Responses
OpenClaw 2026.6.1 Chat, optional

Hermes 0.16.0 is supported through a credential-free runtime snapshot. The runner copies only allowlisted source/data files into the fresh run root, rejects .env, auth.json, databases, logs, sessions, cookies, symlinks, and hard links, and uses a fresh HERMES_HOME. It deliberately never starts Hermes from the existing source checkout because that checkout's normal startup loads its repository .env. The installed Python runtime and packages are read-only dependencies; the Hermes process still runs inside the same Seatbelt, fresh-home, loopback-only boundary as the other clients.

Commands

# Starts an ephemeral authenticated proxy and runs all default clients.
npm run test:agents

# Strict Hermes callback evidence.
node scripts/agent_e2e.js \
  --clients hermes \
  --model nim-auto \
  --keep \
  --report-dir reports/e2e-live-hermes-evidence

# Select several clients or another model.
node scripts/agent_e2e.js \
  --clients hermes,codex,claude,opencode-chat \
  --model nim-auto

# Real, recognizable Hermes classic CLI for a screen recording.
# This is a visual demo; run the strict JSONL command above for evidence.
node scripts/agent_e2e.js \
  --hermes-cli-demo \
  --model nim-auto \
  --keep \
  --report-dir reports/e2e-live-hermes-cli-recording

# Use an already-running proxy. The local key must match.
FREENIM_E2E_LOCAL_KEY=local-e2e \
node scripts/agent_e2e.js \
  --base-url http://127.0.0.1:3000 \
  --mode scripted-client-transport

# Deterministic protocol/client loop with a scripted mock model.
npm run test:agents:mock

# The same strict nominal for the real installed Claude Code client only.
FREENIM_MOCK_CLIENTS=claude npm run test:agents:mock

The mock run is explicitly labeled scripted-client-transport. It proves that real clients can receive tool calls, execute them, return tool outputs, and finish through the proxy. It never counts as native-model evidence. Live mode requires the runner-owned proxy, verifies the official NVIDIA upstream origin, and refuses --base-url; an external loopback proxy can only be labeled scripted transport.

Claude Code connects to FreeNIMAPI as an Anthropic Messages gateway. The recommended environment uses a separate local bearer token, never the NVIDIA credential:

export ANTHROPIC_BASE_URL=http://127.0.0.1:3000
export ANTHROPIC_AUTH_TOKEN="$FREENIM_LOCAL_API_KEY"
export ANTHROPIC_MODEL=nim-auto
export ANTHROPIC_DEFAULT_OPUS_MODEL=nim-auto
export ANTHROPIC_DEFAULT_SONNET_MODEL=nim-auto
export ANTHROPIC_DEFAULT_HAIKU_MODEL=nim-auto

The bridge accepts Claude Code's current mid-conversation system messages and translates them to ordered Chat Completions system messages before the NVIDIA request. The 2026-07-17 Claude Code 2.1.169 scripted nominal exercised this path after the intentional RED command and passed every behavioral gate.

For Hermes, strict evidence comes from callback-emitted JSONL events (tool.started, tool.completed, and turn.completed) produced by the real AIAgent. Narration cannot satisfy the evaluator. The optional --hermes-cli-demo mode instead opens the real classic Hermes CLI in the same clean runtime and mirrors its concise tool activity and inline diffs to the terminal for recording. That visual mode is intentionally labeled visual-hermes-cli-demo; its raw transcript is not persisted, and it does not replace the strict callback-based nominal.

Current Hermes qualification (2026-07-16): scripted-client-transport PASS; live NVIDIA NOT RUN because NVIDIA_API_KEY was absent from the launching process environment. The scripted result proves the real Hermes binary/runtime, FreeNIMAPI protocol path, file/terminal tools, and verifier workflow. It does not prove NVIDIA model quality.

Reports

Sanitized reports are written under reports/e2e, reports/e2e-mock, reports/e2e-live*, or reports/live and are excluded from git and npm packages. Reports include evidence mode, proxy/upstream provenance for owned live runs, and exact client version. Client output is redacted for likely bearer keys, API tokens, and the exact runtime local key. The synthetic fixture contains no private data.