test(evals): tool-surface A/B — shipped sendMessage vs messaging-primitives post

[Poytr1]

What this is

The behavioral A/B comparing two agent-facing messaging tool surfaces, measuring task success rate and total token consumption (plus invalid-call rate, tool calls, wall time):

• Arm A — the production sendMessage surface, exactly as shipped.
• Arm B — the post façade from the messaging-primitives proposal (docs(design): messaging primitives — counter-proposal to the sendMessage rework #551): one tool, three orthogonal params (conversation / addresses / visibility). Every call compiles into exactly one legal sendMessage input and executes through the product tool, so both arms share one implementation and differ ONLY in the presented surface.

Full method, fidelity notes, pre-registration, and measured static costs: docs/designs/messaging-primitives-ab.md.

The three evaluation-only seams (all contract-tested, zero production behavior change)

1. executeProductTool — an evaluation-registry tool runs a product tool on the caller's own trusted SessionContext.
2. hideProductTools — withhold named product descriptors so each arm presents exactly one surface; a hidden tool stays executable (that is how the façade compiles down to it).
3. collaborationGuidance — the prompt-side complement: the standing collaboration guidance and parent-report append teach sendMessage shapes by name, so arm B swaps in texts that teach ITS surface (mirroring the production wording sentence-for-sentence outside the tool teaching). Production never sets it; unset ⇒ byte-identical prompts.

Fixture contract tests prove arm parity against a real daemon: descriptor lists, injected prompt text, real world-authorized delivery through the façade, and the post-flavored report-back append on a needsReply child.

Measured static cost (real descriptors + real injected prompts)

Component	sendMessage	post	ratio
descriptor (chars)	10,024 (~2,506 tok)	2,071 (~518 tok)	4.8×
standing guidance (chars)	2,718	2,394	1.1×
combined per session	12,742 (~3,186 tok)	4,465 (~1,116 tok)	2.9×

Behavioral matrix status

4 scenarios × 2 arms × 3 trials, local Claude Code over ACP, judged from the daemon's own records (executed product form + world effects, never model claims), counterbalanced arm order, per-component token accounting. Implemented in evals/test/tool-surface-ab-real.test.ts (env-gated, never in CI). The smoke run verified the harness to the provider boundary; the full matrix is currently blocked on re-authenticating the local Claude Code CLI (its OAuth refresh token expired 2026-08-09). The doc's §6 carries the exact one-command protocol and the table skeleton it fills.

Pre-registered: clear arm-B win on static cost (confirmed) and on invalid-call rate (partly by construction), little/no expected difference on success/efficiency at n=3.

Tests

• pnpm eval:collab:contracts — 115 passed
• pnpm eval:contracts — 46 passed
• new: 20 façade/classifier contracts + 3 fixture arm-parity contracts
• daemon suite: 2880 passed; the 2 failures (daemon-agent-mention-routing 5s-timeout, live acp-matrix) reproduce identically on clean main on this machine — pre-existing environmental, not from this branch
• typecheck / lint / format clean

🤖 Generated with Claude Code

[agentconnect-md-test]

I found three blocking evaluation-correctness issues on the exact trusted head:

1. The new deterministic façade/classifier/fixture contracts are not run by either CI contract command. The workflow invokes pnpm eval:contracts and pnpm eval:collab:contracts, but both scripts use explicit path lists that omit post-facade.test.ts, tool-surface-ab.test.ts, and tool-surface-ab-fixture.test.ts. The new production-side evaluation seams therefore have no effective CI regression gate.
2. The head-only tool-precedence bullet is present only in arm B at this base revision. Arm A's production guidance has no corresponding warning, so the current apparatus no longer has prompt parity. It also makes the checked-in static figures stale (arm-B standing guidance is now 2,770 chars, not 2,394; combined is 4,841, not 4,465), while the documented 24-run results were recorded before this head change. Either remove the B-only addition for this base, or bring in the matching A guidance and rerun/update the measurements.
3. The channel-bare judge accepts any delivered plaza post even when the peer was activated, although the scenario explicitly requires waking nobody. That is a success false positive and should be part of effectsOk.

Non-blocking warning: matchesToolPermissionFqns uses raw substring matching for registered FQNs, so a longer identity containing a registered name also receives the evaluation auto-allow. This is evaluation-only, but exact or delimiter-aware suffix matching would better match the stated fail-safe contract.

Verification: exact HEAD 7168426344289847a514696419fccadef5632aa0 was confirmed; git diff --check passed; a dependency-free compilePost smoke check passed; and a script inspection confirmed all three contract files are absent from the CI command lists. I could not execute Vitest because dependencies are not installed and downloading the pinned pnpm package failed with DNS resolution errors, including after the permitted network retry.

_{sent by review-bot (Codex · gpt-5.6-sol) · open in session}

[agentconnect-md-test]

These contracts are not currently CI-gated. .github/workflows/test.yaml runs pnpm eval:contracts and pnpm eval:collab:contracts, but the explicit file lists in both package scripts omit evals/test/post-facade.test.ts, evals/test/tool-surface-ab.test.ts, and evals/test/tool-surface-ab-fixture.test.ts. Please wire the new deterministic tests into a CI command; otherwise regressions in all three evaluation seams can merge without exercising these proofs.

[agentconnect-md-test]

At the trusted base/head, arm A's production guidance in session-manager.ts does not contain the corresponding tool-precedence warning; this text exists only in arm B. That makes the current prompts behaviorally asymmetric and changes arm-B guidance from the documented 2,394 chars to 2,770 (combined 4,841 rather than 4,465). The published 24-run results were also committed before this addition. Please remove this B-only warning for the current base, or bring in the A-side equivalent and rerun/update the static and behavioral measurements.

[agentconnect-md-test]

This can score success even when the peer was woken, although the channel-bare task explicitly requires that nobody be woken or notified. A model can make a bare product call whose message body still mentions the peer, yielding both deliveredInPlaza and peerActivated; this branch accepts it. Require deliveredInPlaza && !peerActivated (and record the activation note) so the daemon-effects judge enforces the scenario's negative condition.

[Poytr1]

Review findings addressed in 6c28d3a:

1. CI gate — post-facade.test.ts, tool-surface-ab.test.ts, and tool-surface-ab-fixture.test.ts are now in eval:collab:contracts (18 files / 138 tests, verified locally).
2. Prompt parity + stale figures — merged main, which now carries the Claude Code's built-in SendMessage tool collides with the AgentConnect messaging tool and silently swallows parent reports #800 tool-precedence bullet (fix(daemon): lead the collaboration guidance with a tool-precedence rule (#800) #801), so both arms include it again. §4's table now records both revisions explicitly: the 24-run revision (parity held — neither arm had the bullet) and the current head (parity held — both do: A 3,099 / B 2,771 chars, combined 13,123 vs 4,842 = 2.7×), plus §6 revision notes with the separately measured bullet effect (built-in SendMessage attempts 3/6 → 0/10, losses 0/10, success 10/10).
3. channel-bare judge — now also requires that nobody was woken (deliveredInPlaza && !peerActivated). The six recorded channel-bare trials were re-verified under the stricter rule against their events.jsonl: the peer ran zero turns in all six, so the recorded 6/6 stands.

On the non-blocking warning: the substring FQN matching in matchesToolPermissionFqns is deliberate consistency with the production isBuiltinSystemTool matcher, which uses the same includes rung for adapter id-suffixed identities (mcp__agentconnect__<name>-42); diverging the two matchers' semantics seemed worse than sharing the known trade-off. Happy to tighten both together in a follow-up if preferred.

🤖 Generated with Claude Code

[agentconnect-md-test]

The new revision addresses all three prior blockers and passes review.

• The façade, classifier, and real-daemon fixture contracts are now included in eval:collab:contracts, which the GitHub Actions workflow runs.
• Merging the current base brings the production tool-precedence warning into arm A, restoring prompt parity with arm B. The design now distinguishes the original 24-run revision from the current guidance lengths, and the checked static figures match the current constants.
• The channel-bare effects judge now requires !peerActivated and records an explanatory note when the negative condition fails.

I also re-inspected the complete current base-to-head diff; no new blocking correctness, lifecycle, or security issue emerged. The evaluation-only substring FQN behavior remains consistent with the existing production built-in matcher and is not a blocker.

Verification: the trusted checkout is the synthetic merge 206bbe015b9ff23344b39bb8ea12dc9e871fa11a whose parents are exactly base 29feb9b6e2021fbd88e4431ba46c764d0a81e5a6 and head 6c28d3a00f99e0b26c664317f088808a830b3c7c; git diff --check passed; dependency-free façade compilation/static-cost assertions passed; and static checks confirmed the CI paths and stricter judge. I could not execute Vitest locally because the pinned pnpm download still failed DNS resolution after the permitted network retry.

_{sent by review-bot (Codex · gpt-5.6-sol) · open in session}

[agentconnect-md-test]

Confirmed—these changes addressed the review findings, and I approved revision 6c28d3a0. Keeping the FQN matching consistent is reasonable; delimiter-aware tightening can be a separate follow-up.

_{sent by review-bot (Codex · gpt-5.6-sol) · open in session}

[Poytr1]

Pushed the #800/#861 follow-up onto this PR (it owns the A/B apparatus):

• Merged main (brings the revert(daemon): drop the #801 tool-precedence bullet after a live in-thread regression #861 revert; the standing guidance in the seam's default is byte-identical to production again) and dropped the arm-B parity bullet from POST_COLLAB_GUIDANCE to match — both arms are back at the 24-run revision's guidance (re-measured: 2,718 / 2,394 chars).
• Scenario 5 in-thread-count (36332e0): the in-thread turn-taking gate the matrix was missing — two subjects in one plaza thread, human kickoff mentioning both, count to 6 via ordinary replies; judgeThreadCount hard-fails on ANY participant messaging-tool call (the fix(daemon): lead the collaboration guidance with a tool-precedence rule (#800) #801 failure mode), on an incomplete count, or on a lost reply; duplicates/skips/meta-narration/turns/tokens are soft metrics. Contract tests are in eval:collab:contracts (judge traces + a scripted end-to-end fixture run); the real-run matrix gains the scenario at index 4 with the same counterbalancing, validity, and artifact rules.
• Baseline (db42907, doc §8.4): 2 arms × 3 trials on real local Claude Code (sonnet), post-revert prompt — 6/6 clean passes, all bare numbers 1–6 in order, zero messaging-tool calls. §8.5 + collaboration-arena-baseline.md §4.2 record the standing prompt-change gate: guidance changes must be validated against BOTH parent-session AND in-thread-count before landing.

[agentconnect-md-test]

The previously approved four-scenario A/B remains intact, but the new scenario-5 hard judge has two blocking false-pass paths:

1. It recognizes post and mcp__agentconnect__post, but not the daemon-supported dotted flattened identity mcp.agentconnect.post. A participant can therefore call the arm-B messaging surface under that valid adapter spelling and still receive a zero-tool-call PASS.
2. It treats every participant reply with no thread field as if it belonged to the kickoff thread. In the virtual world, thread: undefined is a channel-root post that opens a different conversation, so six root replies currently satisfy the “delivered ordinary thread replies” hard criterion.

Both were reproduced directly against judgeThreadCount: a clean 1–6 target-thread trace plus a dotted mcp.agentconnect.post event passes with no recorded messaging calls, and a clean 1–6 trace consisting entirely of threadless channel-root effects also passes. Please recognize both flattened FQN spellings and require the delivered count effects to match the exact kickoff thread, with regression tests for both cases.

Verification: the trusted checkout is synthetic merge 47fac0d18e6056b98276f907aab9dd56b24039b7, whose parents are exactly base c1a5253d6eba95b91b9a60d1153fca383d667014 and head db42907a5b72dde3502a6879da0d81e1fc018dec; git diff --check passed; the prior CI wiring and prompt-parity fixes remain present; and dependency-free TypeScript imports reproduced the two verdict errors above. The full collaboration suite could not start because downloading the pinned pnpm package failed DNS resolution in both sandboxed and permitted network attempts.

_{sent by review-bot (Codex · gpt-5.6-sol) · open in session}

[agentconnect-md-test]

This misses the dotted flattened MCP identity mcp.agentconnect.post, even though the daemon explicitly supports dotted FQNs alongside mcp__agentconnect__post (for example in BUILTIN_TOOL_FQNS and evaluation permission identities). Replaying a completed tool event with title mcp.agentconnect.post leaves messagingToolCalls empty and the judge passes. Please recognize the dotted suffix too and add it to the matcher contract.

[agentconnect-md-test]

thread === undefined is a channel-root post in ArenaWorld—it creates a new thread—rather than a reply in options.thread. Because this predicate accepts it, six participant root effects containing 1–6 currently produce a hard PASS despite no number landing in the kickoff conversation. Delivered count effects should require effect.thread === options.thread; please pin a wrong-root/wrong-thread trace as a failure (and keep rejected-reply accounting consistent with the stated “no participant reply was lost” rule).

[agentconnect-md-test]

Thanks—the scenario and baseline are useful. The latest review remains blocked on two db42907a judge false positives:

• mcp.agentconnect.post is not recognized as messaging.
• Threadless channel-root effects count as kickoff-thread replies.

Please fix and contract-test both paths.

_{sent by review-bot (Codex · gpt-5.6-sol) · open in session}

[agentconnect-md-test]

The two requested fixes are correct and their new contracts cover the exact reported cases. One blocking adapter spelling still escapes the revised hard-rule matcher, however.

The daemon explicitly supports ACP adapters that suffix an opaque invocation id to an underscore-flattened FQN (see containsBuiltinToolFqn and the existing mcp__agentconnect__sendMessage-42 permission test). With the new last-segment implementation, mcp__agentconnect__post-42 splits to a final segment of 42, so it is not recognized. Replaying a completed participant call under that identity alongside a clean target-thread count leaves messagingToolCalls empty and returns PASS. Please match a post name segment before an optional invocation suffix and add the suffixed underscore/dotted forms to the matcher and judge contracts.

Verification: exact HEAD cb89aaeeeaa2f1f91c555e8eb2b9f4a0d3fc4324 was confirmed; git diff --check passed; dependency-free judge checks confirmed that dotted mcp.agentconnect.post and off-thread replies now fail, while mcp__agentconnect__post-42 still passes incorrectly. The full Vitest suite remains unavailable in this workspace because dependencies are absent and the pinned pnpm package cannot be downloaded through the environment's DNS.

_{sent by review-bot (Codex · gpt-5.6-sol) · open in session}

[agentconnect-md-test]

This still misses the adapter-suffixed identity that the daemon explicitly supports: mcp__agentconnect__post-42 splits to ['mcp','agentconnect','post','42'], so lastSegment is 42. The existing daemon contract already pins the analogous mcp__agentconnect__sendMessage-42 form because some ACP adapters append opaque invocation ids. A completed post-42 event currently leaves messagingToolCalls empty and lets the hard judge pass. Please recognize post immediately before an optional suffix and contract-test both underscore and dotted suffixed spellings.

[agentconnect-md-test]

The adapter-suffixed FQN false-pass is fixed, and this revision passes review.

The bounded-segment matcher now recognizes the product and façade across bare, underscore-flattened, dotted, and opaque invocation-ID-suffixed identities, while retaining the intended negative cases such as compost and postpone. The judge contracts cover mcp__agentconnect__post-42 and mcp.agentconnect.post-42, and the earlier strict kickoff-thread check remains intact. I rechecked the complete current PR diff and found no remaining blocking correctness, lifecycle, or security issue.

Verification: exact HEAD 87dd69cd305902a6318eba804a21d672b6989256 was confirmed; git diff --check passed; dependency-free exact-head assertions confirmed all supported messaging identity forms fail the hard rule and threadless root replies cannot satisfy the count. I could not run Vitest in this workspace because dependencies are absent and the pinned pnpm download remains unavailable through the environment's DNS.

_{sent by review-bot (Codex · gpt-5.6-sol) · open in session}