Observation
The agent and the verifier both ground through Perspicacite with a hardcoded kb_name=None, which resolves to open web search. On paper-reproduction tasks the agent can therefore retrieve the source paper and its reported values, even with the browser MCP disabled, since the grounding call itself reaches the web. A run may then score as a success when the agent retrieved rather than reproduced the result.
Why the two sides differ
The verifier may legitimately use ground truth (the paper or capsule) to grade. The agent may not, and should be restricted to a leak-free source or to none. A single shared setting cannot express this; the two sides need independent configuration.
Fair-conditions ablation this enables
- Baseline: agent grounds from nothing (removes the leak).
- Main arm: agent grounds from a leak-free "brief" KB (methodology only, answers stripped), to test whether safe grounding helps.
- Diagnostics only: paper KB (leak upper bound; scores should rise) and full capsule (development).
A leak-free brief requires a fresh index, as the capsule's index and embeddings already encode the answers.
Mechanism (opt-in, no default change)
Thread kb_name through the grounding query, its transports and the on-disk cache key (existing kb_name=None entries stay valid), and add three fields defaulting to current behaviour: an agent-side grounding switch (on), an agent KB (none) and a verifier KB (none). Orchestrator and planner honour the switch and pass the agent KB; the verifier passes its own.
Points for discussion (Thursday)
- Should evaluation default to agent-side grounding off, so the leak is closed unless explicitly enabled?
- Is the baseline / main-arm split above the right fair condition?
- Should the brief KB be built here or as a separate task?
Related: the verifier also grades paper-specific values a leak-free agent cannot know, which is a connected integrity question.
Observation
The agent and the verifier both ground through Perspicacite with a hardcoded
kb_name=None, which resolves to open web search. On paper-reproduction tasks the agent can therefore retrieve the source paper and its reported values, even with the browser MCP disabled, since the grounding call itself reaches the web. A run may then score as a success when the agent retrieved rather than reproduced the result.Why the two sides differ
The verifier may legitimately use ground truth (the paper or capsule) to grade. The agent may not, and should be restricted to a leak-free source or to none. A single shared setting cannot express this; the two sides need independent configuration.
Fair-conditions ablation this enables
A leak-free brief requires a fresh index, as the capsule's index and embeddings already encode the answers.
Mechanism (opt-in, no default change)
Thread
kb_namethrough the grounding query, its transports and the on-disk cache key (existingkb_name=Noneentries stay valid), and add three fields defaulting to current behaviour: an agent-side grounding switch (on), an agent KB (none) and a verifier KB (none). Orchestrator and planner honour the switch and pass the agent KB; the verifier passes its own.Points for discussion (Thursday)
Related: the verifier also grades paper-specific values a leak-free agent cannot know, which is a connected integrity question.