Bump agent-eval to 0.1.51 and litellm to 1.88.1 (sync cost map to v1.88.1)

[regan-huff]

What

• agent-eval[leaderboard]==0.1.50 → ==0.1.51
• litellm==1.82.3 → ==1.88.1
• astabench version 0.5.3 → 0.5.4, plus a CHANGELOG entry

Why both pins must move together

Cost calculation lives entirely in agent-eval (asta-bench has no cost code), so it's natural to ask why asta-bench's litellm pin needs to move at all. The reason: agent-eval's agenteval.log.compute_model_cost prices each model call via litellm.cost_per_token(model=..., usage_object=...), and that function resolves the model's provider from the installed litellm's built-in model registry — which is independent of the price map that agent-eval#84 registers via prep_litellm_cost_map().

So registering the v1.88.1 price map (what agent-eval 0.1.51 does) is necessary but not sufficient. A/B test with the same v1.88.1 map registered, varying only the installed litellm:

Model	in model_cost dict	litellm 1.82.3	litellm 1.88.1
gpt-5.5-2026-04-23	✅	✅ priced	✅ priced
gemini-3.5-flash	✅	❌ BadRequestError: LLM Provider NOT provided … model=gemini-3.5-flash	✅ priced

When cost_per_token raises, compute_model_cost sets total_cost = None and breaks, nulling the whole task's cost. A real submission that made gemini-3.5-flash calls therefore published to the leaderboard with a null DS_1000 cost under litellm 1.82.3, and a correct non-null cost under 1.88.1.

This matches prior art: #144 bumped agent-eval and litellm together "to support scoring for additional models", while agent-eval-only bumps that added no new models (#146, #147) left the litellm pin alone. agent-eval carries the cost-map SHA (in agenteval/cli.py:prep_litellm_cost_map); asta-bench carries no cost-map of its own, so this is purely a dependency-pin bump.

Validation

• uv pip compile pyproject.toml resolves cleanly with the new pins (agent-eval==0.1.51, litellm==1.88.1, inspect-ai==0.3.203; no lockfile to update).
• End-to-end: with these versions astabench score / lb publish / lb view all succeed, and a previously cost-null submission (used gemini-3.5-flash) now scores a non-null DS_1000 cost.

🤖 Generated with Claude Code