Bump litellm pin to <=1.88.1 and sync cost map to v1.88.1#84
Merged
Conversation
Scoring asta-bench submissions needs the litellm 1.88.1 model cost map. Raise the upper bound of the litellm pin from 1.83.13 to 1.88.1 and point desired_model_costs_url at the model_prices_and_context_window_backup.json from the v1.88.1 release tag (SHA 9c117a5), per the "Bumping litellm" note in Development.md (keep pyproject pin and cost-map SHA in sync). Bump agent-eval to 0.1.51 so the scoring environment version is distinguishable in the submissions-review ledger. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
regan-huff
approved these changes
Jun 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
pyproject.tomlfrom1.83.13to1.88.1.desired_model_costs_urlinprep_litellm_cost_map()(src/agenteval/cli.py) atmodel_prices_and_context_window_backup.jsonfrom the litellm v1.88.1 release tag (SHA9c117a58be8fb8066e833aa92dd98fdf15771e50), replacing the previous319193…SHA.agent-evalversion0.1.50→0.1.51.Why
Scoring an asta-bench submission requires the litellm 1.88.1 model cost map, which is above the current
<=1.83.13ceiling. This follows the "Bumping litellm" guidance inDevelopment.md: when bumping the pin, update the cost-map SHA to the version at the upper bound (here, the v1.88.1 tag). The two are now kept explicitly in sync via a code comment.The cost-map backup JSON at the new SHA was confirmed reachable (HTTP 200).
Requested by @reganh for asta-bench submissions review (see allenai/nora wiki "Asta-Bench-Leaderboard: Submissions Review").
Follow-ups / notes for reviewer
Development.md, bumping the cost map may warrant rescoring results of interest with this version. That is a separate, manual step run against the HuggingFace submissions dataset and is not part of this PR.🤖 Generated with Claude Code