ci(review): canary Claude reviews on claude-sonnet-5 (harper only)#1759
Open
heskew wants to merge 1 commit into
Open
ci(review): canary Claude reviews on claude-sonnet-5 (harper only)#1759heskew wants to merge 1 commit into
heskew wants to merge 1 commit into
Conversation
Override the reusable's model default (claude-sonnet-4-6) for this repo's claude-review caller. harper is the A/B canary: highest review traffic, and every ai-review-log entry records Model:, so calibration can compare sonnet-5 vs sonnet-4-6 verdict mix directly at the same prompt ref (9cf49d2). Intro pricing ($2/$10 through 2026-08-31) offsets the new tokenizer (~30% more tokens for equivalent text). Watch item: Sonnet 5 follows blocker-only severity instructions more literally (documented code-review-harness effect) — if the deflation rate rises in the next calibration cycle, add coverage-first reporting to the run-notes surface before fleet rollout; if clean, promote to the reusable default. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
Note Gemini is unable to generate a review for this pull request due to the file types involved not being currently supported. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
One-input canary: harper's Claude reviews run on claude-sonnet-5 while the fleet default stays claude-sonnet-4-6 (the reusable's default is untouched; this is a caller-side `model:` override).
Why harper, why now
Watch item (decides promote vs revert)
Sonnet 5 follows blocker-only severity instructions more literally — a documented code-review-harness effect where precision rises but measured recall can fall. Our dominant calibration failure is already severity deflation, so the metric to watch in the next calibration cycle is the deflation-partial rate for sonnet-5 entries vs 4.6. If it rises: add coverage-first reporting to the run-notes surface (findings with confidence/severity in the log, blocker-only on the PR) and re-measure. If clean or better: promote to the reusable default and drop this override.
Note: this PR edits the caller workflow, so its own Claude review check fails with the expected anti-tamper guard; clears on merge.
🤖 Generated with Claude Code