fix(estimator): stop infinite re-estimation loop + enrich context#6
Open
pablo-ibco wants to merge 2 commits into
Open
fix(estimator): stop infinite re-estimation loop + enrich context#6pablo-ibco wants to merge 2 commits into
pablo-ibco wants to merge 2 commits into
Conversation
…h context
Two issues with the AI Estimation Engine in production:
1. **Infinite re-estimation loop**: the polling query filters by
`estimate: null`, but the agent only posts a comment without setting the
numeric estimate. Every 2 minutes the same backlog re-appears and gets
re-scored. Burns ~$1/day on AWS doing nothing useful.
Fix: after posting the comment, attach an `ai-estimated` label to the
issue and add `labels.name: { neq: "ai-estimated" }` to the polling
filter so each issue is scored exactly once. Label auto-creates on
first use.
2. **Estimate quality was thin**: the prompt only had title + description
+ labels + a global historical average. Strengthened to include:
- Parent epic title + description (issue's business context)
- Initiative / project description (broader strategic context)
- Top-5 most-similar completed past issues by label overlap, including
their estimate vs actual cycle time (real grounding signal)
- Global calibration baseline (kept)
Cost impact: slightly more tokens per call (~+500 tokens), still
~$0.005 per estimate on Haiku 4.5. With the loop fix, total spend
on the estimator goes from ~$30/month to ~$2/month for our backlog.
Result: each new ticket is scored once with much richer context. Demo-ready.
Co-authored-by: Cursor <cursoragent@cursor.com>
…ching label) Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
The estimator was burning credit re-scoring the same 10 issues every 2 minutes forever, because the polling filter
estimate: nullnever excludes issues we've already commented on (the comment doesn't set the estimate).Cost impact: ~$36/month for a backlog that should cost ~$2/month.
The fix
After posting our estimation comment, attach an
ai-estimatedlabel to the issue. Polling filter now excludes that label, so each issue is processed exactly once.Bonus: better estimates
While in there, enriched the prompt context to include:
The bulk of the estimate now sees concrete past examples in the same area, not just a global average.
Test
fetch_unestimated_issuesfilter excludesai-estimatedlabelmark_issue_as_estimatedadds the label after each commentfetch_similar_completed_issuesranks by label overlapestimate_issuenow accepts and usessimilar_issuesin the promptDeploy
Will deploy directly to AWS after merge.
Made with Cursor