evals: add 3 Python API evals for cuopt-numerical-optimization-api-python skill#1418
Open
rgsl888prabhu wants to merge 5 commits into
Open
evals: add 3 Python API evals for cuopt-numerical-optimization-api-python skill#1418rgsl888prabhu wants to merge 5 commits into
rgsl888prabhu wants to merge 5 commits into
Conversation
…thon skill Adds numopt-py-eval-002, 003, 004 covering skill-specific Python API gotchas: - eval-002: status case sensitivity bug — 'OPTIMAL' vs 'Optimal' silent failure - eval-003: INTEGER vs CONTINUOUS for countable entities (nurses/workers) - eval-004: QP maximize workaround — negate coefficients and MINIMIZE Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
rgsl888prabhu
added
non-breaking
Collaborator
Author
|
/nvskills-ci |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (3)
✅ Files skipped from review due to trivial changes (1)
📝 WalkthroughWalkthroughThis PR updates evals.json (one prompt edit, three new evaluation cases), refreshes BENCHMARK.md and skill-card.md to reflect a 4-task run and updated results, and regenerates the sigstore signature bundle (skill.oms.sig). ChangesEvaluation test cases for cuOpt Python API
🎯 3 (Moderate) | ⏱️ ~20 minutes Suggested reviewers:
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@skills/cuopt-numerical-optimization-api-python/evals/evals.json`:
- Line 53: Update the "ground_truth" string to use a concave maximization
example with negative quadratic coefficients so it aligns with the NSD
requirement: replace "to maximize 0.04*x1*x1 + 0.02*x2*x2, minimize -0.04*x1*x1
- 0.02*x2*x2" with an equivalent that maximizes a concave quadratic (e.g., "to
maximize -0.04*x1*x1 - 0.02*x2*x2, minimize 0.04*x1*x1 + 0.02*x2*x2") and ensure
the surrounding explanation still references the Q matrix being NSD (concave)
and the negation producing a PSD Q for the solver; edit the ground_truth string
accordingly.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 9c7ac972-8967-428f-81cf-22c4a56dae03
📒 Files selected for processing (1)
skills/cuopt-numerical-optimization-api-python/evals/evals.json
Collaborator
Author
|
/nvskills-ci |
Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>
Collaborator
Author
|
/ok to test f87c77a |
tmckayus
approved these changes
Jun 10, 2026
- eval-004: use concave quadratic (-0.04*x1² - 0.02*x2²) as the maximize example — maximizing a convex quadratic is unbounded; clarify NSD requirement must hold for a finite maximum (CodeRabbit) - fix end-of-file newline (pre-commit) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Collaborator
Author
|
/nvskills-ci |
Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>
Collaborator
Author
|
/ok to test e02d54f |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds 3 new evals to
skills/cuopt-numerical-optimization-api-python/evals/evals.json(was 1, now 4), all grounded in skill-specific Python API content from SKILL.md.