ci: opt into org leaderboard metrics collection#304
Conversation
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 29 minutes Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Codex Review
Critical Issues
None.
Security
.github/workflows/leaderboard-metrics.yml:10and.github/workflows/leaderboard-metrics.yml:14: the newpull_request_targetworkflow grantsid-token: writewhile delegating to a reusable workflow pinned to mutable@main. Pin the reusable workflow to a reviewed commit SHA or immutable release tag before allowing OIDC token minting from this PR-triggered path.
Suggestions
No suggestions.
Reviewed by Codex (gpt-5.5)
There was a problem hiding this comment.
Claude Review
Critical issues
on.pull_request_target: [closed]fires on every closed PR, not just merged ones. Confirm the reusable workflow (praetorian-inc/.github) gates ongithub.event.pull_request.merged == true; otherwise metrics get collected for abandoned/closed-unmerged PRs too. This can't be verified from this repo's diff — it's a hidden dependency on the callee's logic.
Security
- The reusable workflow is pinned to a mutable ref (
@main) while this job grantsid-token: write(AWS OIDC). Ifpraetorian-inc/.githubis compromised ormainchanges, this repo automatically runs the new workflow with OIDC credentials on every closed PR. Prefer pinning to a commit SHA (@<sha>) for supply-chain safety. pull_request_targetruns in the trusted base-repo context; here it correctly does not check out PR head code, so untrusted code isn't executed directly. Just confirm the callee never checks outgithub.event.pull_request.head.*on this trigger.
Test coverage
Not applicable — CI configuration only, no application code changed.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0996704f7a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| id-token: write | ||
|
|
||
| jobs: | ||
| metrics: |
There was a problem hiding this comment.
pull_request_target with types: [closed] fires both for merged PRs and for PRs that are closed without merging, but this job calls the metrics collector unconditionally. When an abandoned/rejected PR is closed, the same capability map is sent to the leaderboard workflow even though none of the changes landed, which can skew the collected per-author line stats unless every callee duplicates the filter; add a job-level if: github.event.pull_request.merged == true before invoking the reusable workflow.
Useful? React with 👍 / 👎.
|
|
||
| jobs: | ||
| metrics: | ||
| uses: praetorian-inc/.github/.github/workflows/leaderboard-metrics.yml@main |
There was a problem hiding this comment.
This caller grants id-token: write to a reusable workflow loaded from the mutable main branch. Any later change to praetorian-inc/.github's main will execute in this repo on pull_request_target without a reviewed change here and can mint the OIDC token used for leaderboard ingestion, so the workflow's behavior and cloud access are not reproducible; pin the reusable workflow to a reviewed commit SHA and adjust the OIDC trust if necessary.
Useful? React with 👍 / 👎.
2 participants
Opts this repo into the org-wide engineering leaderboard by calling the reusable workflow in
praetorian-inc/.github. On every merged PR, per-author line stats are collected and sent to the Guard leaderboard ingestion queue (feeds the Code and Assists columns).leaderboardGitHub environment has already been created on this repo (no protection rules).ENGINEER_EMAIL_MAP,LEADERBOARD_METRICS_ROLE_ARN,LEADERBOARD_METRICS_QUEUE_URL) are visible to all org repos.🤖 Generated with Claude Code