ci: opt into org leaderboard metrics collection#213
Conversation
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 30 minutes Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Codex Review
Critical Issues
None.
Security
.github/workflows/leaderboard-metrics.yml:10and.github/workflows/leaderboard-metrics.yml:14: this grantsid-token: writeto apull_request_targetworkflow while delegating execution topraetorian-inc/.github/...@main. Because@mainis mutable, future changes in that external reusable workflow immediately receive OIDC token minting ability in this repo’s privileged PR-target context. Pin the reusable workflow to an immutable commit SHA, ideally with the same version-comment pattern used elsewhere in.github/workflows.
Suggestions
No suggestions.
Reviewed by Codex (gpt-5.5)
There was a problem hiding this comment.
Gemini Review
Critical Issues
None.
Security
No security concerns flagged.
Suggestions
- Pin Reusable Workflow by SHA: Consider pinning the reusable workflow (
praetorian-inc/.github/.github/workflows/leaderboard-metrics.yml@main) to a specific commit SHA (along with a version comment). The rest of the repository adheres to this practice for workflows underpraetorian-inc/public-workflows(as enforced byverify-pins.yml) to ensure consistent supply-chain security and immutability.
Reviewed by Gemini (gemini-3.1-pro-preview)
There was a problem hiding this comment.
Claude Review
Critical issues
- The trigger is
pull_request_target: types: [closed], which fires for every closed PR — including PRs closed without merging. The description says "on every merged PR," so unless the reusable workflow (praetorian-inc/.github@main) internally gates ongithub.event.pull_request.merged == true, this collects/sends metrics for abandoned PRs too. Please confirm the reusable workflow filters on merged, or addif: github.event.pull_request.merged == trueat the job level here.
Security
- The reusable workflow is referenced by the mutable ref
@mainwhile this caller grantsid-token: write. If that ref is moved or the.githubrepo is compromised, an attacker inherits OIDC token minting againstLEADERBOARD_METRICS_ROLE_ARN. Prefer pinning to a commit SHA for anything holdingid-token: write. pull_request_targetruns in the trusted base-repo context with secret/OIDC access. Confirm the reusable workflow does NOT check out and execute untrusted PR head code (build scripts, install hooks, etc.) — line-stat collection should operate on metadata/diffs only. This is the primarypull_request_targetfoot-gun and cannot be verified from this repo.
Test coverage
N/A — CI workflow YAML only, no production *.go/*.ts/*.py changes.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9ac30b7c86
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| id-token: write | ||
|
|
||
| jobs: | ||
| metrics: |
There was a problem hiding this comment.
When a PR is closed without being merged, this pull_request_target.closed trigger still invokes the reusable metrics workflow because the caller job has no if: github.event.pull_request.merged == true guard. For abandoned PRs or externally submitted PRs that are simply closed, any ingestion done by the reusable workflow will count unmerged line churn and grant the OIDC-enabled job unnecessarily, which corrupts the leaderboard the commit says should be fed only by merged PRs.
Useful? React with 👍 / 👎.
2 participants
Opts this repo into the org-wide engineering leaderboard by calling the reusable workflow in
praetorian-inc/.github. On every merged PR, per-author line stats are collected and sent to the Guard leaderboard ingestion queue (feeds the Code and Assists columns).leaderboardGitHub environment has already been created on this repo (no protection rules).ENGINEER_EMAIL_MAP,LEADERBOARD_METRICS_ROLE_ARN,LEADERBOARD_METRICS_QUEUE_URL) are visible to all org repos.🤖 Generated with Claude Code