Skip to content

fix(orb): whitelist gate_verdict and gate_reasoncode_bucket on ingest - #9896

Merged
loopover-orb[bot] merged 1 commit into
JSONbored:mainfrom
RealDiligent:fix/critical-issue-orb-ingest-validate-enums-9642
Jul 29, 2026
Merged

fix(orb): whitelist gate_verdict and gate_reasoncode_bucket on ingest#9896
loopover-orb[bot] merged 1 commit into
JSONbored:mainfrom
RealDiligent:fix/critical-issue-orb-ingest-validate-enums-9642

Conversation

@RealDiligent

Copy link
Copy Markdown
Contributor

Summary

ingestOrbSignals (src/orb/ingest.ts) normalizes untrusted fleet telemetry, whitelisting outcome (VALID_OUTCOMES) and reversal_flag (VALID_REVERSALS) against closed vocabularies. The other two enum-valued columns were stored with only a length check:

typeof event.gate_verdict === "string" && event.gate_verdict.length <= MAX_VERDICT_CHARS ? event.gate_verdict : null
typeof event.gate_reasoncode_bucket === "string" && event.gate_reasoncode_bucket.length <= MAX_BUCKET_CHARS ? event.gate_reasoncode_bucket : null

Both are read downstream as closed enums by exact equality:

  • foldInstance (src/orb/analytics.ts) branches on verdict === "merge" / === "close". Any other value (e.g. the case variant "Merge") matches neither, so the row lands in holds and is published as a deferral in fleetAccuracy.coveragePct on /v1/public/stats — silently understating the coverage figure metrics: publish per-arm precision, coverage, and Wilson intervals — retire the bare accuracy scalar #8829 added specifically so it couldn't be gamed by raising the hold rate.
  • gate_reasoncode_bucket is compared against the literal "policy_action"; the writer's vocabulary (bucketReasonCode, src/selfhost/orb-collector.ts) is nine fixed literals, but the reader accepted anything.

The ingest endpoint is open by design, and storage is INSERT OR REPLACE keyed on (instance_id, repo_hash, pr_hash) — so a registered instance on an older/buggier build can poison its own published coverage, and the whitelist that would stop it already exists three lines above for the sibling column.

Fix

Added VALID_VERDICTS = new Set(["merge", "close", "hold"]) (the honest writer's GateAction vocabulary) and VALID_REASONCODE_BUCKETS (the nine bucketReasonCode literals) next to VALID_OUTCOMES/VALID_REVERSALS, and AND-ed the membership check onto the existing length check for each column (length check kept, not replaced). An off-vocabulary value is stored as null, which foldInstance already treats as a normal quality verdict — no downstream change.

Tests

Added to test/integration/orb-ingest.test.ts: a valid gate_verdict (close/hold) is stored while an off-vocabulary but length-legal value ("Merge", "banana") becomes null; likewise gate_reasoncode_bucket keeps policy_action/other and nulls made_up_bucket. Both fail against the current length-only check. Full suite passes (50/50).

Closes #9642

@RealDiligent
RealDiligent requested a review from JSONbored as a code owner July 29, 2026 19:32
@loopover-orb

loopover-orb Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Tip

✅ LoopOver review result - approve/merge recommended

Review updated: 2026-07-29 19:41:31 UTC

2 files · 1 AI reviewer · no blockers · readiness 98/100 · CI green · clean

✅ Suggested Action - Approve/Merge

  • safe to merge

Review summary
This PR closes issue #9642 by adding closed-vocabulary allowlists (VALID_VERDICTS, VALID_REASONCODE_BUCKETS) for gate_verdict and gate_reasoncode_bucket in src/orb/ingest.ts, mirroring the existing pattern already used for outcome and reversal_flag. The fix is well-targeted: it AND's the new Set.has() check onto the existing length check rather than replacing it, and off-vocabulary values fall through to null, which the PR correctly traces as already being treated as a normal quality verdict downstream in foldInstance. Tests cover both allowlisted and off-vocabulary (but length-valid) values landing correctly, giving real regression coverage for the exact gap described.

Nits — 3 non-blocking
  • src/orb/ingest.ts and test/integration/orb-ingest.test.ts use 'whitelist' terminology in comments/test descriptions — consider 'allowlist' for consistency with more neutral terminology.
  • The VALID_REASONCODE_BUCKETS set duplicates the nine literals from bucketReasonCode in src/selfhost/orb-collector.ts by hand; a shared constant or exported type would prevent the two vocabularies from drifting apart silently if a new bucket is ever added.
  • Consider exporting the bucketReasonCode vocabulary from orb-collector.ts (or a shared module) and importing it into ingest.ts instead of re-declaring the nine literals, so a future addition to one doesn't silently desync from the other.

Decision drivers

  • ✅ Code review — No blockers (1 reviewer)
  • ✅ Gate result — Passing (No configured blocker found.)
Context & advisory signals — never blocks the verdict
Signal Result Evidence
Linked issue ✅ Linked #9642
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (1 linked issue).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 251 registered-repo PR(s), 99 merged, 38 issue(s).
Contributor context ✅ Confirmed Gittensor contributor RealDiligent; Gittensor profile; 251 PR(s), 38 issue(s).
Improvement ✅ Minor risk: clean · value: minor · LLM: moderate
Linked issue satisfaction

Partially addressed
The core whitelist fix in src/orb/ingest.ts correctly adds VALID_VERDICTS and VALID_REASONCODE_BUCKETS and AND-s them onto the existing length checks, matching the required pattern, and the regression test for gate_verdict: 'Merge' storing null is present with a valid-path test covering multiple verdicts/buckets.

Review context
  • Author: RealDiligent
  • Role context: outside_contributor
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: not available
  • Official Gittensor activity: 251 PR(s), 38 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Keep the PR focused and include validation evidence before maintainer review.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

  • @loopover ask &lt;question&gt; answers contribution-quality Q&A with source citations and freshness.
  • @loopover chat &lt;question&gt; answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
  • A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

  • Re-run LoopOver review

@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

ingestOrbSignals whitelists outcome and reversal_flag against closed vocabularies but
stored gate_verdict and gate_reasoncode_bucket with only a length check, though both
are read downstream as closed enums by exact string equality. An off-vocabulary
gate_verdict (a case variant like "Merge", or any foreign string) matches neither
foldInstance branch, so a real decision is silently reclassified into `holds` and
understates the published fleetAccuracy.coveragePct — the exact metric JSONbored#8829 added so
coverage could not be gamed by raising the hold rate. The open ingest endpoint means a
registered instance on an older/buggier build can poison its own published coverage.

Add VALID_VERDICTS (merge|close|hold, the writer's GateAction vocabulary) and
VALID_REASONCODE_BUCKETS (bucketReasonCode's nine literals) next to VALID_OUTCOMES/
VALID_REVERSALS, and AND the membership check onto the existing length check for each
column, coercing an off-vocabulary value to null (which foldInstance already treats as
a normal quality verdict). No downstream change needed.

Closes JSONbored#9642
@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 29, 2026
@codecov

codecov Bot commented Jul 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.33%. Comparing base (af98111) to head (4b7bcb9).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9896      +/-   ##
==========================================
+ Coverage   79.29%   79.33%   +0.03%     
==========================================
  Files         281      282       +1     
  Lines       58566    58667     +101     
  Branches     6785     6822      +37     
==========================================
+ Hits        46440    46541     +101     
  Misses      11843    11843              
  Partials      283      283              
Flag Coverage Δ
backend 100.00% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/orb/ingest.ts 100.00% <100.00%> (ø)

@loopover-orb loopover-orb Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LoopOver approves — the gate is satisfied and CI is green.

@loopover-orb
loopover-orb Bot merged commit 3a10493 into JSONbored:main Jul 29, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

orb(ingest): gate_verdict and gate_reasoncode_bucket are stored unvalidated

1 participant