fix(federated): use analytics.ts's median for the peer benchmark, not nearest-rank - #9839
Conversation
… nearest-rank `refreshFederatedBenchmarkCache` computed the peer "median" with `percentile(sorted, 50)` — a nearest-rank estimator that, on an even-sized peer set, returns one middle element rather than the mean of the two. `computeFleetAnalytics` uses `median` (which averages the two middle values), so the peer benchmark and the fleet aggregation could disagree for the same data. Export `median` from analytics.ts and call it (it sorts internally, so the local `.sort()` is removed), making the two halves share one estimator by construction. Document the two contracts side by side: `median` sorts a copy and averages at even n; `percentile` requires a pre-sorted input and does not sort (its call sites, cycleP50Ms/cycleP95Ms, are named). Those percentiles are unchanged — they are percentiles, not medians. Closes JSONbored#9645
|
Warning ⏸️ LoopOver review result - manual review recommendedReview updated: 2026-07-29 13:19:01 UTC
Review summary Nits — 3 non-blocking
Decision drivers
Context & advisory signals — never blocks the verdict
Linked issue satisfactionAddressed Review context
Contributor next steps
Signal definitions
🧪 Chat with LoopOverAsk LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.
Full command reference: https://loopover.ai/docs/loopover-commands 🧪 Experimental — new and may change. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.
|
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #9839 +/- ##
==========================================
+ Coverage 79.04% 79.09% +0.04%
==========================================
Files 281 283 +2
Lines 58409 58579 +170
Branches 6698 6748 +50
==========================================
+ Hits 46171 46334 +163
- Misses 11955 11959 +4
- Partials 283 286 +3
Flags with carried forward coverage won't be shown. Click here to find out more.
|
What & why
refreshFederatedBenchmarkCache(src/orb/federated-benchmark.ts) computed the peer "median merge precision" withpercentile(sorted, 50)— a nearest-rank estimator that, on an even-sized peer set, returns a single middle element rather than the mean of the two middle values.computeFleetAnalyticscomputes the fleet median withmedian(which averages the two middle values), so the two halves of the same benchmark used different estimators and could disagree for identical data.The fix
medianfromsrc/orb/analytics.tsand call it inrefreshFederatedBenchmarkCache, deleting the now-redundant local.sort(...)(mediansorts a copy internally). The federated peer half and the fleet aggregation now share one estimator by construction.mediansorts internally and averages the two middle values at even n;percentilerequires an already-sorted input and does not sort (its two call sites,cycleP50Ms/cycleP95Ms, are named).percentile's p=50 behaviour, socycleP50Ms/cycleP95Msare unchanged — they are percentiles, not medians.Tests (
test/unit/federated-benchmark.test.ts)mergePrecision0.4and0.8→peerMedianMergePrecision0.6(the mean of the two middle values), where the old nearest-rank returned0.4(fails onmain).0.5and0.9(untrusted one rejected) →0.7, not the old nearest-rank0.5.cycleP50Ms/cycleP95Msfixtures (test/unit/orb-analytics.test.ts) still pin their exact values — thepercentilepath is unmoved.Validation
npm run typecheckgreen; the federated-benchmark + orb-analytics suites green.git diff --check <base> HEADclean; no schema/migration change.Closes #9645