Keep Tier-0 health checks bounded on large databases#593
Conversation
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_40349fb7-e329-4a19-bdd0-213033ee6965) |
|
@codex review |
|
@coderabbitai review |
|
You need to increase your spend limit or enable usage-based billing to run background agents. Go to Cursor |
✅ Action performedReview finished.
|
📝 WalkthroughWalkthroughThe health check now enforces a monotonic execution deadline, records slow-check state, and backs off for uninterruptible launchd processes. The Tier-0 watchdog persists alert state, suppresses repeated failure alerts, resets on recovery, and receives new launchd configuration and test coverage. ChangesHealth-check execution and recovery
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant run_health_check
participant SQLite
participant StateFile
participant tier0-watchdog.sh
participant AlertChannels
run_health_check->>SQLite: run deadline-aware health query
SQLite-->>run_health_check: result or interruption
run_health_check->>StateFile: persist count and slow-check status
tier0-watchdog.sh->>StateFile: read failure and alert state
StateFile-->>tier0-watchdog.sh: failure key and last alert epoch
tier0-watchdog.sh->>AlertChannels: send alert when cooldown permits
tier0-watchdog.sh->>StateFile: record alert or reset recovery state
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 729b51f482
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_0ce95ccb-8803-48fe-b16a-1c44854ded05) |
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
|
@codex review |
|
@coderabbitai review |
✅ Action performedReview finished.
|
|
Codex Review: Didn't find any major issues. Another round soon, please! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_332be0f9-5f46-42c6-a688-b14dd7b5e27d) |
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
|
@coderabbitai review |
|
@codex review |
✅ Action performedReview finished.
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 126893ab54
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| except HealthCheckDeadlineExceeded as exc: | ||
| return finish_slow("missing_embeddings", str(exc)) |
There was a problem hiding this comment.
When the deadline fires here after an earlier healable issue was already collected (for example hotlane_dead or hotlane_backlog_disabled from the process snapshot), this returns before _apply_heals runs, so --heal neither increments heal_failures nor kickstarts the affected daemon even with heal_min_consecutive_failures=1. On a slow DB tick this can repeat every run and suppress the existing self-heal path for the real service outage; run the heal bookkeeping for already-collected labels before taking the slow exit or persist enough state for the next tick.
Useful? React with 👍 / 👎.
Independent standing-reviewer verdict (s:399) — SHIP ✅Reviewed from an isolated worktree at head
|
1 participant
Summary
EXCEPTquery that stays bounded on the 16 GB production databaseslow_checkstate before returningU/Dstate, back off instead of repeatedly kickstarting, and escalate through the existing circuit breakerIncident validation
All production-DB diagnostics were read-only. No VACUUM, schema change, deletion, or repair command was run.
--json, no heal, isolated state file): 0.612s internal / 1.10s wallSCAN cVerification
3601 passed, 60 skipped, 5 xfailed(final branch tree,ulimit -n 4096)3511 passed, 9 skipped, 61 deselected, 1 xfailed; MCP registration, isolated eval/hook routing, Bun regression, and shell FTS regression all passed160 passed10 passed; the unloaded-label drill also passed 20/20 isolated repetitions after one transient full-suite scheduling failureruff check .andruff format --check .sh -n scripts/tier0-watchdog.sh scripts/launchd/install.shplutil -lint scripts/launchd/com.brainlayer.tier0-watchdog.plistgit diff --checkThe local CodeRabbit CLI produced only heartbeats and was stopped after roughly three minutes. PR-level CodeRabbit, Codex, Cursor, and Bugbot reviews are requested below.
Post-merge deploy checklist
main, reinstall the health-check and Tier-0 watchdog LaunchAgents from merged sourcesgui/501watchdog and verifylaunchctl printbrainlayer-tier0-rearm-2026-07-13Note
Medium Risk
Changes scheduled health-check, self-heal, and Tier-0 alerting on production launchd paths; mis-tuned deadlines or alert gating could miss real outages or delay repeat notifications.
Overview
Health checks now cap at 45 seconds end-to-end: DB counts use an indexed
EXCEPTmissing-vector query (replacing a full-chunk scan), SQLite progress handlers and a threaded source scan can abort under budget, and timeouts persistslow_checkstate (stage + duration) instead of leaving the tick hanging.Self-heal skips
launchctl kickstartwhen the target process is in uninterruptible U/D state, recordingheal_backofffor the existing circuit breaker.ps/subprocess failures returnNoneso doctor and health-check no longer treat a failed snapshot ashotlane_dead.Tier-0 watchdog treats
slow_check: trueas unhealthy, lowers stale threshold to 900s, and adds 1800s repeat-alert suppression per failure class while kickstart recovery still runs; plist env documents alert state path and timings.Reviewed by Cursor Bugbot for commit 126893a. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Bound Tier-0 health checks to a 45-second deadline on large databases
run_health_checkin health_check.py; stages that exceed the budget write aslow_checkstate and return early instead of blocking indefinitely.count_missing_embeddings,_enrichment_backlog) via a progress handler that raisesHealthCheckDeadlineExceededwhen the budget is exhausted._source_recentfilesystem scan in a daemon thread bounded by the remaining deadline, raisingHealthCheckDeadlineExceededif it does not finish in time._apply_heals: if a launchd process is in stateU/D, aheal_backoffaction is recorded instead of kickstarting.slow_check:trueas unhealthy and suppresses repeat alerts for the same failure for 1,800 seconds via persisted alert state, while continuing recovery attempts. Stale threshold is reduced from 1,200 s to 900 s.slow_checkrather than succeeding or hanging.Macroscope summarized 126893a.
Summary by CodeRabbit
Improvements
Documentation