feat(throttler): revive redo-aware CPU signal, fall back to Threads_running#1061
Open
morgo wants to merge 1 commit into
Open
feat(throttler): revive redo-aware CPU signal, fall back to Threads_running#1061morgo wants to merge 1 commit into
morgo wants to merge 1 commit into
Conversation
…unning Prefer the perf_schema redo-aware thread count — which excludes threads parked on redo-log flush, so the copy can safely oversubscribe the log — and fall back to Threads_running from global_status when the extra grants are missing. This revives the signal PR block#971 replaced wholesale: the block#971 staging A/B had already confirmed the redo-aware exclusion works on Aurora (the wait/io/redo_log_flush event matches there) for ~18% more copy throughput, and left it "revivable behind the verified event name". See issue block#974. - One AuroraThreads throttler (renamed from ThreadsRunning) carries a threadsMode {query, headroom, label}: redoAwareMode (perf_schema join, excludes its own sampler via CONNECTION_ID(), headroom 1 for the sibling commit-latency poller) or globalStatusMode (Threads_running, headroom 2). AuroraSetup.Build picks the mode with a CanReadRedoAwareThreads privilege probe and logs the choice (suggesting the GRANT on a permissions failure). The threads throttler is still always built on Aurora — mode selection, never disabled. - Autoscaler growth gate revived, mode-scoped: ResolveMaxWriteThreads caps the write-thread pool at the start value in redo-aware mode when --max-commit-latency=0, since the redo-aware signal ignores redo-log waiters and only commit-latency backstops log oversubscription. The Threads_running fallback counts those waiters as load, so it self-limits and keeps growing to 2x start. - Renames aurora_threadsrunning.go -> aurora_threads.go; updates the throttler README and docs/migrate.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates Spirit’s Aurora CPU-throttling load signal to prefer a redo-aware performance_schema thread-count (excluding redo-log waiters) and to fall back to Threads_running when the necessary perf-schema grants are not available, while also reintroducing a mode-aware autoscaler growth cap when commit-latency backstop is disabled.
Changes:
- Add a new
AuroraThreadsthrottler that supports two modes (redo-aware preferred;Threads_runningfallback) and select the mode duringAuroraSetup.Buildvia a privilege probe. - Update migration autoscaler max-write cap logic to avoid scaling above the starting value when using redo-aware mode without commit-latency backstop.
- Refresh throttler/migration documentation and comments to reflect the new throttler name and behavior.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| pkg/throttler/throttler.go | Updates throttler interface docs to reflect renamed Aurora threads throttler. |
| pkg/throttler/stale.go | Updates stale-signal guard docs to reflect renamed Aurora threads throttler. |
| pkg/throttler/README.md | Updates throttler documentation for the new Aurora threads throttler and its two modes. |
| pkg/throttler/aurora.go | Adds mode selection + privilege-aware logging during Aurora throttler setup. |
| pkg/throttler/aurora_threadsrunning.go | Removes the old Threads_running-only Aurora threads throttler implementation. |
| pkg/throttler/aurora_threads.go | Introduces the new mode-capable Aurora threads throttler, plus redo-aware query + probe and updated autoscaler max cap helper. |
| pkg/throttler/aurora_threads_test.go | Updates and expands tests for the new Aurora threads throttler, redo-aware query/probe, and max-cap logic. |
| pkg/migration/runner.go | Integrates redo-aware detection into autoscaler cap selection when autoscaling is enabled. |
| docs/migrate.md | Documents the redo-aware vs fallback behavior and the autoscaler cap exception when commit-latency is disabled. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+335
to
+341
| // NewAuroraThreadsThrottler returns a Throttler that polls the given mode's | ||
| // signal and throttles when it exceeds the instance vCPU count (plus the mode's | ||
| // headroom). | ||
| func NewAuroraThreadsThrottler(db *sql.DB, mode threadsMode, logger *slog.Logger) (*AuroraThreads, error) { | ||
| if db == nil { | ||
| return nil, errors.New("AuroraThreads throttler requires a non-nil DB") | ||
| } |
Comment on lines
98
to
100
| ```go | ||
| throttler, err := throttler.NewThreadsRunningThrottler(db, logger) | ||
| throttler, err := throttler.NewAuroraThreadsThrottler(db, mode, logger) | ||
| ``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Revives the redo-aware
performance_schemaCPU-throttling signal as the preferred Aurora threads signal, falling back toThreads_runningfromglobal_statuswhen the account lacks the extra grants.Closes #974.
Why
The Aurora threads throttler currently counts all running threads via
Threads_running. Threads parked on the redo log are IO-bound (not on CPU) and Aurora group-commits them to storage, so counting them as load caps concurrency near vCPUs and leaves throughput on the table.The original signal (#831) subtracted redo-log waiters via a
performance_schema.threads⋈events_waits_currentjoin, but was replaced wholesale byThreads_runningin #971 on the belief the wait event never matched on Aurora. The #971 staging A/B later corrected that:wait/io/redo_log_flushdoes match on Aurora, and excluding those waiters oversubscribed the redo log for ~18% more copy throughput (~12 workers / ~25m vs ~8 workers / ~30.5m on r6g.4xlarge). #971 keptThreads_runningas a deliberate conservative launch and flagged the redo-aware variant as "revivable behind the verified event name." This is that revival.How
AuroraThreadsthrottler (renamed fromThreadsRunning) carries athreadsMode{query, headroom, label}:redoAwareMode(preferred) — the perf_schema join, subtractingwait/io/redo_log_flushwaiters and excluding its own sampling connection (CONNECTION_ID()). RequiresSELECTonperformance_schema.threads+events_waits_current. Hard-stop headroom 1 (covers the sibling commit-latency poller the query can't exclude).globalStatusMode(fallback) — today'sThreads_running; no grant beyond whatIsAuroraproves. Hard-stop headroom 2 (unchangedselfMonitoringHeadroom).AuroraSetup.Buildpicks the mode via aCanReadRedoAwareThreadsprivilege probe and logs the choice at Info (suggesting theGRANTon a permissions failure). The threads throttler is still always built on Aurora — this is mode selection, never a disable.ResolveMaxWriteThreadscaps the write-thread pool at the start value in redo-aware mode when--max-commit-latency=0, because the redo-aware signal deliberately ignores redo-log waiters and only commit-latency backstops log oversubscription. TheThreads_runningfallback counts those waiters as load, so it self-limits and keeps growing to 2× start.For an account without the perf-schema grants, behavior is identical to today (fallback mode, headroom 2, grows to 2× start under autoscaling).
Testing
go build ./...,go vet, and uncappedgolangci-lint— clean.go test -race ./pkg/throttler/ ./pkg/copier/— pass (incl. new redo-aware headroom, redo-aware query smoke, and privilege-probe tests).TestE2EAutoscalingEnabled+ a basic E2E alter — pass.🤖 Generated with Claude Code