Skip to content

fix: serialize Grok token refresh to stop concurrent invalid_grant races - #2

Merged
moveeeax merged 1 commit into
mainfrom
fix/serialize-token-refresh
Jul 26, 2026
Merged

fix: serialize Grok token refresh to stop concurrent invalid_grant races#2
moveeeax merged 1 commit into
mainfrom
fix/serialize-token-refresh

Conversation

@moveeeax

Copy link
Copy Markdown
Owner

What

auth.Manager.refresh() claimed (via a stale comment) to serialize refreshes, but only held the mutex long enough to copy a few fields, releasing it before the actual HTTP round trip to the token endpoint. Concurrent callers hitting GetAccessToken right at token expiry would each read the same refresh_token and POST it to the provider at the same time.

Why it matters

Most OAuth providers rotate (single-use) refresh tokens: only the first of two concurrent uses of the same refresh_token succeeds, the second comes back invalid_grant. The goroutine that lost that race returned an error to its caller (token expired and refresh failed) even though a sibling goroutine had, moments earlier, successfully refreshed the credential the Manager now holds. Under any real concurrent load this shows up as spurious 503s from the proxy right at the token-expiry boundary, even though a valid token exists.

ForceRefresh (used after an upstream 401) had the identical gap — a burst of 401s across concurrent requests could each trigger a forced refresh concurrently.

How

  • Added a dedicated refreshMu sync.Mutex held for the full duration of the refresh operation (network call + state update), not just the field copy.
  • After acquiring it, a non-forced refresh re-checks whether the entry is already fresh — if a sibling call refreshed while this one waited for the lock, it returns immediately and the caller re-reads the now-current entry, instead of hitting the provider again with a token that may already be stale.
  • ForceRefresh always attempts a refresh (correct — it's called specifically because the current token was rejected), but now goes through the same mutex so it can't stampede the token endpoint either.

Verification

gofmt -l .                                    → clean
go build ./...                                → ok
go vet ./...                                  → ok
go test ./... -race                           → ok (all packages)
go test ./internal/auth/... -run TestConcurrentGetAccessTokenDedupesRefresh -race -v -count=10
                                               → 10/10 PASS

New test TestConcurrentGetAccessTokenDedupesRefresh simulates a provider that rotates refresh tokens and fires 20 concurrent GetAccessToken calls right at expiry. Verified it fails reliably (5/5 runs) against the pre-fix code before restoring the fix.

Pre-existing, unrelated: go test ./... -count=3 panics in internal/config (pflag.CommandLine flag re-registration across repeated in-process runs) — reproduces identically on unmodified main, not touched by this change.

GetAccessToken had no real locking around the actual refresh HTTP call: the
mutex was only held long enough to copy fields, then released before the
network round trip. When multiple requests arrived right at token expiry,
each one read the same stale refresh_token and POSTed it to the provider
concurrently. Since OAuth providers commonly rotate (single-use) refresh
tokens, only one of those calls succeeds; the rest come back invalid_grant
and the losing goroutines returned an error to their callers even though a
sibling had just refreshed the credential successfully.

Add a dedicated refreshMu that is held for the whole refresh operation, and
after acquiring it re-check whether the entry is already fresh so a caller
that lost the race for the lock just observes the token a sibling refreshed
instead of hitting the provider again. ForceRefresh still always attempts a
refresh (used after an upstream 401), but is now serialized against the same
mutex so a burst of 401s does not also stampede the token endpoint.

Added TestConcurrentGetAccessTokenDedupesRefresh, which simulates refresh
token rotation and reliably failed against the old code (5/5 runs) before
the fix.
@moveeeax
moveeeax merged commit 7220ddf into main Jul 26, 2026
2 checks passed
@moveeeax
moveeeax deleted the fix/serialize-token-refresh branch July 26, 2026 11:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant