feat: add WithBatchClaim option to eliminate high-concurrency Firestore contention - #16
Open
anish749 wants to merge 1 commit into
Open
feat: add WithBatchClaim option to eliminate high-concurrency Firestore contention#16anish749 wants to merge 1 commit into
anish749 wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When
WithConcurrency(N)is set to a large value (e.g. 100), all N workers race to claim the same task from Firestore. Each runs an independent transaction that queriesreadyTasks(...).Limit(1)— returning the same document for all workers. Only one wins; the rest get aborted withcross-transaction contention, setshouldWait = true, and sleep for up to a minute. This means:Solution
Adds a
WithBatchClaim(size int)HandlerOptionthat switches to a fetcher/worker topology forRegisterTaskHandler:size= channel capacity)Concurrency) — drain from the channel and execute handlers; never touch Firestore for claimingThis is purely opt-in. The default behaviour (
BatchClaimSize = 0) is unchanged — existing handlers keep the original per-worker claiming loop.RegisterResourceKeyHandleris unaffected; the option is ignored there since resource-key handlers already batch by group.Changes
oncetask/handler_config.go— addsBatchClaimSize intfield +WithBatchClaim(size int)optiononcetask/once_task_firestore.go:executeClaimedTaskshelper fromrunLoop(shared by both original and new paths)runFetcherLoopandrunWorkerLoopRegisterTaskHandlerbased onBatchClaimSizeUsage