Skip to content

feat: add WithBatchClaim option to eliminate high-concurrency Firestore contention - #16

Open
anish749 wants to merge 1 commit into
mainfrom
batch-claim-option
Open

feat: add WithBatchClaim option to eliminate high-concurrency Firestore contention#16
anish749 wants to merge 1 commit into
mainfrom
batch-claim-option

Conversation

@anish749

@anish749 anish749 commented May 5, 2026

Copy link
Copy Markdown
Owner

Problem

When WithConcurrency(N) is set to a large value (e.g. 100), all N workers race to claim the same task from Firestore. Each runs an independent transaction that queries readyTasks(...).Limit(1) — returning the same document for all workers. Only one wins; the rest get aborted with cross-transaction contention, set shouldWait = true, and sleep for up to a minute. This means:

  • N Firestore transactions to process 1 task
  • 99 workers idle for ~1 minute after each burst
  • Effective throughput collapses under high concurrency

Solution

Adds a WithBatchClaim(size int) HandlerOption that switches to a fetcher/worker topology for RegisterTaskHandler:

  • 1 fetcher goroutine — the sole goroutine that claims tasks from Firestore (no contention), pushing claimed tasks onto a local buffered channel (size = channel capacity)
  • N worker goroutines (Concurrency) — drain from the channel and execute handlers; never touch Firestore for claiming

This is purely opt-in. The default behaviour (BatchClaimSize = 0) is unchanged — existing handlers keep the original per-worker claiming loop.

RegisterResourceKeyHandler is unaffected; the option is ignored there since resource-key handlers already batch by group.

Changes

  • oncetask/handler_config.go — adds BatchClaimSize int field + WithBatchClaim(size int) option
  • oncetask/once_task_firestore.go:
    • Extracts executeClaimedTasks helper from runLoop (shared by both original and new paths)
    • Adds runFetcherLoop and runWorkerLoop
    • Branches in RegisterTaskHandler based on BatchClaimSize

Usage

manager.RegisterTaskHandler(
    TaskKindEmail,
    emailHandler,
    oncetask.WithConcurrency(50),
    oncetask.WithBatchClaim(50), // 1 fetcher + 50 workers, zero inter-worker contention
)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant