behavior.md

Behavior Guide

Selection

Among resources that are not disabled, not cooling down, and not at max_in_flight, the pool picks one according to its strategy:

• "round_robin" (default) -- fewest in-flight usages first, then oldest last_acquired_at. Best-effort fairness across resources; the pool cannot predict how long a usage will hold a slot, so this only balances by acquisition time.
• "primary_backup" -- return the first eligible resource in the original list/dict order. Later resources are only used when earlier ones are cooling down, disabled, or at capacity. Resource ordering is load-bearing under this strategy.

Selection and usage registration are atomic under one lock acquisition.

Example: primary_backup for a paid-tier primary with a free-tier fallback

Send every request to the paid key first; only spill over to the free key when the primary is rate-limited (cooling down) or the paid quota is exhausted (disabled). Resource ordering in the list is the priority ranking -- the pool will never reach for free-fallback while paid-primary is still healthy and below capacity.

pool = Pool(
    resources=[
        Resource(resource_id="paid-primary", value="sk-paid-..."),
        Resource(resource_id="free-fallback", value="sk-free-..."),
    ],
    strategy="primary_backup",
)

Example: primary_backup with a capacity cap to fan out under bursts

Combine max_in_flight on the primary with primary_backup to get "use the primary up to N concurrent calls, then overflow to the next tier." Useful when the primary is fastest/cheapest but has a hard concurrency limit you do not want to breach.

pool = Pool(
    resources=[
        Resource(resource_id="region-us",   value=us_client,   max_in_flight=8),
        Resource(resource_id="region-eu",   value=eu_client,   max_in_flight=8),
        Resource(resource_id="region-asia", value=asia_client),
    ],
    strategy="primary_backup",
)
# 1st-8th concurrent calls -> region-us
# 9th-16th             -> region-eu (us is at capacity)
# 17th+                -> region-asia (eu is at capacity)

Cooldown Escalation

Each consecutive CooldownResource from the same resource escalates the cooldown:

Consecutive count	Cooldown
1st	30s
2nd	120s
3rd	300s
4th+	600s

You can override per event: CooldownResource(cooldown_seconds=5), for example from a Retry-After header. The counter resets on the next success.

Custom tables are supported per pool:

pool = Pool(
    resources=[...],
    cooldown_table=(10.0, 30.0, 60.0, 120.0),
)

In-Flight Cancellation

When a resource receives a CooldownResource or DisableResource signal, the framework cancels younger in-flight usages on the same resource. Older usages are left alone -- they may still succeed. This maximizes throughput while avoiding doomed requests.

Cancellation is best-effort: it works when the operation returns a coroutine (the framework wraps it in an asyncio.Task) or an asyncio.Future (cancelled directly). For plain awaitables with no .cancel() handle, cancellation silently no-ops for that usage and it runs to natural completion. Within a coroutine, the underlying I/O is only truly aborted if the operation uses cancellation-aware async libraries such as httpx.AsyncClient or aiohttp.

Retry

pool.run() drives the retry loop. @pool.use() is a thin decorator shim over it. Attempts are capped at min(max_attempts, len(resources)) -- more retries than resources is pointless.

The pause between attempts is jittered: retry_delay * uniform(0.5, 1.5), mean retry_delay. Without jitter, concurrent calls that hit the same cooldown would all wake at the same instant and stampede the next eligible resource.

By default, run() fails fast: when no resource is eligible at the start of an attempt, it raises PoolExhausted immediately -- even if a deadline would outlive the cooldowns. Pass wait_for_cooldown=True to instead sleep until the earliest cooldown_until among cooling resources and select again. Only a cooldown gives a known wake-up time, so this never waits on resources that are disabled or at max_in_flight -- if nothing is cooling, PoolExhausted raises as usual.

With a deadline, the wait only happens when the earliest cooldown ends before it; otherwise PoolExhausted raises immediately rather than sleeping out a wait that provably cannot help.

The wake-up is jittered too: each waiter sleeps an extra retry_delay * uniform(0, 1) past the expiry, capped by deadline, so concurrent waiters do not all fire at the recovered resource in the same instant. As with the retry pause, retry_delay=0 disables the jitter.

Waiters also react to admin calls: pool.add() wakes them so they can acquire newly added capacity immediately, pool.enable() wakes them so they can acquire the now-eligible resource immediately, and pool.disable() wakes them so they can re-evaluate and fail fast instead of sleeping out a cooldown that no longer matters.

Admin Control

pool.add(resource_id, value, max_in_flight=None), pool.enable(resource_id), and pool.disable(resource_id) give operators write access to resource lifecycle state -- the counterpart to snapshot():

• add() adds new capacity at runtime. You pass only resource_id, value, and optional max_in_flight; the pool constructs a fresh healthy Resource with no cooldown history. Duplicate resource_ids raise ValueError. Added resources append to pool order, so under primary_backup they are the lowest-priority fallback until earlier resources become unavailable.
• disable() removes a resource from selection until enable() is called. Unlike an operation raising DisableResource, in-flight usages are not cancelled -- admin disable is policy, not failure evidence, so running work, which may already have upstream side effects, finishes naturally.
• enable() returns a resource to selection: it clears both the disabled state and any active cooldown, and resets consecutive_cooldown to 0. Enable means "the operator says this resource is usable now", for example a rotated key, so if the operator is wrong, escalation restarts from the first cooldown_table slot rather than resuming where it left off.

All three are async because they take the pool lock. enable() / disable() are idempotent, raise KeyError for an unknown resource_id, and wake any run(wait_for_cooldown=True) sleepers so they re-evaluate immediately. add() also wakes those sleepers because a new healthy resource may satisfy them immediately.

Cancellation Discrimination

The framework distinguishes external cancellation, such as client disconnect or shutdown, from internal cancellation, such as resource failure, by checking usage.status. The cooldown/disable handler sets the status to "cancelled" under the pool lock before invoking .cancel() on the handle, so observing that status when CancelledError arrives means "we cancelled ourselves" -- except for the one-tick edge case described in the cancellation gotcha. Works on any Python 3.10+.