djhtmx 2.0: async views, synchronous dispatch on a bounded pool

[edelvalle]

Summary

Makes djhtmx's HTTP / SSE / WebSocket entry points async views, but runs each dispatch as a single synchronous job on a bounded worker pool. This bounds the Postgres connection count by DJHTMX_SYNC_WORKERS regardless of request or SSE-stream concurrency, while letting components define async def event handlers. Bumps to 2.0.0.

Why

An earlier iteration of this branch made the dispatch pipeline fully async (ORM on the event loop via Django async ORM + request.auser()). A load test against a real app disproved the connection model: because django.db.connections is per-async-task, every in-flight request and every idle SSE stream pinned its own Postgres connection, so connections scaled ~1:1 with concurrency and exhausted max_connections. The fix is to keep all ORM off the loop entirely — run the whole dispatch on the pool, where connections are thread-affine and reused.

What changed

• Async views, sync dispatch. endpoint, sse_endpoint, and the WS Consumer are async shells that submit the whole dispatch to the sync-work pool via submit_sync_work. The SSE stream stays on the loop (its redis.asyncio pub/sub) and submits each drain to the pool. CommandProcessor.process is a synchronous generator.
• Async handlers still supported. _invoke_handler runs plain/sync handlers directly on the pool thread and wraps async def handlers (and async generators) via async_to_sync on that same thread — so even ORM awaited inside an async handler shares the one pooled connection. Sync/async handlers interoperate freely across an event cascade.
• The user is resolved on the pool, not via request.auser() on the loop — this is what stops idle SSE streams from each holding a connection.
• DJHTMX_SSE_RENDER_WORKERS → DJHTMX_SYNC_WORKERS (old name kept as a deprecated alias). One pool now bounds all ORM-touching work — it's the single Django DB connection budget.
• ATOMIC_REQUESTS is applied per sync handler. Django refuses ATOMIC_REQUESTS on async views, so the views opt out for every DB and each sync handler is instead wrapped in transaction.atomic per atomic database.
• Model fields keep their pre-2.0 loading (non-breaking): the field validator resolves a bare pk → instance (sync ORM, on the pool thread), and ModelConfig(lazy=True) defers the query to first access via _LazyModelProxy. Resolution running on the pool (not the loop) is what makes it connection-safe.
• Added aemit_sse_event for publishing SSE events from async def handlers without blocking the loop.

Connection model

Peak Postgres connections ≈ DJHTMX_SYNC_WORKERS per process, flat under concurrency; an idle SSE stream holds zero connections (it borrows one only while a drain renders). Size DJHTMX_SYNC_WORKERS × processes under Postgres max_connections (and any pgbouncer pool).

Breaking changes / migration

• Major version bump (2.0.0). Run the endpoints under ASGI (granian/uvicorn) — they're async views; ensure AuthenticationMiddleware is present (the views resolve the user).
• ATOMIC_REQUESTS is now per-handler, not per-request — audit anything relying on whole-request atomicity across a cascade.
• DJHTMX_SSE_RENDER_WORKERS still works but is deprecated in favor of DJHTMX_SYNC_WORKERS.
• Not breaking: Model-field loading is unchanged from 1.x (validator resolves pk; lazy supported).

Testing

• 150 tests pass; ruff + type-check clean.
• New coverage: sync-on-pool dispatch, sync↔async handler interop, per-handler ATOMIC_REQUESTS, Model resolution + lazy proxy (deferred-until-access query counts), and loadtest_async.py asserting no deadlock and pool threads bounded by DJHTMX_SYNC_WORKERS under many concurrent SSE drains.

Reviewing

Best read commit-by-commit: feat(async)! (the pipeline), then tests, then docs (docs/async-architecture.md is the canonical explanation of the connection model).

🤖 Generated with Claude Code