Skip to content

coordination.md

Latest commit

 

History

History
114 lines (85 loc) · 8.87 KB

coordination.md

File metadata and controls

114 lines (85 loc) · 8.87 KB
Raw

Coordination & Consistency

go-atlas ships four mechanisms for keeping a state change consistent with whatever else has to happen alongside it: another domain reacting, a non-transactional side effect, a message to another service, or a multi-step workflow. They are not interchangeable. Each sits at a different point on two axes:

  • Boundary. Does the work stay in-process (one goroutine, one transaction), or cross a process / service boundary (network, another database)?
  • Durability. Is the coordination ephemeral (lost on crash) or persisted (survives a crash, recovers / resumes)?
Mechanism Boundary Delivery Durable Atomicity scope Rollback model
domain/eventbus In-process Synchronous No The caller's single transaction Handler error → that transaction aborts
domain/eventbus/uow In-process Synchronous No One transaction + post-commit effects LIFO compensation of applied effects
data/outbox Cross-process Async, at-least-once Yes The producing transaction only No rollback — reliable forward delivery
transport/broker Cross-process Async pub/sub Yes (with outbox) None (it moves messages) None — consumers must be idempotent
data/saga Cross-service Step sequence Yes (persisted) Each step's own transaction Reverse-order compensation, crash recovery

The shared problem: the dual write

Almost every choice here comes down to one hazard. You change state in a database and you need a second thing to happen: notify another domain, put an object in a store, send a message to another service. If that second thing is not in the same database transaction, a crash or abort between the two leaves the system inconsistent: the row is written but the message never sent, or the message is sent but the row rolled back.

The four mechanisms answer that one hazard differently, and which you reach for depends on where the second thing lives.

The mechanisms

domain/eventbus — decouple domains in one process

A synchronous, in-process event bus. A handler runs in the publisher's goroutine, context, and transaction, so its writes join the same transaction and a handler error rolls the whole thing back (the bus's veto power). Use it to let one domain react to another's change without a direct code dependency, and to fail the originating operation if the reaction can't proceed. No durability, no network. See the Event Bus guide.

domain/eventbus/uow — non-transactional effects with compensation

The companion unit of work. When a handler needs a non-transactional side effect (an object-storage put, a cache write, an external call), it does not run it inline; it registers it with OnCommit, and uow applies it once after the commit, compensating already-applied effects in LIFO order if a later one fails. Closes the dual-write gap inside one process without an outbox. It is not crash-safe: a crash between the commit and the effect leaves inconsistency. That is the deliberate single-process trade-off.

data/outbox — reliable delivery out of a transaction

The transactional-outbox pattern. The event is persisted to a store in the same transaction as the state change, then dispatched later by background workers (dispatch, retry, cleanup, stuck-event recovery). This is the crash-safe answer to the dual write: because the event is written atomically with the state, it cannot be lost; because it is delivered out-of-band, the transaction never waits on the network. Delivery is at-least-once, transport-agnostic (the Handler decides how to deliver: broker, HTTP, gRPC). Lease and retention windows are evaluated against the database server clock (MongoDB $$NOW), not the worker's wall clock, so a clock-skewed worker can neither prematurely steal a locked event nor leak one; timestamps are stored as BSON Date (run MigrateTimestampsToDate once on a legacy collection).

transport/broker — asynchronous messaging between services

A message-broker abstraction (NATS JetStream provider) for async pub/sub across processes and services. It is the transport an outbox delivers to. transport/broker/outbox is the broker-specific adapter over data/outbox (msg.Message conversion + broker retry), implementing broker.Outboxer. Use the broker for fire-and-forget or event-driven integration where producer and consumer are decoupled in time and space. Consumers must be idempotent (at-least-once), so pair them with data/idempotency.

data/saga — orchestrated multi-step distributed transaction

A durable, persisted orchestration saga for a business transaction that spans several steps, each its own local transaction, possibly in different services. Each step has an optional compensating action; on failure the completed steps roll back in reverse order. A checkpoint is persisted after every stage, so an instance survives a crash and can be resumed or automatically rolled back. The pivot marks the point of no return: failures past it roll forward (retry) rather than compensate. See the Saga guide.

Choosing

Walk the questions in order; the first match wins.

  1. Does the reaction need to run inside the same transaction and be able to veto it? → eventbus (in-process, synchronous). If that reaction also has a non-transactional effect to undo on later failure, add uow.
  2. Do you need a second thing to reliably happen in another process/service after this transaction commits? → persist it with data/outbox and deliver via transport/broker. Never publish to the broker directly inside the transaction; that is the dual write.
  3. Is it a multi-step business transaction across services, where each step must be compensable and the whole thing must survive a crash? → data/saga.
  4. Is it just async pub/sub with no transactional coupling to a local write? → transport/broker alone.

How they compose

They are layers, not alternatives. A common reliable-event pipeline uses three of them together:

  1. Inside the database transaction, the domain writes its state and publishes a domain event on eventbus; in-process handlers react synchronously (and may veto).
  2. An eventbus adapter persists an outbox record in the same transaction (data/outbox). The cross-service hop is now crash-safe.
  3. After commit, the outbox worker dispatches the record through transport/broker to the message broker; the remote service consumes it idempotently.

uow slots in for the local non-transactional effects that should not become messages (an object-store write you want to compensate, not relay). data/saga sits above all of this when the operation is a multi-service workflow: the broker/outbox move its messages, while the saga owns the step sequence, compensation, and crash recovery.

Rules and anti-patterns

  • Never dual-write. Writing the database and then publishing to the broker as two separate steps loses messages on a crash in between. Persist via data/outbox in the transaction; deliver after commit.
  • eventbus is in-process only. Do not use it to reach another process or service — it has no network and no durability. Use the broker for that.
  • uow is not durability. Its compensation handles an effect that fails after a successful commit; it does not survive a crash between commit and effect. When you need crash-safety, route through the outbox instead.
  • At-least-once means idempotent consumers. Both data/outbox and transport/broker can deliver a message more than once. Consumers must dedupe — use data/idempotency.
  • Don't reach for a saga for a single local transaction. A saga is for multi-step, multi-transaction (often multi-service) workflows with compensation and recovery. One local transaction with eventbus / uow is simpler and atomic; a saga there is overkill.
  • Keep the boundary honest. In-process coordination (eventbus / uow) gives true atomicity within one transaction. Cross-process coordination (outbox / broker / saga) gives eventual consistency with compensation, never cross-database atomicity. Choose the mechanism that matches the boundary, not the one that's convenient.

See also