Add durable external operations for IOA-triggered integrations

[nikstern]

Problem

Temper describes integrations as outbox-pattern work that runs after a state
transition commits, but the generic runtime contract does not yet make external
operations durable across process failure.

The existing integration engine uses an in-memory channel and background tasks,
and its available dead-letter implementation is in-memory. Other runtime work is
adding durable internal effects, schedules, spawns, timeouts, and retries, but
there is no single operation record that durably connects a committed entity
transition to external egress and a governed settlement callback.

This leaves applications to implement their own intent journals, workers,
leases, retry classification, remote-operation polling, ambiguity handling,
recovery files, metering, and settlement logic.

Desired outcome

A state transition can authorize an external operation with a durable intent.
Temper records that intent before egress, executes it under bounded policy,
survives restart, records the strongest honest outcome available, and settles
the result through a governed entity action.

The operation must remain inspectable and auditable without moving application
business logic into the kernel.

Required semantics

• Atomically co-commit the external-operation intent with the source transition.
• Derive a stable operation ID from tenant, source entity, committed sequence,
  integration, and effect index.
• Record an attempt before each external create/send.
• Use lease generation or equivalent fencing so multiple workers cannot execute
  the same attempt concurrently.
• Recover pending and accepted work after process restart.
• Support immediate responses, accepted remote-operation IDs with polling,
  authenticated callbacks, cancellation, and bounded deadlines.
• Distinguish known Failed outcomes from Ambiguous outcomes where egress may
  have succeeded.
• Never automatically retry an ambiguous non-idempotent operation.
• Reuse a stable provider idempotency key when the remote endpoint supports it.
• Make settlement into entity state idempotent and independently retryable
  without repeating egress.
• Surface terminal failures, expired leases, ambiguous outcomes, and settlement
  failures to operators.
• Preserve originating principal, workflow trace, integration declaration,
  module digest, operation ID, and attempt identity through execution and
  settlement.

Exactly-once external effects are not generally possible without cooperation
from the remote system. The API should declare honest delivery modes such as:

• idempotent
• recoverable_remote_id
• non_idempotent
• callback
• read_only

Proposed lifecycle

Pending
  -> Claimed
  -> Dispatching
  -> Accepted
  -> Polling
  -> Succeeded
  -> Settled

Recovery and terminal paths should include:

Dispatching -> Failed
Dispatching -> Ambiguous
Accepted    -> Failed
Accepted    -> Cancelled
Succeeded   -> SettlementBlocked

The exact states are an ADR decision, but the distinction between operation
execution and settlement must be preserved.

Workstreams

• Write and approve an ADR before implementation.
• Define the IOA declaration and validation rules for external operations.
• Define the durable operation record and query projection.
• Add storage migrations, indexes, retention, and backend parity.
• Co-commit source transitions and operation intents.
• Implement bounded claim, lease, heartbeat, and generation fencing.
• Implement restart recovery and durable deadline scheduling.
• Implement retry and ambiguity classification.
• Implement polling, callback, and cancellation adapters.
• Implement governed, idempotent settlement callbacks.
• Propagate principal, delegation, and workflow trace context.
• Add optional hierarchical attempt and usage budgets.
• Add purpose-bound approval support for high-impact egress.
• Add payload classification, secret isolation, and telemetry redaction.
• Add Observe APIs, timelines, metrics, monitors, and reconciliation actions.
• Extend deterministic simulation with external-outcome nondeterminism.
• Add crash, duplicate-delivery, delayed-callback, and multi-worker tests.
• Build a durable webhook reference application.
• Build a mock long-running-operation reference adapter.
• Document rollout, compatibility, migration, and rollback.

Verification requirements

The verification model should explore:

• success;
• known failure;
• timeout before send;
• ambiguous send;
• accepted then pending;
• accepted then success or failure;
• duplicate or delayed callback;
• worker crash;
• cancellation race;
• settlement failure.

It must prove:

• attempt budgets cannot be exceeded;
• ambiguous non-idempotent operations are not automatically retried;
• terminal operations cannot return to dispatching;
• settlement is idempotent;
• duplicate delivery cannot duplicate entity state changes;
• unauthorized callbacks cannot settle an operation;
• every non-terminal operation state has a timeout or explicit indefinite-state
  justification.

Observability requirements

Expose bounded, redacted fields for:

• operation status and age;
• source entity/action/sequence;
• attempt count and lease generation;
• remote acceptance status;
• safe upstream correlation IDs;
• retry and failure classification;
• settlement status;
• cancellation status;
• workflow trace identity.

Provide metrics for queue depth, oldest pending age, claim latency, dispatch
latency, accepted-to-terminal latency, settlement latency, retries, ambiguous
creates, expired leases, cancellation failures, dead letters, and budget
exhaustion.

Non-goals

• Provider-specific business logic in Temper core.
• A general imperative workflow engine.
• Persisting secrets or unrestricted external response bodies.
• Claiming exactly-once behavior without remote idempotency support.
• Hiding application-domain workflow state inside an opaque worker.
• Replacing Cedar authorization or entity state machines.

Related work

• ARN-179: execute PostgreSQL actor effects durably #372 — durable PostgreSQL actor effects, schedules, tells, and spawns.
• fix(server): re-arm state timeouts after actor hydration #384 — durable state-timeout recovery.
• fix(runtime,verify): make simulation delivery single-owner (ARN-236) #387 — deterministic single-owner delayed-message delivery in simulation.
• fix(dispatch): run scheduled_actions + spawn_requests on the inline integration path (ARN-188) #349 — persistence and replay of deferred spawn/schedule effects.
• fix(spec): reject undeliverable IOA webhooks (ARN-227) #399 / fix(spec): reject undeliverable IOA webhooks #386 / fix(server): execute spec-declared webhook integrations (ARN-227) #403 — competing ARN-227 outbound-webhook approaches.
• ADR-0092 — identifies durable reaction outbox/replay as follow-up work.
• ADR-0067 — trajectory outbox pattern.

Definition of done

• A source transition and its external-operation intent cannot diverge.
• Restart loses neither pending nor accepted work.
• Two executor replicas cannot duplicate an operation attempt.
• Ambiguous sends remain visible and are never retried unsafely.
• Settlement can recover without repeating external egress.
• Budgets and approvals are enforced before egress.
• Sensitive payloads are absent from logs and proof artifacts.
• Simulation covers crash, delay, duplication, and ambiguity.
• A durable webhook and a long-running-operation reference flow pass live
  restart tests.
• Production evidence includes the pinned Temper commit, operation timeline,
  entity settlement transitions, and correlated telemetry.