Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,11 @@

### Added

- **Safe persistence failure boundaries (PR #225).** Persistence stores expose fixed, cause-free failure text; raw paths, SQL, and payloads flow only to `PersistenceFailureDiagnosticObserver`; worker observers receive safe failures; existing exception and store ABI is preserved by the binary fixture. Epic 1.2 Safe Error Boundaries is complete.

- **Safe provider and built-in workflow-step failure boundaries (PRs #222, #223).** Provider HTTP rejections and built-in HTTP, shell, MCP, Codex, and Hermes workflow failures expose fixed cause-free public exceptions with typed failure codes. Original failure detail is retained only by an explicitly configured, fail-open diagnostic observer; public workflow events omit URLs, commands, raw tool names, and failure reasons. Existing public exception constructor descriptors remain compatible with 0.5.0 clients.

- **Safe tool-failure boundaries (PR #219).** Established the first slice of Epic 1.2: raw exception details no longer cross built-in model-visible tool boundaries, while original causes remain available to an explicitly configured fail-open `ToolFailureDiagnosticObserver`. Added diagnostic-only `ToolFailureCode` classifications (`tool.input.invalid`, `tool.execution.failed`, `tool.execution.retry_exhausted`) with fixed model-visible defaults; caller-visible failure mapping remains pending. `ToolResult` retains exactly its four 0.5.0 variants without deprecations, preserving exhaustive-`when` source compatibility, and adds `safeInvalidInput(...)`/`safePermanentFailure(...)` factories for validated or fixed text. `ModelVisibleToolMessage` is a regular class with a private constructor and validated `@JvmStatic trusted(...)` factory (non-blank, ≤512 chars, code-point-aware rejection of control, separator, and FORMAT characters), with no generated `copy` or destructuring bypass. `ToolInvalidInputException(String)` remains public with diagnostic-only text plus `withSafeModelMessage(...)`. The engine and standalone adapter never derive model-visible text from `Throwable.message`, classify diagnostics from their own control flow, emit fixed retry-exhaustion text, preserve cancellation, and treat diagnostic-sink failures as fail-open. The standalone builder still freezes the observer at `build()`. Custom tools written against the short-lived round-1 `SafeInvalidInput`/`SafePermanentFailure` variants must migrate to the safe factories or stable plain constructors. Workflow-step, persistence, MCP, shell, structured-output, and caller-visible tool boundaries remain later slices.
- **Safe tool-failure boundaries (PR #219).** Established the first slice of Epic 1.2: raw exception details no longer cross built-in model-visible tool boundaries, while original causes remain available to an explicitly configured fail-open `ToolFailureDiagnosticObserver`. Added diagnostic-only `ToolFailureCode` classifications (`tool.input.invalid`, `tool.execution.failed`, `tool.execution.retry_exhausted`) with fixed model-visible defaults; caller-visible failure mapping remains pending. `ToolResult` retains exactly its four 0.5.0 variants without deprecations, preserving exhaustive-`when` source compatibility, and adds `safeInvalidInput(...)`/`safePermanentFailure(...)` factories for validated or fixed text. `ModelVisibleToolMessage` is a regular class with a private constructor and validated `@JvmStatic trusted(...)` factory (non-blank, ≤512 chars, code-point-aware rejection of control, separator, and FORMAT characters), with no generated `copy` or destructuring bypass. `ToolInvalidInputException(String)` remains public with diagnostic-only text plus `withSafeModelMessage(...)`. The engine and standalone adapter never derive model-visible text from `Throwable.message`, classify diagnostics from their own control flow, emit fixed retry-exhaustion text, preserve cancellation, and treat diagnostic-sink failures as fail-open. The standalone builder still freezes the observer at `build()`. Custom tools written against the short-lived round-1 `SafeInvalidInput`/`SafePermanentFailure` variants must migrate to the safe factories or stable plain constructors. Caller-visible tool failure mapping and approval boundaries remain later slices.

- **Durable file and JDBC step-attempt stores (PR #218).** Added independent `FileStepAttemptRecordStore` and `JdbcStepAttemptRecordStore` implementations with canonical schema-versioned encoding, SHA-256 record fingerprints, strict fail-closed decoding, exact attempt identity preservation, atomic compare-and-set, deterministic ordering, cancellable file/JDBC operations, and configurable validated JDBC identifiers. A shared 20-case TCK covers the in-memory, file, and JDBC contracts, while restart-level file/JDBC tests prove recovery approvals, approved idempotency keys, safe partial transitions, and retained failed-workflow evidence. Checkpoint and attempt stores remain explicitly composed and independently managed; this adds no cross-store transaction or exactly-once external-side-effect guarantee.

Expand Down
16 changes: 8 additions & 8 deletions docs/ROADMAP-0.6.0.md
Original file line number Diff line number Diff line change
Expand Up @@ -321,27 +321,27 @@ This phase is intentionally completed before large decomposition work.

## Epic 1.2: Safe error boundaries

> **Status:** in progress — tool failures (PR #219) and provider HTTP/transport failures (PR #222) are implemented. Workflow-step, persistence, MCP, shell, and structured-output boundaries remain.
> **Status:** ✅ Complete — tool failures (PR #219), provider HTTP/transport failures (PR #222), workflow-step/MCP/shell boundaries (PR #223), structured-output failures (PR #224), and persistence failures (PR #225) are implemented.

**Goal:** Separate internal diagnostic detail from public, model-visible, audit-visible, and telemetry-visible errors.

### Tasks

1. Define `SafeFailureCode` or domain-specific typed reason-code families. — **PR #219:** `ToolFailureCode`; **PR #222:** `ProviderFailureCode`; **PR #223:** `WorkflowStepFailureCode` for built-in external workflow steps. Other domains remain later slices.
1. Define `SafeFailureCode` or domain-specific typed reason-code families. — **PR #219:** `ToolFailureCode`; **PR #222:** `ProviderFailureCode`; **PR #223:** `WorkflowStepFailureCode` for built-in external workflow steps; **PR #225:** `PersistenceFailureCode`. Approval and policy domains remain later slices.
2. Define four explicit error surfaces:
- internal cause; — **PR #219:** `ToolFailureDiagnosticObserver`; **PR #222:** `ProviderFailureDiagnosticObserver`; **PR #223:** `WorkflowStepFailureDiagnosticObserver` (all fail-open and diagnostic-only).
- public caller message; — **PR #222:** fixed provider HTTP/transport messages; **PR #223:** fixed built-in workflow-step messages.
- internal cause; — **PR #219:** `ToolFailureDiagnosticObserver`; **PR #222:** `ProviderFailureDiagnosticObserver`; **PR #223:** `WorkflowStepFailureDiagnosticObserver`; **PR #225:** `PersistenceFailureDiagnosticObserver` (all fail-open and diagnostic-only).
- public caller message; — **PR #222:** fixed provider HTTP/transport messages; **PR #223:** fixed built-in workflow-step messages; **PR #225:** fixed persistence messages.
- model-visible message; — **PR #219:** `ModelVisibleToolMessage.trusted(...)`.
- audit/telemetry metadata. — **PR #222:** provider logs and telemetry expose fixed messages and trusted metadata only; other surfaces remain.
- audit/telemetry metadata. — **PR #222:** provider logs and telemetry expose fixed messages and trusted metadata only; **PR #225:** persistence raw detail remains diagnostic-observer-only while ordinary worker logs expose exception class names only. Approval surfaces remain.
3. Remove arbitrary exception messages from model-visible tool results. — **PR #219:** `ToolResult` retains its four 0.5.0 variants for exhaustive-`when` compatibility; `safeInvalidInput(...)`/`safePermanentFailure(...)` factories wrap validated or fixed text in the existing variants. Built-in engine and standalone paths never derive text from `Throwable.message`.
4. Review provider HTTP failure handling so response bodies are:
- bounded;
- sanitised;
- disabled or redacted by default;
- never copied wholesale into public exceptions. — **PR #222:** complete for provider HTTP failures; bounded previews are available only through `ProviderFailureDiagnosticObserver`.
5. Review debug logging of provider bodies and secret-related paths. — **PR #222:** complete for provider adapters; debug logs are metadata-only.
6. Centralize safe error sanitisation for shell, HTTP, MCP, tools, providers, persistence, and approvals. — **PR #222:** provider HTTP/transport helpers; **PR #223:** built-in HTTP, shell, MCP, Codex, and Hermes workflow-step helpers. Persistence and approvals remain later slices.
7. Add negative tests with tokens, prompts, paths, SQL fragments, command arguments, and malformed payloads. — **PR #219:** tool boundary; **PR #222:** provider HTTP, transport, adapter, and telemetry boundaries. Other surfaces remain.
5. Review debug logging of provider bodies and secret-related paths. — **PR #222:** complete for provider adapters; debug logs are metadata-only. **PR #225:** persistence worker logging emits exception class names only.
6. Centralize safe error sanitisation for shell, HTTP, MCP, tools, providers, persistence, and approvals. — **PR #222:** provider HTTP/transport helpers; **PR #223:** built-in HTTP, shell, MCP, Codex, and Hermes workflow-step helpers; **PR #225:** persistence boundaries. Approvals remain a later slice.
7. Add negative tests with tokens, prompts, paths, SQL fragments, command arguments, and malformed payloads. — **PR #219:** tool boundary; **PR #222:** provider HTTP, transport, adapter, and telemetry boundaries; **PR #224:** structured-output boundary; **PR #225:** persistence paths, SQL, payloads, observer behavior, cancellation, and worker failures. Approval surfaces remain.

### Acceptance criteria

Expand Down
30 changes: 26 additions & 4 deletions docs/concepts/safe-error-boundaries.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Safe Error Boundaries

> **Status:** implemented for tool execution (PR #219), provider HTTP/transport failures (PR #222), external workflow steps incl. MCP/shell (PR #223), and structured-output failures (PR #224). Persistence remains a future slice.
> **Status:** implemented for tool execution (PR #219), provider HTTP/transport failures (PR #222), external workflow steps incl. MCP/shell (PR #223), structured-output failures (PR #224), and persistence failures (PR #225). Epic 1.2 is complete.

> **Structured-output note (PR #224):** engine-produced `StructuredOutputException` instances no longer populate `originalPrompt`, `lastRawResponse`, or `validationError` — those fields stay ABI-compatible but are intentionally null for built-in runtime failures. Raw detail flows only to `StructuredOutputFailureDiagnosticObserver`; `OperationObservation.onStructuredParseFailure` receives redacted/fixed text; OTel emits typed metadata only.

Expand All @@ -27,7 +27,7 @@ Tool failures are classified with stable machine-readable codes — never by par
| `EXECUTION_FAILED` | `tool.execution.failed` | Tool execution failed |
| `RETRY_EXHAUSTED` | `tool.execution.retry_exhausted` | Tool execution failed |

Codes are domain-specific. `ToolFailureCode` covers tools, while `ProviderFailureCode` covers provider HTTP rejection, timeout, connection, transport, and unexpected failures. They classify diagnostic events and select fixed safe defaults; retry remains represented by the relevant domain contract rather than inferred from message text. Workflow, approval, persistence, and policy failures can gain their own code families without introducing a repository-wide "god enum".
Codes are domain-specific. `ToolFailureCode` covers tools, `ProviderFailureCode` covers provider HTTP rejection, timeout, connection, transport, and unexpected failures, and `PersistenceFailureCode` covers persistence failures. They classify diagnostic events and select fixed safe defaults; retry remains represented by the relevant domain contract rather than inferred from message text. Approval and policy failures can gain their own code families without introducing a repository-wide "god enum".

## Tool results

Expand Down Expand Up @@ -92,6 +92,28 @@ Provider adapters use shared safe boundaries for HTTP and transport failures:

Retry behavior remains structural: `statusCode`, `retryable`, and `retryAfterMillis` survive HTTP mapping, while transport categories select their established retryability. Cancellation input is rethrown before classification. Observer-thrown cancellation is swallowed only while the current coroutine remains active; cancellation of the enclosing job remains primary.

## Scope and non-claims (PRs #219 and #222)
## Persistence failures

These slices do not complete Epic 1.2. Built-in HTTP, shell, MCP, Codex, and Hermes workflow steps now use the same safe-boundary shape; persistence and structured-output exception fields remain outside this slice. The work does not add automatic secret detection, guarantee application-supplied trusted messages are secret-free, introduce a universal failure-code taxonomy, or change tool retry/idempotency semantics.
Persistence uses typed context rather than exception text: `PersistenceFailureCode` classifies read, write, delete, list, conflict, and corrupted-data failures; `PersistenceResourceKind` identifies checkpoints, leases, step attempts, or the worker registry; and `PersistenceOperation` identifies the affected store operation.

`PersistenceFailureDiagnosticObserver` receives a `PersistenceFailureDiagnosticEvent` containing that typed context and the original failure. It is diagnostic-only and fail-open: observer failures do not replace the safe persistence failure. Cancellation remains primary; genuine coroutine cancellation is rethrown and is not delivered as an ordinary diagnostic failure.

Built-in public failures use fixed text only:

- `Workflow checkpoint conflict`;
- `Workflow lease conflict`;
- `Persisted workflow checkpoint is invalid`;
- `Persisted step-attempt record is invalid`;
- `Workflow persistence read/write/delete/list failed`.

File, markdown, JDBC, and in-memory checkpoint stores; file and JDBC lease stores; and file and JDBC step-attempt stores wrap every persistence operation in `persistenceBoundary`. Raw paths, SQL, and persisted payloads reach only the diagnostic observer. `StaleWorkflowLeaseException` remains a semantic class for worker fencing, with the fixed text `Workflow lease is no longer active`.

Ordinary worker callbacks (`onPollFailed`, `onLeaseRenewalFailed`, `onLeaseReleaseFailed`, and `onStepAttemptFailed`) receive safe failures rather than raw persistence exceptions. `LoggingTramaiWorkerObserver` logs their exception class names only.

The persistence exception constructors remain ABI-compatible. Class-body `failureCode` and `safeFactoryTrusted` properties and additive observer-taking store constructors extend the API without changing those constructor descriptors. The binary fixture exercises v0.5.0 persistence exception and store constructors.

This boundary does not add cross-store transactions, schema changes, encryption, retries, lease-semantic changes, or cancellation-semantic changes.

## Scope and non-claims (PRs #219–#225)

These slices complete Epic 1.2. Built-in HTTP, shell, MCP, Codex, and Hermes workflow steps, structured-output failures, and persistence failures now use the same safe-boundary shape; approval-gateway exception fields remain outside this slice. The work does not add automatic secret detection, guarantee application-supplied trusted messages are secret-free, introduce a universal failure-code taxonomy, or change tool retry/idempotency semantics.
23 changes: 23 additions & 0 deletions docs/guides/orchestration-persistence.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,29 @@ The important contract detail is revision handling:

That gives database, object-store, and filesystem implementations the same optimistic-concurrency model.

## Safe Failure Boundaries

Built-in persistence stores expose fixed-text public failures. They do not copy filesystem paths, SQL, persisted payloads, or arbitrary storage exception messages into public exceptions, ordinary worker callbacks, or default logs. The original failure is available only through `PersistenceFailureDiagnosticObserver`.

Configure the observer with a built-in store's additive observer constructor overload:

```kotlin
val checkpointStore = FileWorkflowCheckpointStore(
Path.of(".tramai/workflows"),
DefaultWorkflowCheckpointPathStrategy("checkpoint.properties"),
PersistenceFailureDiagnosticObserver { event ->
// Route diagnostic-only event.failure to an access-controlled sink.
diagnosticSink.record(event.resourceKind, event.operation, event.failureCode, event.failure)
},
)
```

The observer is fail-open: an observer failure does not replace the persistence failure. Cancellation remains first: genuine coroutine cancellation propagates rather than becoming a persistence diagnostic. This is an API-level constructor setting; no YAML or configuration property is required.

Checkpoint conflicts, lease conflicts, invalid checkpoints, invalid step-attempt records, and general read/write/delete/list failures have fixed public text. `StaleWorkflowLeaseException` remains the semantic fencing signal with fixed text. In a worker, `onPollFailed`, `onLeaseRenewalFailed`, `onLeaseReleaseFailed`, and `onStepAttemptFailed` receive safe failures; `LoggingTramaiWorkerObserver` logs only their exception class names.

The built-in file, markdown, JDBC, and in-memory checkpoint stores; file and JDBC lease stores; and file and JDBC step-attempt stores apply this boundary to every persistence operation. Custom stores should preserve the same separation when they handle raw backend errors.

## Wiring A Workflow

Once you have a codec and store, attach them through `WorkflowPersistence`:
Expand Down
Loading
Loading