feat: Extensible Cofferdam HTTP Transport Pipeline

[brian-pond]

Proposal: Extensible Cofferdam HTTP Transport Pipeline

Summary

Add a richer, extensible HTTP client API to Cofferdam so application integrations can reuse Cofferdam's policy enforcement, credential handling, redirect safety, and redacted evidence model without losing application-specific ledgering, response classification, correlation IDs, or retry/replay semantics.

The current cofferdam.http.request(...) helper is useful for simple calls, but larger integrations need more structured lifecycle hooks and richer request/response result objects.

Problem

Some applications need Cofferdam to be the required outbound safety boundary, but they also need their own integration ledger and domain-specific transport handling.

Example: an ERPNext-to-Windmill integration needs to:

• run Cofferdam preflight before any bytes leave the server
• resolve bearer tokens only after the policy allows the call
• record allow/deny decisions in an application-owned attempt ledger
• capture exact HTTP status, response headers, elapsed time, raw body, parsed JSON, and exceptions
• attach correlation IDs and idempotency keys
• classify HTTP success separately from application-level failure
• treat a Windmill synchronous HTTP 200 response containing a script error as a failed Windmill execution
• ensure retry/replay/scheduler/background-worker/user-triggered sends all go through the same policy-checked transport

The application can build all of this around raw httpx, but then Cofferdam's HTTP helper becomes hard to reuse directly. Ideally, Cofferdam should provide a policy-enforced transport pipeline that is extensible enough for these integrations.

Design Goal

Cofferdam should own:

• policy preflight
• fail-closed behavior
• host/method/operation/credential checks
• secret resolution and injection
• redirect revalidation before credential forwarding
• no-secret request/response snapshots
• structured policy decisions

Application code should own:

• application ledger writes
• correlation ID and idempotency conventions
• response classification
• business outcome classification
• retry/replay semantics
• framework-specific persistence
• alerting and operational workflows

Proposed API Shape

Introduce a second-generation API alongside the current convenience helper:

from cofferdam.http import CofferdamHttpClient, CofferdamHttpRequest

Request Object

from dataclasses import dataclass, field
from typing import Mapping

from cofferdam.models import Policy


@dataclass(frozen=True)
class CofferdamHttpRequest:
    policy: Policy
    integration: str
    kind: str
    operation: str
    method: str
    url: str
    credential: str | None = None
    headers: Mapping[str, str] = field(default_factory=dict)
    json: object | None = None
    data: object | None = None
    content: bytes | None = None
    timeout: float | object | None = None
    follow_redirects: bool = False
    correlation_id: str | None = None
    metadata: Mapping[str, object] = field(default_factory=dict)

Result Object

from dataclasses import dataclass

from cofferdam.decisions import Decision


@dataclass(frozen=True)
class CofferdamHttpResult:
    decision: Decision
    request: RedactedRequestSnapshot
    response: ResponseSnapshot | None
    exception: ExceptionSnapshot | None
    elapsed_ms: int
    redirect_chain: tuple[ResponseSnapshot, ...] = ()

The result should preserve transport evidence without exposing secret values.

Snapshot Objects

@dataclass(frozen=True)
class RedactedRequestSnapshot:
    method: str
    url: str
    host: str
    headers: Mapping[str, str]
    credential: str | None
    integration: str
    kind: str
    operation: str
    correlation_id: str | None = None
    metadata: Mapping[str, object] = field(default_factory=dict)


@dataclass(frozen=True)
class ResponseSnapshot:
    status_code: int
    headers: Mapping[str, str]
    text: str
    json_value: object | None
    elapsed_ms: int


@dataclass(frozen=True)
class ExceptionSnapshot:
    type: str
    message: str

Headers should be redacted by default for sensitive names such as:

• Authorization
• Cookie
• Set-Cookie
• X-Api-Key
• X-Auth-Token

Hook / Observer Pattern

Support lifecycle hooks via a protocol or callbacks. Prefer composition over inheritance.

from typing import Protocol


class CofferdamHttpObserver(Protocol):
    def before_decision(self, event: BeforeDecisionEvent) -> None: ...
    def after_decision(self, event: AfterDecisionEvent) -> None: ...
    def before_send(self, event: BeforeSendEvent) -> None: ...
    def after_response(self, event: AfterResponseEvent) -> None: ...
    def on_exception(self, event: ExceptionEvent) -> None: ...

Example usage:

client = CofferdamHttpClient(
    policy=policy,
    http_client=httpx.Client(timeout=10),
    observers=[
        integration_attempt_ledger,
    ],
)

result = client.request(
    CofferdamHttpRequest(
        policy=policy,
        integration="windmill",
        kind="vendor_api",
        operation="run_wait_result",
        method="POST",
        url=url,
        credential="windmill_default",
        headers={"X-Correlation-ID": correlation_id},
        json=envelope,
        correlation_id=correlation_id,
        metadata={"message_id": message_id},
    )
)

An application observer could create/update its own ledger without Cofferdam needing to know about the framework:

class IntegrationAttemptLedger:
    def after_decision(self, event: AfterDecisionEvent) -> None:
        record_cofferdam_decision(event.decision.as_log_dict())

    def before_send(self, event: BeforeSendEvent) -> None:
        mark_attempt_sending(event.request)

    def after_response(self, event: AfterResponseEvent) -> None:
        record_transport_response(event.result)

    def on_exception(self, event: ExceptionEvent) -> None:
        record_transport_exception(event.result)

Response Classification

Cofferdam should keep application success separate from transport success.

It should provide transport-level facts:

• policy allowed/denied
• request sent/not sent
• HTTP completed/failed
• response status/body/headers
• exception details
• elapsed time

It should not decide whether a response body means application success.

Optionally support a classifier callback:

class ResponseClassifier(Protocol):
    def classify(self, result: CofferdamHttpResult) -> object: ...

For example, an ERPNext Windmill integration could classify:

• HTTP timeout -> transport_timeout
• HTTP 401/403 -> windmill_auth_failed
• HTTP 200 with Windmill script error body -> windmill_execution_failed
• HTTP 200 with expected payload -> transport_succeeded

Cofferdam can pass classifier output through without understanding the application domain.

Request Mutation Rules

The API should make it hard to accidentally bypass policy decisions:

• callers may add headers such as correlation IDs before the policy decision
• Cofferdam injects secrets only after an allow decision
• URL, method, operation, integration, kind, host, and credential must not change after the decision unless Cofferdam re-runs policy
• every redirect target must be revalidated before credentials are forwarded
• redacted snapshots must never include resolved secret values

Dry-Run / Preflight API

Expose a no-network preflight API:

decision = client.preflight(
    CofferdamHttpRequest(
        policy=policy,
        integration="windmill",
        kind="vendor_api",
        operation="run_wait_result",
        method="POST",
        url=url,
        credential="windmill_default",
    )
)

This lets applications record denied attempts cleanly without preparing or sending a live request.

Compatibility

Keep the existing cofferdam.http.request(...) helper as a convenience wrapper over the richer client.

Simple users can continue to do:

from cofferdam.http import request

response = request(
    policy=policy,
    integration="vendor",
    operation="sync",
    method="POST",
    url=url,
    credential="vendor_token",
    json=payload,
)

Advanced users can opt into:

• structured request objects
• structured result objects
• lifecycle observers
• custom httpx.Client / test transports
• classifier callbacks
• dry-run preflight

Acceptance Criteria

• Add a structured request object for policy-checked HTTP calls.
• Add a structured result object that preserves decision, redacted request snapshot, response snapshot, exception snapshot, elapsed time, and redirect chain.
• Add observer/callback hooks for before decision, after decision, before send, after response, and exception.
• Allow callers to supply their own httpx.Client or transport for pooling, timeouts, proxies, and tests.
• Ensure credentials are resolved and injected only after an allow decision.
• Ensure resolved secrets never appear in exceptions, snapshots, hook events, or logs.
• Revalidate every redirect target before forwarding credentials.
• Provide a dry-run/preflight API that performs no network I/O.
• Keep existing cofferdam.http.request(...) working as a simple wrapper.
• Add tests covering allow, deny, missing credential, secret resolution failure, timeout, non-2xx response, redirect revalidation, redaction, hook ordering, and custom transport usage.

Notes

This would let integrations such as ERPNext-to-Windmill use Cofferdam's HTTP layer directly while still preserving application-owned ledgering, response classification, correlation IDs, retry/replay behavior, and operational evidence.

The core design principle is:

Cofferdam owns outbound execution authority. Applications own business semantics and operational records.