From 9b3be17c4e6161f9fe3a0980e9411997e380ea2a Mon Sep 17 00:00:00 2001 From: Artur Shiriev Date: Fri, 5 Jun 2026 22:13:53 +0300 Subject: [PATCH 01/18] docs(spec): extension-slot docs design (Epic 3 story 3-6) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the deferred-tutorial half of story 3-6 (the docs-sync-0.4 pass shipped the freshness half; the walkthrough was explicitly punted there). Scope: one new docs/middleware.md page (~150 lines) — Middleware protocol, phase decorators, Request-ID propagation example, when NOT to write a middleware. Plus four small touchups (mkdocs nav, README pointer, docs/index.md pointer, engineering.md §8 SHIPPED line) and release notes for 0.7.0. Non-resilience example by design — Request-ID pairs naturally with 0.6.0's observability events (correlate via the URL attribute) without shipping a half-baked CircuitBreaker that would get cargo-culted. Closes Epic 3. --- .../2026-06-05-extension-slot-docs-design.md | 138 ++++++++++++++++++ 1 file changed, 138 insertions(+) create mode 100644 planning/specs/2026-06-05-extension-slot-docs-design.md diff --git a/planning/specs/2026-06-05-extension-slot-docs-design.md b/planning/specs/2026-06-05-extension-slot-docs-design.md new file mode 100644 index 0000000..9f607dc --- /dev/null +++ b/planning/specs/2026-06-05-extension-slot-docs-design.md @@ -0,0 +1,138 @@ +# Spec: Extension-slot docs (Epic 3 story 3-6) + +**Date:** 2026-06-05 +**Topic slug:** `extension-slot-docs` +**Status:** drafted, awaiting user review +**Target release:** 0.7.0 (docs-only minor) +**Epic 3 stories closed:** 3-6 (the last leftover). Closes Epic 3 entirely. + +## Purpose + +Document `httpware`'s primary extension point — the **Middleware protocol** — as a user-facing page so library consumers can write their own cross-cutting middleware (request-ID propagation, auth header injection, custom resilience policies, structured tracing, etc.) without reading the source. + +This is the deferred-tutorial half of story 3-6. The docs-sync-0.4 pass (PR #25) shipped the freshness fixes and explicitly punted "_write your own middleware_" walkthrough to a future docs PR. This is that PR. + +## Background — how 3-6 got here + +- **Original framing (pre-pivot):** "Document the extension slot for custom resilience policies." A tutorial framed around hand-rolling CircuitBreaker / RateLimiter / custom backoff. +- **docs-sync-0.4 re-scope:** Folded the *freshness* half of 3-6 into a 0.3→0.4 docs catch-up PR; explicitly deferred the tutorial. +- **This spec:** Closes the tutorial half, scoped to **the Middleware seam only** (Seam A in `engineering.md §3`). ResponseDecoder (Seam B) and the optional-extras pattern (Seam C) stay contributor-facing in `engineering.md` — surfacing them in user docs over-promises an extension surface users shouldn't be touching. +- **Worked-example flavor:** non-resilience (Request-ID propagation) rather than CircuitBreaker. Demonstrates the protocol applies to anything cross-cutting, pairs naturally with the 0.6.0 observability events (correlate a `httpware.retry` record's `url` attribute with the X-Request-Id the middleware set), and avoids shipping a half-baked CircuitBreaker that would get cargo-culted into production. + +## Deliverable + +### New page: `docs/middleware.md` + +Approximately 150 lines markdown, structured as: + +1. **Intro (~5 lines).** What a middleware is in httpware; cross-cutting concerns it's the right tool for (auth, tracing, logging, custom resilience). Pointer to built-in `Retry`/`Bulkhead` for the common cases. + +2. **The Middleware protocol (~25 lines).** The `Middleware` `Protocol` and `Next` type alias, both already exported from `httpware.middleware`: + + ```python + from collections.abc import Awaitable, Callable + from typing import Protocol, TypeAlias, runtime_checkable + import httpx2 + + Next: TypeAlias = Callable[[httpx2.Request], Awaitable[httpx2.Response]] + + @runtime_checkable + class Middleware(Protocol): + async def __call__(self, request: httpx2.Request, next: Next) -> httpx2.Response: ... + ``` + + Explain: chain composed at `AsyncClient.__init__`, frozen for the client's lifetime. First in the `middleware=[...]` list is outermost (so `[Bulkhead, Retry]` puts Bulkhead outside Retry — one slot covers all attempts). `await next(request)` invokes the next layer; returning without calling it short-circuits the chain (synthesize a `Response` directly). + +3. **Phase decorators (~25 lines).** `@before_request`, `@after_response`, `@on_error` from `httpware.middleware` as ergonomic shortcuts for the common cases: + + - **Use these when:** you don't need state-keeping on `self`, and you don't need to wrap the full `await next(...)` call. + - **Reach for the raw Protocol when:** you need instance state (e.g., a counter), you need to inspect both the request AND the response (e.g., timing), or you need to interleave behavior around the call (e.g., circuit-breaker state mutation on both success and failure paths). + + Show one minimal pair — a `@before_request` adding a header, and a `@on_error` translating an exception type — without dwelling. + +4. **Worked example: Request-ID propagation (~50 lines).** Full class-based middleware demonstrating the raw `Middleware` protocol with state-keeping (a configurable header name) plus both phases (set request header before forwarding, log the ID after the response). Uses `logging.getLogger("myapp.request_id")` — explicitly a *consumer* logger, NOT a `httpware.*` logger, to reinforce that the `httpware.*` namespace is reserved for library-emitted events. The example: + + ```python + import logging + import uuid + + import httpx2 + from httpware import AsyncClient, Retry + from httpware.middleware import Next + + _LOGGER = logging.getLogger("myapp.request_id") + + + class RequestIdMiddleware: + """Propagate a per-call X-Request-Id; log it on response. + + Place OUTSIDE Retry so all attempts of the same call share one ID + (callable from the consumer's logs to httpware.retry's emitted events + via the matching `url` attribute). + """ + + def __init__(self, *, header: str = "X-Request-Id") -> None: + self._header = header + + async def __call__(self, request: httpx2.Request, next: Next) -> httpx2.Response: # noqa: A002 + request_id = str(uuid.uuid4()) + request.headers[self._header] = request_id + response = await next(request) + _LOGGER.info("request complete", extra={"request_id": request_id, "status": response.status_code}) + return response + + + async def main() -> None: + async with AsyncClient( + base_url="https://api.example.com", + middleware=[RequestIdMiddleware(), Retry()], # ID outside Retry + ) as client: + await client.get("/users/1") + ``` + + Brief paragraph after: "Correlate with the 0.6.0 observability events — a `httpware.retry` `retry.giving_up` record carries the same `url` your middleware logged the ID against." + +5. **When NOT to write a middleware (~15 lines).** Tight callbacks to existing patterns: + - **Redaction:** use a `logging.Filter` on the consumer side (per the 0.6.0 observability spec's no-redaction-in-httpware stance). + - **URL / header validation:** `httpx2` owns it; don't reimplement. + - **Per-call behavior with no cross-cutting state:** pass through `request.extensions=` or the call-site `extensions=` kwarg instead. + - **Span creation for HTTP tracing:** install `opentelemetry-instrumentation-httpx` — don't write an OTel middleware in httpware (see `engineering.md §8` for why `5-4` was retired). + +6. **Cross-references (~5 lines).** + - `engineering.md §3 Seam A` — the formal protocol contract + - `src/httpware/middleware/resilience/` — `Retry`, `Bulkhead`, `RetryBudget` as real-world examples reading the same protocol + - `docs/index.md#with-resilience-middleware` — composition with built-ins + +### Touchups + +- **`mkdocs.yml`:** add `- Middleware: middleware.md` to the nav, between `Quick-Start` and `Development`. +- **`README.md`:** in the existing "With resilience middleware" subsection, append one sentence: "_Need a custom middleware (auth, tracing, request-ID propagation)? See [`docs/middleware.md`](docs/middleware.md)._" +- **`docs/index.md`:** in the "Where to go next" section, add one bullet: "**[Middleware guide](middleware.md)** — write your own middleware (Request-ID example included)." +- **`planning/engineering.md` §8:** replace the existing Epic 3 closing line ("**Remaining:** `3-6` extension-slot docs.") with: "**Epic 3 — Resilience: SHIPPED.** v0.4 shipped `Retry` + `RetryBudget` + `Bulkhead`; v0.7 ships `3-6` extension-slot docs (`docs/middleware.md`)." +- **`planning/releases/0.7.0.md`:** new file. Short doc-only release notes — calls out the new middleware guide, closes Epic 3, no API changes. + +## Non-goals (explicit) + +- **No code changes.** This is a docs-only PR. No middleware additions, no protocol extensions, no new public exports. +- **No CircuitBreaker / RateLimiter / custom-resilience example.** The user explicitly chose a non-resilience example to avoid shipping a half-baked toy that gets cargo-culted. +- **No ResponseDecoder (Seam B) or optional-extras (Seam C) coverage.** Those stay in `engineering.md` (contributor-facing). +- **No mkdocs publish / docs-site infra work.** That's Epic 6 story `6-2`; the site_url is still readthedocs.io and we don't try to make it actually publish here. +- **No version bump in `pyproject.toml`.** Tag-driven release (`uv version $GITHUB_REF_NAME` overwrites at build time). +- **No `# noqa`s in the example code beyond `# noqa: A002`** (matches the convention already in `src/httpware/middleware/__init__.py` for the `next` parameter name). +- **No CLAUDE.md changes.** + +## Verification gates + +- `uv run --with mkdocs --with mkdocs-material mkdocs build --strict 2>&1 | tail -10` → 0 warnings (matches the gate the 0.6.0 work used). +- All cross-reference links in the new page and the README/docs touchups resolve. +- The Request-ID example compiles under `ty` if extracted (verified locally during implementation; not committed as a test). +- Architecture-invariant grep suite still PASSes (no source files modified, but the grep should run anyway for hygiene). +- Full test suite still passes (no code changes, but `just test` should be a no-op confirmation). + +## Release shape + +- **Version:** 0.7.0 (semver minor — public docs surface grows but no API). +- **Branch:** `feat/v0.7-middleware-docs`. +- **PR:** docs-only, expected ~250 lines markdown net new. +- **Tag:** `0.7.0` after merge; GitHub Release reads from `planning/releases/0.7.0.md`. +- **Publish workflow:** unchanged — the tag-driven publish runs even for docs-only releases, but the only artifact difference is the package metadata's classifier set is unchanged. From 9adddb35434da72f4a7207308529465cf9019739 Mon Sep 17 00:00:00 2001 From: Artur Shiriev Date: Fri, 5 Jun 2026 22:17:41 +0300 Subject: [PATCH 02/18] docs(plan): extension-slot docs implementation plan (Epic 3 story 3-6) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 7 tasks for docs-only PR feat/v0.7-middleware-docs: 1. Branch + docs/middleware.md (~150 lines markdown — full page content) 2. mkdocs.yml nav entry + strict-build verification 3. README.md pointer in 'With resilience middleware' subsection 4. docs/index.md bullet in 'Where to go next' 5. engineering.md §8 — mark 3-6 SHIPPED, Epic 3 closed 6. planning/releases/0.7.0.md release notes 7. Final verification + push (mkdocs --strict, lint sanity, test sanity, push branch) Each task is a single commit. No source code changes; verification is mkdocs strict build + the existing test/lint suites as no-op confirmation. Spec at planning/specs/2026-06-05-extension-slot-docs-design.md. --- .../2026-06-05-extension-slot-docs-plan.md | 532 ++++++++++++++++++ 1 file changed, 532 insertions(+) create mode 100644 planning/plans/2026-06-05-extension-slot-docs-plan.md diff --git a/planning/plans/2026-06-05-extension-slot-docs-plan.md b/planning/plans/2026-06-05-extension-slot-docs-plan.md new file mode 100644 index 0000000..5854561 --- /dev/null +++ b/planning/plans/2026-06-05-extension-slot-docs-plan.md @@ -0,0 +1,532 @@ +# Extension-slot docs (0.7.0, Epic 3 story 3-6) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Ship `docs/middleware.md` — a user-facing guide to writing custom middleware against `httpware`'s Middleware protocol — plus the four small touchups that hang off it (mkdocs nav, README pointer, docs/index pointer, engineering.md §8 SHIPPED line) and 0.7.0 release notes. Closes Epic 3. + +**Architecture:** Docs-only PR. One new markdown page (~150 lines), four small textual edits to existing files, one new release-notes file. No source code changes. Verification is `mkdocs build --strict` + link resolution + the existing test/lint suites as no-op confirmation. + +**Tech Stack:** Markdown, mkdocs-material (strict build), no source code. + +**Target branch:** `feat/v0.7-middleware-docs`. Create from `main` before Task 1: `git checkout main && git pull && git checkout -b feat/v0.7-middleware-docs`. + +**Source spec:** [`planning/specs/2026-06-05-extension-slot-docs-design.md`](../specs/2026-06-05-extension-slot-docs-design.md). Read the spec's "Background" + "Deliverable" sections before starting — the *why* for non-resilience example choice and Seam-A-only scope lives there. + +--- + +## File structure + +**New files:** +- `docs/middleware.md` — the guide itself (~150 lines) +- `planning/releases/0.7.0.md` — release notes + +**Modified files:** +- `mkdocs.yml` — add nav entry between Quick-Start and Development +- `README.md` — one-sentence pointer in the existing "With resilience middleware" subsection +- `docs/index.md` — one bullet in the existing "Where to go next" section +- `planning/engineering.md` §8 — replace the "**Remaining:** `3-6` extension-slot docs." line under Epic 3 + +**Commit cadence:** one commit per task. Per-task commits keep history reviewable. + +--- + +## Task 1: Branch + create `docs/middleware.md` + +**Files:** +- Create: `docs/middleware.md` + +- [ ] **Step 1: Create the branch** + +```bash +git checkout main && git pull && git checkout -b feat/v0.7-middleware-docs +``` +Expected: switched to a new branch. + +- [ ] **Step 2: Create `docs/middleware.md` with the full content below** + +````markdown +# Writing custom middleware + +`httpware`'s primary extension point is the **Middleware protocol**. Middleware lets you add cross-cutting behavior — request-ID propagation, auth header injection, structured tracing, custom resilience policies, anything that wraps "send a request, get a response" — without subclassing `AsyncClient` or touching the transport. + +The built-in `Retry` and `Bulkhead` middleware are themselves implementations of this protocol; nothing about them is privileged. If you want a circuit breaker, a rate limiter, or a header-injecting auth layer, write a middleware. If your need is per-call (not cross-cutting), pass it through `request.extensions=` instead. + +## The protocol + +Two symbols, both exported from `httpware.middleware`: + +```python +from collections.abc import Awaitable, Callable +from typing import Protocol, TypeAlias, runtime_checkable +import httpx2 + +Next: TypeAlias = Callable[[httpx2.Request], Awaitable[httpx2.Response]] + + +@runtime_checkable +class Middleware(Protocol): + async def __call__(self, request: httpx2.Request, next: Next) -> httpx2.Response: ... +``` + +The chain is composed once at `AsyncClient.__init__` and frozen for the client's lifetime. The first entry in `middleware=[...]` is the outermost layer: when you write `middleware=[Bulkhead(...), Retry()]`, the bulkhead sees every request before the retry layer does, so one slot covers all retry attempts of the same call. + +Calling `await next(request)` forwards to the next layer (or, eventually, to the terminal that hits `httpx2`). You can: + +- **Forward unchanged:** `return await next(request)` +- **Modify the request first:** mutate `request.headers` (or build a replacement) before forwarding +- **Inspect or replace the response:** call `await next(...)`, then act on what comes back +- **Short-circuit:** return a synthesized `httpx2.Response` without calling `next` at all +- **Wrap the call in error handling:** `try: return await next(...) except ...` to translate failures + +Whatever you do, return an `httpx2.Response`. Raising an exception propagates up the chain (Retry catches retryable exceptions; everything else surfaces to the caller). + +## Phase decorators + +For the common cases where you don't need state-keeping on `self` and don't need to wrap the full `await next(...)` call, `httpware.middleware` exports three decorators that turn a single async function into a `Middleware`: + +```python +from httpware.middleware import before_request, after_response, on_error +``` + +| Decorator | Function signature | When to use | +|---|---|---| +| `@before_request` | `async (request) -> request` | Transform the outgoing request (add a header, rewrite a URL). | +| `@after_response` | `async (request, response) -> response` | Transform the incoming response (decode, log, attach metadata). | +| `@on_error` | `async (request, exc) -> response \| None` | Translate or absorb a failure. Return `None` to re-raise. Catches `Exception` (not `BaseException`), so `asyncio.CancelledError` propagates. | + +Brief example — adding an `Authorization` header before every request: + +```python +import httpx2 + +from httpware import AsyncClient +from httpware.middleware import before_request + + +@before_request +async def add_bearer(request: httpx2.Request) -> httpx2.Request: + request.headers["Authorization"] = "Bearer secret-token" + return request + + +async def main() -> None: + async with AsyncClient(base_url="https://api.example.com", middleware=[add_bearer]) as client: + await client.get("/me") +``` + +**Reach for the raw `Middleware` protocol when:** you need instance state (a counter, a CircuitBreaker's open/closed flag), you need to inspect both the request AND its response (e.g., timing), or you need to interleave behavior around the `await next(...)` call (e.g., emit one log line at the start and one at the end). The decorators are a convenience for the cases where a single function suffices. + +## Worked example: request-ID propagation + +A `RequestIdMiddleware` that assigns a per-call UUID, injects it as an outgoing header, and logs it alongside the response status. This is the canonical "trace every request through your distributed system" pattern. + +```python +import logging +import uuid + +import httpx2 + +from httpware import AsyncClient, Retry +from httpware.middleware import Next + + +_LOGGER = logging.getLogger("myapp.request_id") + + +class RequestIdMiddleware: + """Assign a per-call X-Request-Id; log it on response. + + Place OUTSIDE Retry so all attempts of the same call share one ID + (so a single call's retries all surface under the same correlation + key in your logs, and match the URL attribute on httpware.retry's + emitted events). + """ + + def __init__(self, *, header: str = "X-Request-Id") -> None: + self._header = header + + async def __call__(self, request: httpx2.Request, next: Next) -> httpx2.Response: # noqa: A002 + request_id = str(uuid.uuid4()) + request.headers[self._header] = request_id + response = await next(request) + _LOGGER.info( + "request complete", + extra={"request_id": request_id, "status": response.status_code}, + ) + return response + + +async def main() -> None: + async with AsyncClient( + base_url="https://api.example.com", + middleware=[RequestIdMiddleware(), Retry()], # ID outside Retry + ) as client: + await client.get("/users/1") +``` + +A note on logger names: the example logs under `myapp.request_id`, NOT under `httpware.*`. The `httpware.*` namespace is reserved for events emitted by the library itself (see [Observability](index.md#observability) — `httpware.retry` and `httpware.bulkhead` are stable contracts). Consumer middleware should use your application's own logger namespace. + +The example pairs naturally with the 0.6.0 observability events: a `httpware.retry` `retry.giving_up` log record carries a `url` attribute, and your `RequestIdMiddleware` set an `X-Request-Id` for that same call. Correlate the two in your log aggregator and you have end-to-end visibility from "this user's request" to "we gave up after N retries." + +## When NOT to write a middleware + +- **Redaction:** Use a `logging.Filter` on the consumer side. `httpware` deliberately does no redaction in-library (per the 0.6.0 observability design). +- **URL or header validation:** `httpx2` owns it. Don't reimplement. +- **Per-call behavior that doesn't apply to other calls:** Pass through `request.extensions=` (or the `extensions=` kwarg at the call site) instead. Middleware exists for *cross-cutting* concerns. +- **HTTP-level span creation for tracing:** Install `opentelemetry-instrumentation-httpx` instead of writing an OTel middleware in httpware. We retired story `5-4` (standalone OTel middleware) for this reason — `opentelemetry-instrumentation-httpx` already covers transport-level tracing, and a separate httpware layer would duplicate it. See `planning/engineering.md` §8. + +## See also + +- **`planning/engineering.md` §3 (Seam A)** — the formal protocol contract and why the chain is frozen at construction. +- **`src/httpware/middleware/resilience/`** — `Retry`, `Bulkhead`, `RetryBudget` as real-world consumers of this exact protocol. +- **[Quick-Start composition example](index.md#with-resilience-middleware)** — composing built-in middleware. +```` + +- [ ] **Step 3: Commit** + +```bash +git add docs/middleware.md +git commit -m "docs(middleware): write custom-middleware guide (3-6) + +New docs/middleware.md covering: +- The Middleware Protocol + Next type, exported from httpware.middleware +- Phase decorators (@before_request, @after_response, @on_error) as + ergonomic shortcuts for the no-state-keeping cases +- Worked example: a RequestIdMiddleware that assigns a per-call UUID + via X-Request-Id and logs it alongside the response status. Placed + outside Retry on purpose so all attempts of the same call share one + ID and correlate with the 0.6.0 observability events' url attribute +- 'When NOT to write a middleware' section covering redaction (use a + logging.Filter), URL/header validation (httpx2 owns it), per-call + behavior (use request.extensions=), and HTTP-tracing (install + opentelemetry-instrumentation-httpx instead) + +Closes the deferred-tutorial half of story 3-6. See spec at +planning/specs/2026-06-05-extension-slot-docs-design.md." +``` + +--- + +## Task 2: Add nav entry to `mkdocs.yml` + verify strict build + +**Files:** +- Modify: `mkdocs.yml` + +- [ ] **Step 1: Add nav entry** + +The current `nav:` block reads: +```yaml +nav: + - Quick-Start: index.md + - Development: + - Contributing: dev/contributing.md +``` + +Change to: +```yaml +nav: + - Quick-Start: index.md + - Middleware: middleware.md + - Development: + - Contributing: dev/contributing.md +``` + +- [ ] **Step 2: Verify mkdocs strict build is clean** + +```bash +uv run --with mkdocs --with mkdocs-material mkdocs build --strict 2>&1 | tail -20 +``` +Expected: `Documentation built in