docs: restructure DESIGN.md + surface the cross-domain examples

docs/architecture.md

@@ -121,6 +121,42 @@ and amortized across every sample that touches it; **read concurrency
 total memory is the budget + the prefetch queue (depth `d`) + the in-flight tiles —
 every term a tunable cap, none scaling with batch size or epoch length.
 
+### Event-loop ownership (fsspec backends)
+
+A genuinely-async fsspec backend (gcsfs, s3fs) binds its aiohttp session to the *first
+event loop that awaits it*, permanently — no constructor knob pins it. For a zarr store
+that first loop is **zarr's** process-wide store-IO loop (`zarr.core.sync._get_loop()`),
+because any zarr sync call (`open_geometries`, `xr.open_zarr`, user code) touches the
+store first. That is the correct owner: the session living on zarr's loop is what keeps
+the store drivable by *any* zarr code — so insitubatch **conforms** (routes its reads
+there) rather than hijacking the session onto its own loop, which would break every other
+zarr-sync consumer in the process. The scheduler keeps its own orchestration loop and
+bridges each fsspec read to zarr's loop (`run_coroutine_threadsafe`); obstore is
+loop-agnostic (Rust runtime) and is awaited inline. Teardown closes the session on its own
+loop (`close_store`), so gcsfs's finalizer — which wrongly targets
+`fsspec.asyn.get_loop()` — becomes a no-op.
+
+```mermaid
+flowchart TD
+    subgraph proc["insitubatch process (one training run)"]
+      OG["any zarr sync call<br/>open_geometries / xr.open_zarr"]
+      SL["Scheduler loop<br/>own loop, new thread per pass<br/>orchestration + decode/scatter pool"]
+      ZL["zarr store-IO loop<br/>zarr.core.sync._get_loop()"]
+      SESS[("gcsfs aiohttp session")]
+      OBS["obstore ObjectStore<br/>Rust tokio runtime, loop-agnostic"]
+      FL["fsspec.asyn.get_loop()<br/>NOT where the session lives"]
+    end
+    OG -->|"first await creates the session here"| ZL
+    ZL -->|owns| SESS
+    SL -->|"fsspec read: run_coroutine_threadsafe (bridge)"| ZL
+    SL -->|"obstore read: await inline"| OBS
+    FL -.->|"gcsfs finalizer wrongly targets this;<br/>close_store fixes the mismatch at teardown"| SESS
+```
+
+Collapsing the two loops into one — running the scheduler's orchestration *on* zarr's
+loop for all backends, deleting the bridge and the per-pass thread churn — is planned
+under M-GCS in [DESIGN.md](https://github.com/emfdavid/insitubatch/blob/main/DESIGN.md).
+
 ## Sample geometry — the axis-role contract
 
 This is the stable contract the engine commits to, and the extension points reserved

docs/benchmarks.md

@@ -101,6 +101,35 @@ grid (`era5_fat_g4/g16/g36`):
 You dial throughput from 23 → ~1130 MB/s purely via `max_inflight`, at **constant memory**
 — concurrency and residency are independent knobs.
 
+### The acceptance gate that justified the rewrite (exp_c, c6id.8xlarge)
+
+Kept for the record: the v1-vs-V2 A/B that decided the scheduler rewrite. Fat data
+(`sample_chunk=200`, ~830 MB outer chunks), in-region S3, median of 5.
+
+**v1 baseline** — read concurrency rode on `block_chunks`, so throughput *peaked* at the
+smallest window and fell as the window grew (the nested caps overshoot):
+
+| `block_chunks` (≈ resident) | single-inner MB/s | spatial (grid 15, read_conc 16) MB/s |
+|---|--:|--:|
+| 2 (~3.3 GB) | 76 | **930** |
+| 4 (~6.6 GB) | 120 | 871 |
+| 8 (~13 GB) | 178 | 724 (oversubscribed) |
+
+**V2 result** — same box, `block_chunks=2` fixed, sweeping the single `max_inflight` budget:
+
+| `max_inflight` | 8 | 16 | **32** | 64 | 128 |
+|---|--:|--:|--:|--:|--:|
+| MB/s | 388 | 788 | **1052** | 970 | 970 |
+| `resident` (chunks) | 4 | 4 | 4 | 4 | 4 |
+
+V2 **beats** the v1 spatial peak (1052 vs 930) at the *same* low memory, and residency is
+**flat at `2×block_chunks` for every `max_inflight`** — concurrency dialed independently of
+memory. Past the knee (`mi≈32`) throughput settles to a stable 970 plateau instead of v1's
+collapse to 724 under oversubscription. Re-run after the B2 admission rewrite
+(`resident_cap` → byte-budget pin/LRU) to confirm no regression: `981 / 981 / 988` MB/s at
+`mi = 32 / 64 / 128`, `resident = 4` throughout — the plateau sits a touch below the
+original 1052, inside the cold-S3 run-to-run spread.
+
 ---
 
 ## Story 3 — the cache (decode-once across epochs)
@@ -222,6 +251,36 @@ because WB2's fat time-chunks punish the per-sample decode while insitu reads ea
 
 ---
 
+## Store backends — obstore vs fsspec/gcsfs on HTTP GCS
+
+The engine reads a zarr `Store`, so the backend is swappable (`--backend obstore|fsspec`).
+On **plain HTTP GCS** (n2-standard-8), obstore wins the transfer floor decisively.
+Decode-free raw concurrent GET, best of a 4→32 concurrency sweep, MB/s:
+
+| chunking | obstore | gcsfs (`cat_file`) | obstore lead |
+|---|--:|--:|--:|
+| c1 | 1211 | 529 | 2.3× |
+| c4 | 1581 | 606 | 2.6× |
+| c16 | 802 | 487 | 1.6× |
+
+The mechanism is visible in the sweep: obstore **scales** with concurrency (c1 raw:
+376 → 681 → 1042 → 1211), while gcsfs **plateaus at ~500–600 MB/s and degrades past 16
+threads** (c1: 343 → 513 → 529 → 358) — the per-request Python/aiohttp path on the single
+fsspec loop is the ceiling.
+
+End-to-end (fetch + decode) the gap **compresses to ~1.15–1.2×**, but only because this
+8-core box is **decode-bound at ~450 MB/s** (the decode-threads sweep saturates 445 → 456
+at 4–8 threads, well under obstore's raw 0.8–1.6 GB/s), so both backends hit the same
+decode wall. Read that ~20% as a *floor* on fsspec's HTTP penalty: on more cores or with a
+faster codec pipeline the fetch gap re-widens toward the ~2× raw number.
+
+**Takeaway:** obstore stays the HTTP default; fsspec is *not* a co-equal fast path on HTTP,
+but it is not a correctness regression either. Its case rests entirely on the
+**Rapid/zonal gRPC** path obstore cannot reach — that experiment has not run yet, so read
+this section as "obstore wins HTTP", not "fsspec is worse".
+
+---
+
 ## Free-threading readiness
 
 insitubatch's throughput is **GIL-independent by design**: the heavy work already runs

docs/examples.md

@@ -0,0 +1,126 @@
+# Examples
+
+The sample axis is a **role, not a fixed dimension** — so the same engine trains on weather
+over time, segments microscopy volumes over `Z`, and streams telescope frames out of an
+archival format that was never zarr. Each example below is a *different geometry* on a real
+public store, with **no reshard** anywhere.
+
+All of them are runnable from a checkout (they are not shipped in the wheel), and every one
+has an offline synthetic `--source` so you can run it with no network or cloud credentials.
+
+| Example | Domain & store | What is different about the data | What it proves |
+|---|---|---|---|
+| [`advection/`](#advection-a-24-hour-forecast-in-three-frameworks) | Weather — WeatherBench2 ERA5 (`gs://`, anonymous), Arraylake/Icechunk, or synthetic | Input at *t*, target at *t+24 h* as **offset views of one array** | Windowed multi-offset sampling; one dataset → **torch, JAX and TF** |
+| [`microscopy/`](#microscopy-cell-segmentation-over-z) | Bio-imaging — IDR OME-NGFF `(T,C,Z,Y,X)` on `s3://idr` | Samples a **middle** axis (`sample_axis=2`); two co-registered variables chunked **1 vs 30 planes** deep | Arbitrary sample axis + per-variable chunk size — the engine is not weather-specific |
+| [`hubble/`](#hubble-denoising-real-telescope-frames-from-fits) | Astronomy — Hubble WFC3/IR frames of M16 on MAST's `s3://stpubdata` | **FITS, not zarr** — indexed as virtual byte-range references; one frame *is* one chunk | Training in place over an archival format; streaming value without decode amortization |
+| [WB2 pair](#the-weatherbench2-cold-start-pair) | Weather — WeatherBench2 ERA5 | The same task on **two engines** | The cold-start / memory trade-off vs an xbatcher worker stack |
+| [`transforms.py`](#transforms-and-normalization) · [`fit_scaler.py`](#transforms-and-normalization) | Any — tiny offline store | — | Why there are *two* transform stages, and how to fit a scaler over the loader |
+
+Full flags, data-source tables and design notes for each live in
+[`examples/README.md`](https://github.com/emfdavid/insitubatch/blob/main/examples/README.md).
+
+## advection — a 24-hour forecast in three frameworks
+
+One [`InSituDataset`](https://github.com/emfdavid/insitubatch/blob/main/examples/advection/data.py)
+reads three fields at time *t* (temperature `t2m` and the 10 m wind `u10`, `v10`) and the
+target `t2m` 24 hours later via `g.shift(horizon)`. Input and target are **offset views of
+the same in-place array** — the windowing unlock — and nothing is resharded. The resulting
+numpy `Batch` then trains the **same tiny CNN** in three frameworks through the DLPack
+adapters; the three files differ only in framework calls.
+
+```bash
+uv sync --extra bench --extra torch               # PyTorch   (torch.nn)
+uv run python -m examples.advection.train_torch
+
+uv sync --extra bench --extra jax                 # JAX       (flax + optax)
+uv run python -m examples.advection.train_jax
+
+uv sync --extra bench --extra tf                  # TensorFlow (Keras)
+uv run python -m examples.advection.train_tf
+```
+
+Each run prints 24-hour forecast skill on held-out data — a model that *reads the wind* to
+predict advection, versus the persistence baseline. `--source wb2` runs the same code
+against real ERA5 in the cloud; there the claim is "same pipeline, real data", **not** SOTA
+skill (24 h temperature persistence is a strong baseline).
+
+!!! warning "Install one framework at a time"
+    Having torch, JAX and TensorFlow present in the same uv venv can segfault. Sync the
+    extra for the one you are running.
+
+## microscopy — cell segmentation over Z
+
+The cross-domain showcase: same engine, different geometry. Where advection samples the
+*outer* time axis, [`microscopy/`](https://github.com/emfdavid/insitubatch/blob/main/examples/microscopy/data.py)
+samples a *middle* axis — one Z-plane of an OME-NGFF `(T,C,Z,Y,X)` confocal stack — and
+gathers two co-registered variables per anchor: a 2-channel `raw` image chunked **one plane
+deep** on Z, and its `mask` label chunked **30 planes deep** and tiled in Y/X. Different
+physical chunking, different channel counts, one sample grid, no reshard.
+
+```bash
+uv sync --extra torch
+uv run python -m examples.microscopy.train_torch                 # synthetic cells (offline)
+uv run python -m examples.microscopy.train_torch --source idr    # the real IDR image (streamed)
+```
+
+The task is per-plane foreground segmentation and the baseline is a global **Otsu**
+threshold — the segmentation analogue of persistence. Otsu reads each pixel's intensity
+alone, so a smooth autofluorescence haze gradient defeats it; a tiny CNN that reads the
+neighbourhood beats it. Each run prints held-out foreground IoU, model vs Otsu.
+
+## hubble — denoising real telescope frames from FITS
+
+The **archival-format** showcase: this data never was zarr.
+[`hubble/`](https://github.com/emfdavid/insitubatch/blob/main/examples/hubble/data.py)
+indexes real Hubble WFC3/IR frames of M16 (the Eagle Nebula) on MAST's public AWS bucket as
+**virtual references** — [VirtualiZarr](https://github.com/zarr-developers/VirtualiZarr)
+parses each `_flt.fits` header and commits byte-range references to a local Icechunk repo.
+No pixels are copied: the store is a few kB pointing at the original FITS objects, and
+insitubatch streams frames straight from S3 with `sample_axis=0`. The indexing libraries are
+build-time only — the training hot path is `icechunk` + numpy.
+
+```bash
+uv sync --extra torch
+uv run python -m examples.hubble.train_torch      # offline synthetic frames (default)
+
+# real Hubble frames on S3 — build the virtual-reference store, then stream and train:
+uv run --with virtualizarr --with kerchunk --with astropy --with icechunk --with s3fs \
+  python -m examples.hubble.train_torch --source hubble --build
+```
+
+Because a FITS image is one chunk, this demonstrates the **streaming-in-place** value (no
+reshard over a giant archive) rather than the many-samples-per-chunk decode amortization the
+chunked-zarr examples show — the honest boundary of the thesis, kept visible on purpose.
+
+## The WeatherBench2 cold-start pair
+
+The same task two ways, so you can see the trade-off and pick per workload:
+[`wb2_dataloader.py`](https://github.com/emfdavid/insitubatch/blob/main/examples/wb2_dataloader.py)
+is the insitu single-event-loop loader (with `--backend fsspec` for the gcsfs A/B), and
+[`wb2_xbatcher.py`](https://github.com/emfdavid/insitubatch/blob/main/examples/wb2_xbatcher.py)
+is the xbatcher + torch `DataLoader` worker stack, following Earthmover's `dataloader-demo`,
+focused on cold-start latency and how `forkserver-preload` cuts it.
+
+```bash
+uv run python -m examples.wb2_dataloader          # tiny synthetic data, no network
+```
+
+The [WeatherBench2 walkthrough](walkthrough.md) narrates this pair end to end, and
+[Benchmarks](benchmarks.md) has the measured numbers.
+
+## Transforms and normalization
+
+[`transforms.py`](https://github.com/emfdavid/insitubatch/blob/main/examples/transforms.py)
+puts the two user transform stages side by side on a tiny offline store — a Kelvin→Celsius
+`chunk_transform` (per chunk, one variable, cached) and a cross-variable windspeed
+`batch_transform` (needs the assembled batch, uncached). It is the clearest illustration of
+*why there are two*; see [Transforms](architecture.md#transforms-three-stages-placed-by-cost)
+for the placement model.
+
+[`fit_scaler.py`](https://github.com/emfdavid/insitubatch/blob/main/examples/fit_scaler.py)
+fits a `StandardScaler` over the loader with sklearn `partial_fit` — the recommended pattern
+(it warms the cache) versus caching scaled chunks.
+
+```bash
+uv run python -m examples.transforms              # no network
+```

docs/index.md

@@ -9,10 +9,10 @@ data source built to **keep the GPU fed** — **with no reshard** — and a Pyth
 hot path that scales with **chunks, not samples**.
 
 It is **domain-general**: the sample axis is a *role*, not a fixed dimension. The same engine
-trains on ERA5/weather over time, segments **OME-NGFF microscopy** volumes over `Z`
-([runnable example](https://github.com/emfdavid/insitubatch/blob/main/examples/microscopy) —
-raw image + label mask co-batched with no reshard), and maps cleanly onto **radio-astronomy**
-visibilities. See the [use-case tables](architecture.md#use-case-support).
+forecasts ERA5 weather over time, segments **OME-NGFF microscopy** volumes over `Z`, and
+denoises **Hubble telescope frames** streamed straight out of FITS — each a different
+geometry on a real public store, none of them resharded. See [Examples](examples.md) and the
+[use-case tables](architecture.md#use-case-support).
 
 !!! quote
     The IO race is over (obstore/icechunk saturate the NIC). The *loader* race is
@@ -28,6 +28,19 @@ where per-sample workers re-read it (the win grows with samples-per-chunk). It i
 universal speed win: at the one-sample-per-chunk (GRIB) end, or against an unbounded gather on
 large fields, a tuned pool can edge ahead per byte. Numbers: [Benchmarks](benchmarks.md).
 
+## Cross-domain examples
+
+Four runnable showcases, each a *different* geometry on a real public store — and each with
+an offline synthetic mode so it runs with no network or credentials. Details and commands:
+[Examples](examples.md).
+
+| | Domain & store | The geometry it exercises |
+|---|---|---|
+| **advection** | WeatherBench2 ERA5 (`gs://`, anonymous) | Input at *t*, target at *t+24 h* as offset views of one array — then the *same* CNN trained in **torch, JAX and TF** |
+| **microscopy** | IDR OME-NGFF `(T,C,Z,Y,X)` on `s3://idr` | Samples a **middle** axis (`Z`); raw image and label mask chunked 1 vs 30 planes deep, co-batched |
+| **hubble** | Hubble WFC3/IR frames on MAST's `s3://stpubdata` | **FITS, not zarr** — virtual byte-range references, streamed and trained in place |
+| **wb2 pair** | WeatherBench2 ERA5 | The same task on insitu and on an xbatcher worker stack, for the cold-start trade-off |
+
 ## The problem, and the inversion
 
 The classic PyTorch `DataLoader` puts parallelism in worker **processes**, each running a
@@ -87,9 +100,8 @@ jbatch = to_jax(next(iter(ds.train)))                                    # JAX:
 tfds = as_tf_dataset(ds.val)                                             # TF:    tf.data.Dataset
 ```
 
-See [`examples/advection`](https://github.com/emfdavid/insitubatch/blob/main/examples/advection) for
-working CNN forecast models using insituBatch implemented with Torch, Jax and Tensorflow with
-real ERA5 data.
+See [Examples](examples.md) for working CNN models built on `InSituDataset` — a three-framework
+ERA5 forecast, OME-NGFF microscopy segmentation over `Z`, and Hubble frame denoising from FITS.
 
 A runnable, network-free version of this — paralleling the Earthmover
 `dataloader-demo`, with a spatial subregion pulled out by a `batch_transform` —
@@ -119,9 +131,11 @@ uv sync --extra gpu      # CUDA box only: cupy + kvikio zero-copy path
 obstore reads, the decoupled fetch **`Scheduler`** + **`ChunkPool`** (assembly buffer
 *and* cache — byte budget + pin/LRU, heap or mmap-on-NVMe, with **cross-run
 persistence** via `persist=True`), approximate (shuffle-block) shuffle, chunk/batch
-**transforms** (incl. a fitted `StandardScaler`), **prefetch**, and the **torch / JAX /
-TF** surfaces. Not yet built: `Regrid` + the **GPU/device** transform stage, and
-multi-timestep windows that cross chunk boundaries.
+**transforms** (incl. a fitted `StandardScaler`), **prefetch**, **windowed multi-offset
+sampling**, an **arbitrary sample axis** with per-variable chunk sizes, and the **torch /
+JAX / TF** surfaces. Not yet built: a production `Regrid` + the **GPU/device** transform
+stage. (A window that crosses chunk boundaries as a single slab read is an explicit
+non-goal, not a gap — discrete offsets from an anchor already express it.)
 
 [DESIGN.md](https://github.com/emfdavid/insitubatch/blob/main/DESIGN.md) is the single
 source of truth for status, the roadmap, and the scope limits.

mkdocs.yml

@@ -30,6 +30,7 @@ theme:
 
 nav:
   - Home: index.md
+  - Examples: examples.md
   - Architecture: architecture.md
   - Tuning: tuning.md
   - Benchmarks: benchmarks.md