Skip to content

docs: restructure DESIGN.md + surface the cross-domain examples#18

Merged
emfdavid merged 2 commits into
mainfrom
docs/design-restructure
Jul 20, 2026
Merged

docs: restructure DESIGN.md + surface the cross-domain examples#18
emfdavid merged 2 commits into
mainfrom
docs/design-restructure

Conversation

@emfdavid

@emfdavid emfdavid commented Jul 20, 2026

Copy link
Copy Markdown
Owner

Docs-only. Cuts DESIGN.md from 839 lines of sprawl to a structure where each kind of open work has exactly one home, and moves measurements/diagrams to the pages that own them.

What moved

  • M1.6 V2 scheduler: 175 → 84 lines. Kept the design; deleted the Phasing / B1 task list blocks (finished sprint planning — git history has them); moved the exp_c acceptance tables to docs/benchmarks.md.
  • Roadmap: 333 → ~130 lines. Every milestone is intent + state + a pointer. M-W's design rationale moved up into the geometry ladder where readers look for it; its Phase-4 example plan is dropped (examples/advection/ is the record).
  • M-GCS obstore-vs-gcsfs numbers → docs/benchmarks.md; event-loop-ownership diagram → docs/architecture.md (it's architecture, not roadmap).

What's new

  • Known limitations & defects — the section that didn't exist, which is why several items only lived in session notes: window_factor sizes residency by span rather than the offset set; the silent edge-anchor drop; whole-pipeline (not per-variable) cache invalidation; windowed+shuffle residency; the accepted ~3.5% refcount cost; and the inherited zarr-python sharded-read cliff.
  • Upstream capabilities we plan to inherit — the design dependency on zarr-python lazy indexing (#3906) and batched zero-copy out= decode (#3060/#2904), including what stays ours either way. Deliberately records the dependency, not PR status — that state goes stale in-repo.
  • zarrs ZarrsCodecPipeline spike. Attributes the cost to zarr-python's pipeline rather than numcodecs (the uncompressed row isolates it — ~1.9× with no codec invoked), names the allocate-then-copy interface as the real target, ties it to #3060 as the alternative attack on the same copy, and notes our upside is well under the 3.5× headline since we already call the pipeline directly. Gated on it serving our direct-bytes path and pinning its Rust thread pool to 1.

Accuracy fixes found in review

Verification

uv run mkdocs build --strict passes. Note DESIGN.md is linked from the nav as an external GitHub URL, so the strict build does not validate it — the rendered diff here is the check.


Second commit — surface the cross-domain examples on the site

The site had one reference to the WeatherBench2 walkthrough and a single inline mention of microscopy; hubble/ and the three-framework advection story were invisible to anyone who never opened the repo.

  • New docs/examples.md, in the nav directly under Home — a table of the four showcases (domain, store, the geometry each exercises, what it proves), run commands, and the offline synthetic mode for each. Full flags stay in examples/README.md, which it links rather than duplicates.
  • Home page gains a Cross-domain examples table in headline position, and its domain-general paragraph now names weather, microscopy and astronomy instead of only microscopy.
  • Replaced the two scattered one-off example links with pointers to the new page.
  • Corrected the home-page status line: windowed multi-offset sampling and the arbitrary sample axis shipped, and a cross-chunk slab read is an explicit non-goal rather than an unbuilt feature.

examples/sdss is not on main yet, so it is not listed — it can land with its own PR.

mkdocs build --strict passes and every internal anchor was verified against the built HTML.

emfdavid and others added 2 commits July 20, 2026 03:27
DESIGN.md had grown to 839 lines with 40% of it roadmap prose, a
175-line finished sprint plan wedged into the middle of the
architecture narrative, and "what isn't built" spread across three
sections. Open work that only existed in session notes (the
window_factor residency bug, the silent edge-anchor drop) had no home
in the doc at all.

Restructured with a single home per concern:

- Fold the M1.6 V2 section from 175 -> 84 lines: keep the design,
  delete the Phasing/B1 task lists (git history has them), move the
  exp_c acceptance tables to docs/benchmarks.md.
- Collapse the roadmap from 333 -> ~130 lines; every milestone is now
  intent + state + a pointer. M-W's design rationale moved up into the
  geometry ladder where readers look for it; its Phase-4 example plan
  is dropped (examples/advection/ is the record).
- Move the M-GCS obstore-vs-gcsfs numbers to docs/benchmarks.md and the
  event-loop-ownership diagram to docs/architecture.md.
- New "Known limitations & defects": window_factor sizes residency by
  span not offset set; edge-anchor drop is silent; cache invalidation is
  whole-pipeline not per-variable; windowed+shuffle residency; the
  accepted ~3.5% refcount cost; the inherited zarr-python sharded-read
  cliff.
- New "Upstream capabilities we plan to inherit": the design dependency
  on zarr-python lazy indexing (#3906) and batched zero-copy out=
  decode (#3060/#2904), with what stays ours either way. Records the
  dependency, not PR status.
- New spike: zarrs ZarrsCodecPipeline as a decode accelerator. Attributes
  the cost to zarr-python's pipeline rather than numcodecs (the
  uncompressed row isolates it: 1.9x with no codec invoked), names the
  allocate-then-copy interface as the real target, ties it to #3060, and
  notes our upside is well under the 3.5x headline since we already call
  the pipeline directly. Gated on it serving our direct-bytes path and
  pinning its Rust thread pool to 1.
- Trim Status to maturity + a pointer; numbers live on the benchmarks
  page only.

Two accuracy fixes found in review: JAX/TF surfaces shipped in M3, not
"planned"; and the Earth2Studio obstore store-swap is an unmeasured
hypothesis with a mechanism, not a result -- neither we nor the upstream
PRs have published a cold-cache A/B.

Docs only; mkdocs build --strict passes locally.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The site had one reference to the WeatherBench2 walkthrough and a
single inline mention of microscopy; hubble and the three-framework
advection story were invisible to anyone who never opened the repo.

- New docs/examples.md, added to the nav directly under Home: a table
  of the four showcases (domain, store, the geometry each exercises,
  what it proves) plus run commands and the offline synthetic mode for
  each. Full flags stay in examples/README.md, which it links.
- Home page gains a "Cross-domain examples" table in headline position
  and its domain-general paragraph now names weather, microscopy and
  astronomy instead of just microscopy.
- Replace the two scattered one-off example links with pointers to the
  new page.

Also corrects the home-page status line: windowed multi-offset sampling
and the arbitrary sample axis shipped, and a cross-chunk slab read is an
explicit non-goal rather than an unbuilt feature.

examples/sdss is not on main yet, so it is not listed here.

mkdocs build --strict passes; internal anchors verified against the
built HTML.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@emfdavid emfdavid changed the title docs: restructure DESIGN.md — one home per kind of open work docs: restructure DESIGN.md + surface the cross-domain examples Jul 20, 2026
@emfdavid
emfdavid merged commit 65e0470 into main Jul 20, 2026
8 checks passed
@emfdavid
emfdavid deleted the docs/design-restructure branch July 20, 2026 03:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant