docs: restructure DESIGN.md + surface the cross-domain examples#18
Merged
Conversation
DESIGN.md had grown to 839 lines with 40% of it roadmap prose, a 175-line finished sprint plan wedged into the middle of the architecture narrative, and "what isn't built" spread across three sections. Open work that only existed in session notes (the window_factor residency bug, the silent edge-anchor drop) had no home in the doc at all. Restructured with a single home per concern: - Fold the M1.6 V2 section from 175 -> 84 lines: keep the design, delete the Phasing/B1 task lists (git history has them), move the exp_c acceptance tables to docs/benchmarks.md. - Collapse the roadmap from 333 -> ~130 lines; every milestone is now intent + state + a pointer. M-W's design rationale moved up into the geometry ladder where readers look for it; its Phase-4 example plan is dropped (examples/advection/ is the record). - Move the M-GCS obstore-vs-gcsfs numbers to docs/benchmarks.md and the event-loop-ownership diagram to docs/architecture.md. - New "Known limitations & defects": window_factor sizes residency by span not offset set; edge-anchor drop is silent; cache invalidation is whole-pipeline not per-variable; windowed+shuffle residency; the accepted ~3.5% refcount cost; the inherited zarr-python sharded-read cliff. - New "Upstream capabilities we plan to inherit": the design dependency on zarr-python lazy indexing (#3906) and batched zero-copy out= decode (#3060/#2904), with what stays ours either way. Records the dependency, not PR status. - New spike: zarrs ZarrsCodecPipeline as a decode accelerator. Attributes the cost to zarr-python's pipeline rather than numcodecs (the uncompressed row isolates it: 1.9x with no codec invoked), names the allocate-then-copy interface as the real target, ties it to #3060, and notes our upside is well under the 3.5x headline since we already call the pipeline directly. Gated on it serving our direct-bytes path and pinning its Rust thread pool to 1. - Trim Status to maturity + a pointer; numbers live on the benchmarks page only. Two accuracy fixes found in review: JAX/TF surfaces shipped in M3, not "planned"; and the Earth2Studio obstore store-swap is an unmeasured hypothesis with a mechanism, not a result -- neither we nor the upstream PRs have published a cold-cache A/B. Docs only; mkdocs build --strict passes locally. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The site had one reference to the WeatherBench2 walkthrough and a single inline mention of microscopy; hubble and the three-framework advection story were invisible to anyone who never opened the repo. - New docs/examples.md, added to the nav directly under Home: a table of the four showcases (domain, store, the geometry each exercises, what it proves) plus run commands and the offline synthetic mode for each. Full flags stay in examples/README.md, which it links. - Home page gains a "Cross-domain examples" table in headline position and its domain-general paragraph now names weather, microscopy and astronomy instead of just microscopy. - Replace the two scattered one-off example links with pointers to the new page. Also corrects the home-page status line: windowed multi-offset sampling and the arbitrary sample axis shipped, and a cross-chunk slab read is an explicit non-goal rather than an unbuilt feature. examples/sdss is not on main yet, so it is not listed here. mkdocs build --strict passes; internal anchors verified against the built HTML. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Docs-only. Cuts DESIGN.md from 839 lines of sprawl to a structure where each kind of open work has exactly one home, and moves measurements/diagrams to the pages that own them.
What moved
Phasing/B1 task listblocks (finished sprint planning — git history has them); moved the exp_c acceptance tables todocs/benchmarks.md.examples/advection/is the record).docs/benchmarks.md; event-loop-ownership diagram →docs/architecture.md(it's architecture, not roadmap).What's new
Known limitations & defects— the section that didn't exist, which is why several items only lived in session notes:window_factorsizes residency by span rather than the offset set; the silent edge-anchor drop; whole-pipeline (not per-variable) cache invalidation; windowed+shuffle residency; the accepted ~3.5% refcount cost; and the inherited zarr-python sharded-read cliff.Upstream capabilities we plan to inherit— the design dependency on zarr-python lazy indexing (#3906) and batched zero-copyout=decode (#3060/#2904), including what stays ours either way. Deliberately records the dependency, not PR status — that state goes stale in-repo.ZarrsCodecPipelinespike. Attributes the cost to zarr-python's pipeline rather than numcodecs (the uncompressed row isolates it — ~1.9× with no codec invoked), names the allocate-then-copy interface as the real target, ties it to #3060 as the alternative attack on the same copy, and notes our upside is well under the 3.5× headline since we already call the pipeline directly. Gated on it serving our direct-bytes path and pinning its Rust thread pool to 1.Accuracy fixes found in review
Verification
uv run mkdocs build --strictpasses. NoteDESIGN.mdis linked from the nav as an external GitHub URL, so the strict build does not validate it — the rendered diff here is the check.Second commit — surface the cross-domain examples on the site
The site had one reference to the WeatherBench2 walkthrough and a single inline mention of microscopy;
hubble/and the three-framework advection story were invisible to anyone who never opened the repo.docs/examples.md, in the nav directly under Home — a table of the four showcases (domain, store, the geometry each exercises, what it proves), run commands, and the offline synthetic mode for each. Full flags stay inexamples/README.md, which it links rather than duplicates.Cross-domain examplestable in headline position, and its domain-general paragraph now names weather, microscopy and astronomy instead of only microscopy.examples/sdssis not on main yet, so it is not listed — it can land with its own PR.mkdocs build --strictpasses and every internal anchor was verified against the built HTML.