docs: annbatch contrast + read-once/sample-once shuffle contract#19
Open
emfdavid wants to merge 5 commits into
Open
docs: annbatch contrast + read-once/sample-once shuffle contract#19emfdavid wants to merge 5 commits into
emfdavid wants to merge 5 commits into
Conversation
…riants Add annbatch (scverse) to the "what it is, by contrast" table as the sharpest mirror: same premise, opposite bet (reshard-for-randomness vs train-in-place). Add a mermaid figure staged against their Fig 1C showing the bounded block-shuffle — chunks along the sample axis, permutation in the read plan (metadata) kept distinct from byte movement into the pool. Document the two orthogonal invariants (read-once = plan + ChunkPool; sample-once = the `order` ledger; fancy index recomputed in gather, never stored), and flag per-block ragged batches as a possible wart under evaluation (no drop_last; short final batch per block, not per epoch). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Flip the figure to flowchart TB (stages ①→②→③→④ top-down) with each subgraph laid out LR internally, so the stages render wide instead of squished into a horizontal strip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…Fig 1C) Stage ④ now renders each batch as a row of colour-coded slice nodes, one per source chunk (same colours as the resident pool in ③) — mirroring annbatch Fig 1C's striped mini-batches. The contrast: our chunk boxes stay whole in ③ and feed many batches, rather than ceasing to exist. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Enforce the doc division DESIGN.md's own header already declares (evolution ledger vs live contract in architecture.md): - architecture.md gains "Read-once and sample-once": the two invariants, the archive→batch figure (moved here, reframed neutral with one contrast line), and the ragged-per-block tail as a user caveat (step-count, BatchNorm). - DESIGN.md's "Two invariants" subsection and the shuffle figure/mechanism collapse to short pointers; the Shuffle section keeps only the thesis (why approximate) + the annbatch contrast. Ragged-batch Known-limitations entry trimmed to the open decision, linking to architecture for behaviour. - Fix a stale "gather map" phrase that implied a stored index. Net: less sprawl, one home per topic, cross-linked both ways. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…t vectorized - architecture.md: pipeline Properties gains a line that batch buffers are not pooled (DLPack aliasing) and host memory is not pinned yet, with the unified pinned-ring-buffer fix; the whole-field-crop scope note flags that the shipped random-crop example uses a per-sample Python loop (illustrative, not vectorized). - DESIGN.md: matching Known-limitations entry for the fresh/unpinned batch buffers — one pinned ring buffer (depth ~prefetch_depth) fixes reuse + non_blocking H2D together; profile the H2D ceiling first (payload-dependent, ties to M2 GPU stage). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
emfdavid
added a commit
that referenced
this pull request
Jul 23, 2026
Fold last night's Ilan Gold / annbatch work (PR #19) into the Pangeo Showcase draft. Adds the annbatch "sharp mirror" framing (same premise, opposite bet on where batch randomness comes from) and the honest block-local shuffle compromise as backing material, then three new abstract cuts: a maximal draft, a punchy ~200w cut, and a lead-with-the-thesis cut that opens on the differentiators (O(chunks) hot path, flat memory, in-place remote) before deferring to xbatcher and annbatch. Links out to the hosted architecture/examples/benchmarks pages and caveats xbatcher's GRIB-end sweet spot. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Documents where insitubatch sits relative to annbatch (scverse, Ilan Gold et al., arXiv:2604.01949), and pins down the shuffle/gather contract that comparison turns on.
DESIGN.md — the evolution ledger (why / alternatives)
m≈ ourblock_chunks); the difference is the currency — a one-time rewrite vs. read locality.docs/architecture.md — the live contract (how it works)
ChunkPool; sample-once = theorderledger, fancy index recomputed at gather, never stored).Σ ⌈block/bs⌉, not⌈N/bs⌉); nodrop_lasttoday.Why
DESIGN.md's own header already declares the division (evolution ledger vs. live contract in architecture.md). This enforces it: contract + diagram live in architecture, why/alternatives in DESIGN, cross-linked both ways. Net −9 lines — less sprawl, one home per topic.
Notes for review
#known-limitations--defects/#read-once-and-sample-oncecross-link anchors.test_order_covers_every_sample_exactly_once,test_order_handles_partial_final_chunk, the decode-once suite).🤖 Generated with Claude Code