[WIP] Extract ZarrCore as a separate library

.github/workflows/CI.yml

@@ -31,6 +31,9 @@ jobs:
           arch: ${{ matrix.arch }}
           show-versioninfo: true
       - uses: julia-actions/cache@v2
+      - name: Dev ZarrCore local package (Julia < 1.11 lacks [sources] support)
+        if: matrix.version == 'lts'
+        run: julia --project=. -e 'using Pkg; Pkg.develop(path="lib/ZarrCore")'
       - uses: julia-actions/julia-buildpkg@v1
         env:
           PYTHON:

CLAUDE.md

@@ -4,97 +4,113 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 
 ## Project Overview
 
-Zarr.jl is a Julia implementation of the Zarr specification for chunked, compressed, N-dimensional arrays. It supports Zarr v2 (stable) and v3 (experimental, in development on `as/continue_v3` branch). The package provides multiple storage backends (filesystem, in-memory, S3, GCS, HTTP, ZIP) and compressors (Blosc, zlib, Zstandard).
+Zarr.jl is a Julia implementation of the Zarr specification for chunked, compressed, N-dimensional arrays. It supports Zarr v2 (stable) and v3 (experimental). The package provides multiple storage backends (filesystem, in-memory, S3, GCS, HTTP, ZIP) and compressors (Blosc, zlib, Zstandard).
 
 ## Build & Test Commands
 
 ```bash
 # Run all tests
 julia --project -e 'using Pkg; Pkg.test()'
 
-# Run a single test file interactively (use the test/ project environment)
+# Run a single test file (use the test/ project environment)
 julia --project=test -e 'using Test, Zarr, JSON; include("test/v3_codecs.jl")'
 
-# Instantiate test dependencies (after Julia version change or first time setup)
+# Instantiate test dependencies (first time or after Julia version change)
 julia --project=test -e 'using Pkg; Pkg.instantiate()'
+
+# Run ZarrCore tests only
+julia --project=lib/ZarrCore -e 'using Pkg; Pkg.test()'
 ```
 
 Julia version requirement: 1.10+. CI tests against Julia LTS, stable (`1`), nightly, and pre-release on Ubuntu, macOS, and Windows.
 
+### Test Structure
+
+- `test/runtests.jl` — Main test suite: ZArray, groups, metadata, indexing, resize, concat, strings, ragged arrays, fill values. Includes the other test files.
+- `test/storage.jl` — Storage backend tests
+- `test/Filters.jl` — Filter tests
+- `test/python.jl` — Interoperability tests with Python-generated Zarr files
+- `test/v3_codecs.jl` — V3 codec tests (BytesCodec, TransposeCodec, etc.)
+- `test/v3_julia.jl` — Julia-generated V3 test fixtures
+- `test/v3_python.jl` — Python-generated V3 fixtures (requires PythonCall + CondaPkg; CI generates these before running tests)
+
 ## Architecture
 
+### Package Structure
+- `lib/ZarrCore/` — Minimal core package with types, registries, NoCompressor, BytesCodec, TransposeCodec, DirectoryStore, DictStore, ConsolidatedStore, all pure-Julia filters
+- `src/` (Zarr.jl) — Wrapper that depends on ZarrCore, adds Blosc/Zlib/Zstd compressors, V3 compression codecs, HTTP/GCS/S3/ZIP stores
+- ZarrCore's `default_compressor()` returns `NoCompressor()`; Zarr.jl overrides it to `BloscCompressor()`
+- V3 codecs use `v3_codec_parsers` registry for extensible parsing; Zarr.jl registers compression codec parsers
+
 ### Core Type Hierarchy
 
 ```
-AbstractMetadata{T,N}
-├── MetadataV2{T,N,C,F}   # .zarray + .zattrs (v2)
-└── MetadataV3{T,N,P}     # zarr.json (v3), P<:AbstractCodecPipeline
+AbstractMetadata{T,N,E<:AbstractChunkKeyEncoding}
+├── MetadataV2{T,N,C,F}       # .zarray + .zattrs (v2)
+└── MetadataV3{T,N,P,E}       # zarr.json (v3), P<:AbstractCodecPipeline, E<:AbstractChunkKeyEncoding
 
 AbstractStore
-├── DirectoryStore          # Filesystem
-├── DictStore               # In-memory Dict{String,Vector{UInt8}}
-├── S3Store                 # AWS S3 (via AWSS3 extension)
-├── GCStore                 # Google Cloud Storage
-├── ConsolidatedStore       # Consolidated metadata wrapper
-├── HTTPStore               # HTTP-based read-only
-└── ZipStore                # ZIP archive read-only
+├── DirectoryStore              # Filesystem
+├── DictStore                   # In-memory Dict{String,Vector{UInt8}}
+├── S3Store                     # AWS S3 (via AWSS3 extension)
+├── GCStore                     # Google Cloud Storage
+├── ConsolidatedStore           # Consolidated metadata wrapper
+├── HTTPStore                   # HTTP-based read-only
+└── ZipStore                    # ZIP archive read-only
+
+AbstractCodecPipeline
+├── V2Pipeline{C,F}            # compressor + filters
+└── V3Pipeline{AA,AB,BB}       # three-phase codec chain
 
 ZArray{T,N,S<:AbstractStore,M<:AbstractMetadata} <: AbstractDiskArray{T,N}
 ZGroup{S<:AbstractStore}
 ```
 
 ### Module/File Layout
 
-- `src/Zarr.jl` — Module entry point, defines `ZarrFormat{V}` (Val-parameterized version tag, default `DV = ZarrFormat(Val(2))`)
-- `src/metadata.jl` — `MetadataV2` struct, type string encoding (`typestr`), fill value encoding/decoding, `Metadata()` constructors for V2; dispatches V3 to `metadata3.jl`
-- `src/metadata3.jl` — All V3-specific code: `MetadataV3` struct and constructors, `Metadata3(dict)` parsing, `lower3` serialization, codec pipeline parsing, `get_order`, `JSON.lower(::MetadataV3)`
-- `src/chunkencoding.jl` — `ChunkEncoding` struct (separator char + prefix bool), `citostring()` for chunk path generation. V2 default: `'.'` separator, no prefix. V3 default: `'/'` separator, `"c/"` prefix
+- `src/Zarr.jl` — Module entry point. Defines `ZarrFormat{V}` (Val-parameterized version tag) and default `DV = ZarrFormat(Val(2))`. Default chunk separator constants: `DS = '.'` (v2), `DS2 = '.'`, `DS3 = '/'`.
+- `src/metadata.jl` — `MetadataV2` struct, type string encoding (`typestr`), fill value encoding/decoding, `Metadata()` constructors; dispatches V3 to `metadata3.jl`
+- `src/metadata3.jl` — `MetadataV3{T,N,P,E}` struct and constructors, `Metadata3(dict)` parsing, `lower3` serialization, `get_order`, `JSON.lower(::MetadataV3)`. V3 type mappings use lowercase strings (e.g. `"float32"`, `"complex128"`)
+- `src/chunkkeyencoding.jl` — `ChunkKeyEncoding` struct (separator char + prefix bool) and `SuffixChunkKeyEncoding{E}`, `citostring()` for chunk path generation, registry via `chunk_key_encoding_parsers` dict. V2 default: `'.'` separator, no prefix. V3 default: `'/'` separator, `"c/"` prefix
+- `src/pipeline.jl` — `V2Pipeline` (wraps compressor + filters) and `V3Pipeline` (three-phase: array→array, array→bytes, bytes→bytes). `pipeline_encode`/`pipeline_decode!` interface
 - `src/ZArray.jl` — Core array type, `readblock!`/`writeblock!` (DiskArrays interface), `zcreate`, `zzeros`, `zopen`, resize/append
 - `src/ZGroup.jl` — Hierarchical group support, `zopen`, `zgroup`, auto-detection of zarr version via `ZarrFormat(store, path)`
 - `src/Compressors/` — `Compressor` abstract type, `compressortypes` registry (keyed by spec name string), implementations: `blosc.jl`, `zlib.jl`, `zstd.jl`, `v3.jl` (v3 wrapper `Compressor_v3{C}`)
-- `src/Codecs/` — V3 codec system (`Codec` abstract type), `V3/V3.jl` defines `V3Codec{In,Out}` with `BloscV3Codec`, `BytesCodec`, `CRC32cV3Codec`, `GzipV3Codec`, `ShardingCodec`, `TransposeCodec`, `ZstdV3Codec`
+- `src/Codecs/` — V3 codec system (`Codec` abstract type), `V3/V3.jl` defines `V3Codec{In,Out}` with `BloscV3Codec`, `BytesCodec`, `CRC32cV3Codec`, `GzipV3Codec`, `ShardingCodec`, `TransposeCodec`, `ZstdV3Codec`. Registry: `codectypes` dict
 - `src/Filters/` — `Filter{T,TENC}` abstract type, implementations for variable-length arrays, strings, Fletcher32, shuffle, delta, quantize
 - `src/Storage/Storage.jl` — `AbstractStore` interface, I/O strategy (`SequentialRead`/`ConcurrentRead`), chunk read/write/delete helpers, metadata read/write dispatched on `ZarrFormat{2}` vs `ZarrFormat{3}`
 
 ### Key Design Patterns
 
 - **Format version dispatch**: `ZarrFormat{V}` (where V is 2 or 3) used throughout for multiple dispatch on version-specific behavior (metadata file names, chunk encoding, serialization format)
+- **Registry pattern**: Used pervasively — `compressortypes` (compressors), `codectypes` (V3 codecs), `chunk_key_encoding_parsers` (chunk key encodings), filter dict. New implementations register via these dicts.
 - **Shape is mutable**: `metadata.shape` is `Base.RefValue{NTuple{N,Int}}` to allow `resize!` without replacing the metadata struct
 - **Column-major ↔ row-major**: Zarr stores shapes/chunks in row-major (C order); Julia uses column-major. All conversions happen via `reverse()` at metadata boundaries
-- **Compressor registry**: `compressortypes` dict maps spec names (e.g., `"blosc"`, `"zlib"`) to compressor types. V3 uses `Compressor_v3{C}` wrapper to change JSON serialization format
 - **DiskArrays integration**: `ZArray <: AbstractDiskArray`, chunk-aware I/O via `readblock!`/`writeblock!`, `eachchunk`, `haschunks`
 - **Async chunk I/O**: Channel-based parallel chunk reads/writes for stores supporting `ConcurrentRead`
 
 ### Storage Interface Requirements
 
 New store backends must implement: `getindex(store, key)::Union{Vector{UInt8}, Nothing}`, `setindex!(store, data, key)`, `storagesize(store, path)`, `subdirs(store, path)`, `subkeys(store, path)`, `isinitialized(store, key)`, `storefromstring(Type, string, create)`.
 
+### V3 Pipeline Architecture
+
+V3 encoding/decoding follows a three-phase codec chain in `V3Pipeline`:
+
+1. **array→array** codecs (e.g., `TransposeCodec`) — applied in forward order during encode
+2. **array→bytes** codec (e.g., `BytesCodec`, `ShardingCodec`) — single codec, converts array to byte stream
+3. **bytes→bytes** codecs (e.g., `GzipV3Codec`, `BloscV3Codec`, `CRC32cV3Codec`) — applied in forward order during encode
+
+Decoding reverses all three phases. The `MetadataV3` convenience constructor builds the pipeline from `order` (→ `TransposeCodec`), `endian` (→ `BytesCodec`), and `compressor` (→ bytes→bytes codecs).
+
 ### V3 Status (Experimental)
 
-V3 support is under active development. Current state:
-
-**Codecs (`src/Codecs/V3/V3.jl`)**
-- `BytesCodec` — stores `endian::Symbol` (`:little` or `:big`); encode/decode byte-swap elements when the target endian differs from the system byte order (`Base.ENDIAN_BOM`). Default is `:little`.
-- `TransposeCodec` — array→array permutation codec (renamed from `TransposeCodecImpl`)
-- `BloscV3Codec` — shuffle stored as integer (0=noshuffle, 1=shuffle, 2=bitshuffle); parsed from spec strings (`"noshuffle"`, `"shuffle"`, `"bitshuffle"`) and serialized back to strings
-- Sharding codec (`sharding_indexed`) has struct definitions and encode/decode logic but is not yet wired into the main read/write pipeline (throws `ArgumentError` when encountered)
-- `crc32c` codec has encode/decode implementations and is parseable from metadata
-
-**Metadata (`src/metadata3.jl`)**
-- `MetadataV3{T,N,P}` has no `order` field; storage order is encoded in the pipeline via `TransposeCodec`
-- Two constructors:
-  - Primary inner constructor: `MetadataV3{T,N,P}(zarr_format, node_type, shape, chunks, dtype, pipeline, fill_value, chunk_encoding)` — takes a pre-built pipeline, no `order` argument
-  - Convenience outer constructor: `MetadataV3{T,N}(...; order, endian, compressor, chunk_encoding)` — builds the pipeline from `order` (→ `TransposeCodec`), `endian` (→ `BytesCodec`), and `compressor` (→ bytes→bytes codecs)
-- `get_order(md::MetadataV3)` — derives `'C'`/`'F'` from the pipeline; throws `ArgumentError` when order is ambiguous: multiple array→array codecs, an unrecognized array→array codec, or a `TransposeCodec` with a permutation that is neither the identity (C) nor the full reversal (F)
-- `get_order(md::MetadataV2)` — returns `md.order`
-- `==` for `MetadataV3` compares `pipeline` (not `order`)
-
-**Other V3 status**
+- `MetadataV3{T,N,P,E}` has no `order` field; storage order is encoded in the pipeline via `TransposeCodec`. `get_order(md::MetadataV3)` derives `'C'`/`'F'` from the pipeline.
+- Sharding codec (`sharding_indexed`) has struct definitions and encode/decode logic but throws `ArgumentError` when encountered (not wired into main pipeline)
+- Filters are not implemented for V3 (`filters = nothing` hardcoded)
+- `zgroup()` always creates v2 groups (hardcoded `DV` format)
+- `crc32c` codec has full encode/decode implementations
 - V3 groups work via `zarr.json` with `node_type: "group"`
-- `zgroup()` currently always creates v2 groups (hardcoded `DV` format)
-- Filters are not implemented for v3 (`filters = nothing` hardcoded in `Metadata3`)
-- Test fixtures: `test/v3_julia.jl` (Julia-generated), `test/v3_python.jl` (Python-generated via PythonCall)
-- V3 codec tests are in `test/v3_codecs.jl`
 
 ### Extension System
 

Project.toml

@@ -9,7 +9,6 @@ CRC32c = "8bf52ea8-c179-5cab-976a-9e18b702a9bc"
 ChunkCodecCore = "0b6fb165-00bc-4d37-ab8b-79f91016dbe1"
 ChunkCodecLibZlib = "4c0bbee4-addc-4d73-81a0-b6caacae83c8"
 ChunkCodecLibZstd = "55437552-ac27-4d47-9aa3-63184e8fd398"
-DataStructures = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8"
 DateTimes64 = "b342263e-b350-472a-b1a9-8dfd21b51589"
 Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"
 DiskArrays = "3c3547ce-8d99-4f5e-a174-61eb10b00ae3"
@@ -19,8 +18,12 @@ OffsetArrays = "6fe1bfb0-de20-5000-8ca7-80f57d26f881"
 OpenSSL = "4d8831e6-92b7-49fb-bdf8-b643e874388c"
 Pkg = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"
 URIs = "5c2747f8-b7ea-4ff2-ba2e-563bfd36b1d4"
+ZarrCore = "d6e7b3f4-6eca-4ed7-8aa6-35dc33f59f61"
 ZipArchives = "49080126-0e18-4c2a-b176-c102e4b3760c"
 
+[sources.ZarrCore]
+path = "lib/ZarrCore"
+
 [weakdeps]
 AWSS3 = "1c724243-ef5b-51ab-93f4-b0a88ac62a95"
 
@@ -34,7 +37,6 @@ ChunkCodecCore = "1"
 ChunkCodecLibZlib = "1"
 ChunkCodecLibZstd = "1"
 CRC32c = "1.10, 1.11"
-DataStructures = "0.17, 0.18, 0.19"
 DateTimes64 = "1"
 DiskArrays = "0.4.2"
 HTTP = "^1.3"

docs/plans/2026-03-20-zarrcore-extraction-design.md

@@ -0,0 +1,133 @@
+# ZarrCore.jl Extraction Design
+
+## Goal
+
+Factor Zarr.jl into two packages:
+- **ZarrCore.jl** (`lib/ZarrCore/`) — minimal core sufficient to read/write uncompressed Zarr v2 and v3 arrays
+- **Zarr.jl** — depends on ZarrCore, adds all compressors, V3 codecs, and network/archive storage backends
+
+## Design Decisions
+
+1. **Abstract pipeline in Core, concrete codecs split across packages.** ZarrCore defines `AbstractCodecPipeline`, `V2Pipeline`, `V3Pipeline`, and only `BytesCodec` + `TransposeCodec`. Zarr.jl registers compression codecs into the existing `codectypes` registry. Unknown codecs at parse time produce a clear error.
+
+2. **All pure-Julia filters in Core.** FixedScaleOffset, Fletcher32, Shuffle, Delta, Quantize — all tiny, no external deps. Avoids complex split of the Filters directory.
+
+3. **Core storage: DirectoryStore + DictStore + ConsolidatedStore.** No external deps. ConsolidatedStore is a thin JSON wrapper commonly used for read-only access.
+
+4. **Monorepo subdirectory package** at `lib/ZarrCore/`. Own `Project.toml`, publishable to General registry. Zarr.jl uses `path = "lib/ZarrCore"` during dev.
+
+5. **Zarr.jl re-exports everything from ZarrCore.** `using Zarr` is fully backward-compatible. `using ZarrCore` gives the minimal API.
+
+6. **Convenience functions in Core.** `zcreate`, `zopen`, `zzeros`, `zgroup` live in ZarrCore — without them Core has no user-facing functionality.
+
+7. **Extensible store resolution.** ZarrCore owns `zopen` dispatch through `storageregexlist`. Core handles local paths + DictStore. Zarr.jl registers HTTP/GCS/S3/Zip resolvers.
+
+## Package Structure
+
+### lib/ZarrCore/
+
+```
+lib/ZarrCore/
+  Project.toml          # JSON, DiskArrays, OffsetArrays, DateTimes64, Dates
+  src/
+    ZarrCore.jl         # module entry, ZarrFormat, constants, includes, exports
+    chunkkeyencoding.jl
+    metadata.jl
+    metadata3.jl
+    MaxLengthStrings.jl
+    pipeline.jl
+    Compressors/
+      Compressors.jl    # Compressor, NoCompressor, compressortypes registry
+    Codecs/
+      Codecs.jl         # abstract Codec, zencode/zdecode
+      V3/
+        V3.jl           # BytesCodec, TransposeCodec only
+    Filters/
+      Filters.jl        # Filter + all pure-Julia filter implementations
+      fixedscaleoffset.jl, fletcher32.jl, shuffle.jl, delta.jl, quantize.jl, vlenfilters.jl
+    Storage/
+      Storage.jl        # AbstractStore, storageregexlist, metadata I/O
+      directorystore.jl
+      dictstore.jl
+      consolidated.jl
+    ZArray.jl
+    ZGroup.jl
+```
+
+### Zarr.jl (top-level, rewritten as thin wrapper)
+
+```
+src/
+  Zarr.jl              # using ZarrCore, re-exports, registrations
+  Compressors/
+    blosc.jl
+    zlib.jl
+    zstd.jl
+  Codecs/
+    V3/
+      blosc.jl
+      gzip.jl
+      zstd.jl
+      crc32c.jl
+      sharding.jl
+  Storage/
+    http.jl
+    gcstore.jl
+    zipstore.jl
+```
+
+## Registration Pattern
+
+Zarr.jl's `__init__` populates ZarrCore's registries:
+
+```julia
+# Compressors
+ZarrCore.compressortypes["blosc"] = BloscCompressor
+ZarrCore.compressortypes["zlib"]  = ZlibCompressor
+ZarrCore.compressortypes["zstd"]  = ZstdCompressor
+
+# V3 codecs
+ZarrCore.codectypes["blosc"]  = BloscV3Codec
+ZarrCore.codectypes["gzip"]   = GzipV3Codec
+ZarrCore.codectypes["zstd"]   = ZstdV3Codec
+ZarrCore.codectypes["crc32c"] = CRC32cV3Codec
+
+# Store URL resolvers
+push!(ZarrCore.storageregexlist, r"^https?://" => HTTPStore)
+push!(ZarrCore.storageregexlist, r"^gs://"     => GCStore)
+```
+
+## Migration Strategy
+
+- **Move** files from `src/` to `lib/ZarrCore/src/` (preserves git history)
+- **Split** `Compressors/Compressors.jl`: abstract type + NoCompressor + registry to Core; blosc/zlib/zstd stay
+- **Split** `Codecs/V3/V3.jl`: BytesCodec + TransposeCodec to Core; compression codecs stay
+- **Split** `Storage/`: directorystore + dictstore + consolidated to Core; http/gcstore/zipstore stay
+- **Rewrite** top-level `src/Zarr.jl` as thin wrapper
+
+## ZarrCore Dependencies
+
+```toml
+[deps]
+JSON = "..."
+DiskArrays = "..."
+OffsetArrays = "..."
+DateTimes64 = "..."
+Dates = "..."        # stdlib
+```
+
+## Zarr.jl Dependencies (in addition to ZarrCore)
+
+```toml
+[deps]
+ZarrCore = "..."
+Blosc = "..."
+CRC32c = "..."
+ChunkCodecCore = "..."
+ChunkCodecLibZlib = "..."
+ChunkCodecLibZstd = "..."
+HTTP = "..."
+OpenSSL = "..."
+URIs = "..."
+ZipArchives = "..."
+```