From f7a60376b145295594eee4c69586814076db94ff Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Francis=20Th=C3=A9rien?= Date: Fri, 10 Jul 2026 14:09:57 -0400 Subject: [PATCH 1/6] Add spec (1st draft) --- SPEC.md | 85 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 SPEC.md diff --git a/SPEC.md b/SPEC.md new file mode 100644 index 0000000..e148410 --- /dev/null +++ b/SPEC.md @@ -0,0 +1,85 @@ +# Zarr SQLiteStore Specification Version 1.0 + +## Context + +SQLiteStore is a single-file storage backend for [Zarr (v3)](https://zarr.dev/) +that stores array metadata and chunk data in an SQLite database. + +Key advantages of SQLiteStore over alternative single-file formats (e.g., Zip): + + - Full support for key deletion, overwriting, and partial value writes. + - Full ACID guarantees provided by SQLite. + - High availability of SQLite implementations across programming languages + and environments. + +SQLiteStore implements the [*Abstract store +interface*](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html#abstract-store-interface) +as defined by the *Zarr v3 core specification*. SQLiteStore as defined by the +current document is a store implementation which is **Readable**, **Writeable** +and **Listable**. However, implementations of SQLiteStore may choose to support +only some of these capabilities (for example, an SQLiteStore implementation may +support only Readable and Listable). + +## File Format + +Zarr SQLiteStore files MUST conform to the on-disk SQLite file format used by +SQLite releases 3.0.0 and later. This file format is described by the document +[*Database File Format*](https://sqlite.org/fileformat.html). Zarr SQLiteStore +files are therefore regular SQLite files, but are structured with a specific +schema, defined by this document. + +The file extension `.zarrdb` is **recommended** for Zarr SQLiteStore files. + +## Database schema + +The database contains two tables, as described below. + +The schema may be created by the following SQL statements: + +```sql +CREATE TABLE IF NOT EXISTS sqlitestore_metadata(k TEXT PRIMARY KEY, v TEXT); +CREATE TABLE IF NOT EXISTS zarr(k TEXT PRIMARY KEY, v BLOB); +``` + +### Table `sqlitestore_metadata` + +| Column | Type | Description | +|--------|------|-------------| +| k | `TEXT` | Primary key. The Zarr key (full path as unicode string). | +| v | `TEXT` | Binary value. The chunk or metadata content. | + +The `sqlitestore_metadata` table stores metadata that is useful for the SQLiteStore +implementation. + +The `sqlitestore_metadata` table MUST include the following records. + +| Key | Description of value | +|-----|----------------------| +| `sqlitestore_version` | Version of the SQLiteStore file format in `MAJOR.MINOR` format. As of the present document, this should be the string "1.0". | +| `compatible_flags` | List of flag strings separated by a comma (`0x2C`) character. Reserved for future extensions to the store format. | +| `incompatible_flags` | List of flag strings separated by a comma (`0x2C`) character. Reserved for future extensions to the store format. | + +Compatible flags are used to declare that the store uses optional features but +may still be read or written to by an implementation which does not implement +these features. + +Incompatible flags are used to declare that the store uses optional features +that must be supported by the implementation in order to read or write the +store. + +### Table `zarr` + +| Column | Type | Description | +|--------|------|-------------| +| k | `TEXT` | Primary key. The Zarr key (full path as unicode string). | +| v | `BLOB` | Binary value. The chunk or metadata content. | + +The `zarr` table acts as a key-value store for the Zarr data. + +Keys are stored exactly as provided by the Zarr client, without modification. +Per the *Abstract store interface* specification, keys are Unicode strings. Keys +may be stored in any of the Unicode encodings supported by SQLite (database +encoding). + +Values are stored as binary blobs (SQLite type `BLOB`) exactly as provided by +the Zarr client, without modification. From 65fba8fa66beaae681a78be9090f7145a4b4eff0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Francis=20Th=C3=A9rien?= Date: Sat, 11 Jul 2026 12:23:56 -0300 Subject: [PATCH 2/6] Spec clarifications and add created-by metadata - Clarifications - Add description of which metadata records are required - Add created-by record to metadata table (not required) - Add link to the spec in README --- README.md | 13 +++++++++++++ SPEC.md | 40 ++++++++++++++++++++++++++-------------- 2 files changed, 39 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index 34e2c07..c4402cb 100644 --- a/README.md +++ b/README.md @@ -2,6 +2,13 @@ Experimental implementation of a SQLite-based store for [zarr](https://zarr.dev/) v3, in python. +SQLiteStore provides a single-file storage backend for Zarr. Key advantages over alternative single-file formats (e.g., ZipStore): + + - Support for key deletion, overwriting, and partial value writes. + - Full ACID guarantees provided by SQLite. + - High availability of SQLite implementations across programming languages + and environments. + Example usage: ```python @@ -20,3 +27,9 @@ with SQLiteStore("my_zarr_file.sqlite") as store: `SQLiteStore` otherwise behaves identically to other stores used with zarr, see the [zarr user guide](https://zarr.readthedocs.io/en/stable/user-guide/storage.html) for more information. + +## Specification + +The store format is described in the document [SPEC.md](/SPEC.md). This document +should allow the implementation of SQLiteStore for other programming languages +or Zarr libraries. \ No newline at end of file diff --git a/SPEC.md b/SPEC.md index e148410..be8863d 100644 --- a/SPEC.md +++ b/SPEC.md @@ -7,26 +7,28 @@ that stores array metadata and chunk data in an SQLite database. Key advantages of SQLiteStore over alternative single-file formats (e.g., Zip): - - Full support for key deletion, overwriting, and partial value writes. + - Support for key deletion, overwriting, and partial value writes. - Full ACID guarantees provided by SQLite. - High availability of SQLite implementations across programming languages and environments. +## Store Capabilities + SQLiteStore implements the [*Abstract store interface*](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html#abstract-store-interface) -as defined by the *Zarr v3 core specification*. SQLiteStore as defined by the -current document is a store implementation which is **Readable**, **Writeable** -and **Listable**. However, implementations of SQLiteStore may choose to support -only some of these capabilities (for example, an SQLiteStore implementation may -support only Readable and Listable). +as defined by the *Zarr v3 core specification*. SQLiteStore provides the store +capabilities **Readable**, **Writeable** and **Listable**. However, +specific implementations of SQLiteStore may choose to support only some of these +capabilities (for example, an SQLiteStore implementation may support only +Readable and Listable). ## File Format Zarr SQLiteStore files MUST conform to the on-disk SQLite file format used by SQLite releases 3.0.0 and later. This file format is described by the document [*Database File Format*](https://sqlite.org/fileformat.html). Zarr SQLiteStore -files are therefore regular SQLite files, but are structured with a specific -schema, defined by this document. +files are therefore regular SQLite files which use a specific database schema, +defined by this document. The file extension `.zarrdb` is **recommended** for Zarr SQLiteStore files. @@ -43,6 +45,8 @@ CREATE TABLE IF NOT EXISTS zarr(k TEXT PRIMARY KEY, v BLOB); ### Table `sqlitestore_metadata` +Schema: + | Column | Type | Description | |--------|------|-------------| | k | `TEXT` | Primary key. The Zarr key (full path as unicode string). | @@ -51,13 +55,14 @@ CREATE TABLE IF NOT EXISTS zarr(k TEXT PRIMARY KEY, v BLOB); The `sqlitestore_metadata` table stores metadata that is useful for the SQLiteStore implementation. -The `sqlitestore_metadata` table MUST include the following records. +The `sqlitestore_metadata` table MUST include records which are marked REQUIRED in the table below. -| Key | Description of value | -|-----|----------------------| -| `sqlitestore_version` | Version of the SQLiteStore file format in `MAJOR.MINOR` format. As of the present document, this should be the string "1.0". | -| `compatible_flags` | List of flag strings separated by a comma (`0x2C`) character. Reserved for future extensions to the store format. | -| `incompatible_flags` | List of flag strings separated by a comma (`0x2C`) character. Reserved for future extensions to the store format. | +| Key | Required | Description of value | +|-----|----------|----------------------| +| `sqlitestore_version` | Yes | Version of the SQLiteStore file format in `MAJOR.MINOR` format. As of the present document, this should be the string "1.0". | +| `compatible_flags` | Yes | List of flag strings separated by a comma (`0x2C`) character. Reserved for future extensions to the store format. | +| `incompatible_flags` | Yes | List of flag strings separated by a comma (`0x2C`) character. Reserved for future extensions to the store format. | +| `created_by` | No | Arbitrary string describing the software that wrore the file. | Compatible flags are used to declare that the store uses optional features but may still be read or written to by an implementation which does not implement @@ -67,8 +72,15 @@ Incompatible flags are used to declare that the store uses optional features that must be supported by the implementation in order to read or write the store. +The record `created_by` MUST NOT be relied on by readers to determine the method +by which the file is to be interpreted. The file format MUST be fully described +by the metadata records `sqlitestore_version`, `compatible_flags` and +`incompatible_flags`. + ### Table `zarr` +Schema: + | Column | Type | Description | |--------|------|-------------| | k | `TEXT` | Primary key. The Zarr key (full path as unicode string). | From 36886559e859e7f6a9aa3bf5dcb9a0713187af40 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Francis=20Th=C3=A9rien?= Date: Fri, 17 Jul 2026 21:03:03 -0400 Subject: [PATCH 3/6] Various additions and modifications to the proposed spec - Replace the Context section with a Scope section with more neutral terminology. - Add a section for notes about design decisions - Add document conventions with a reference to RFC-2119 - Add a definitions section - Add application_id and require sqlite 3.7.17 (released 2013-05-20) - Specify explicitly how readers must behave in various places - Add a NOT NULL constraint to the schema - Add an optional created_time metadata record - Specify explicitly how readers should handle versions and flags - Add a recommendation on journaling mode - Add an informative section on limitations. --- SPEC.md | 186 ++++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 147 insertions(+), 39 deletions(-) diff --git a/SPEC.md b/SPEC.md index be8863d..5983e0b 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1,78 +1,163 @@ -# Zarr SQLiteStore Specification Version 1.0 +# Zarr SQLiteStore specification version 1.0 -## Context +## Scope -SQLiteStore is a single-file storage backend for [Zarr (v3)](https://zarr.dev/) -that stores array metadata and chunk data in an SQLite database. +This document describes **SQLiteStore**, a [Zarr (v3)](https://zarr.dev/) store +that allows the storage of a complete Zarr hierarchy inside a single file by +using the SQLite database engine. -Key advantages of SQLiteStore over alternative single-file formats (e.g., Zip): +## Notes about the design decisions - - Support for key deletion, overwriting, and partial value writes. - - Full ACID guarantees provided by SQLite. - - High availability of SQLite implementations across programming languages - and environments. +SQLiteStore is designed primarily for sharing moderately sized datasets by storing an entire dataset in a single file. + +Although SQLite supports databases several terabytes in size, no file size limit is defined by this specification. However, SQLiteStore is intended for datasets up to approximately 10 GB. + +SQLite was chosen to accomplish this goal for the following reasons: + +- SQLite has been in widespread use for a long time and is considered to be + highly robust software. +- Unlike many archive formats, SQLite natively supports Zarr store operations such as key deletion, overwriting values, and modifying stored values. +- SQLite libraries are available for nearly all programming languages and + computers. + +SQLiteStore was not designed to compete in read or write speed with other +stores, such as *FileSystemStore*. + +## Document conventions + +The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, +“SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be +interpreted as described in [IETF RFC +2119](https://datatracker.ietf.org/doc/html/rfc2119). + +## Definitions + +The terms *store*, *hierarchy*, *chunk* and *array* are defined by the [Zarr core specification](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html#concepts-and-terminology). + +A SQLiteStore file is a SQLite database conforming to this specification. + +An *implementation* refers to library or software that is designed to read and +write SQLiteStore files by implementing the operations defined by the [*Abstract store interface*](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html#abstract-store-interface). + +A *reader* is a implementation which is performing read operations on a SQLiteStore +file. + +A *writer* is a implementation which is creating, modifying or adding data to a +SQLiteStore file. + +A *client* refers to software which uses an *implementation*, for example +by calling the functions defined by the implementation. Examples of clients +may include [zarr-python](https://zarr.readthedocs.io/en/stable/) and [zarrs]( +https://zarrs.dev/). ## Store Capabilities SQLiteStore implements the [*Abstract store interface*](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html#abstract-store-interface) -as defined by the *Zarr v3 core specification*. SQLiteStore provides the store -capabilities **Readable**, **Writeable** and **Listable**. However, -specific implementations of SQLiteStore may choose to support only some of these -capabilities (for example, an SQLiteStore implementation may support only -Readable and Listable). +as defined by the *Zarr v3 core specification*. SQLiteStore supports the +capabilities **Readable**, **Writable** and **Listable**. However, specific +implementations of SQLiteStore may choose to support only some of these +capabilities (for example, a read-only SQLiteStore implementation may support +only Readable and Listable). -## File Format +## Canonical URI -Zarr SQLiteStore files MUST conform to the on-disk SQLite file format used by +## File format + +Zarr SQLiteStore files must conform to the on-disk SQLite file format used by SQLite releases 3.0.0 and later. This file format is described by the document [*Database File Format*](https://sqlite.org/fileformat.html). Zarr SQLiteStore files are therefore regular SQLite files which use a specific database schema, defined by this document. -The file extension `.zarrdb` is **recommended** for Zarr SQLiteStore files. +The file extension `.zarrdb` is recommended for Zarr SQLiteStore files. + +### Minimal SQLite version + +Files conforming to this specification require features introduced in SQLite +3.7.17. + +### Application ID + +The SQLite file format defines a 4-byte integer at offset 68 which can be used +to identify the application-specific file type. SQLiteStore files must have this +value set to the hexadecimal integer `0x10b50760`. This requirement may achieve +this by executing the following SQL pragma statement: + +```sql +PRAGMA application_id = 0x10b50760 +``` ## Database schema -The database contains two tables, as described below. +SQLiteStore files must contain two tables, as described below. The value columns +*v* must not contain NULL values and must therefore be specified with a +`NOT NULL` constraint in the table schemas. The schema may be created by the following SQL statements: ```sql -CREATE TABLE IF NOT EXISTS sqlitestore_metadata(k TEXT PRIMARY KEY, v TEXT); -CREATE TABLE IF NOT EXISTS zarr(k TEXT PRIMARY KEY, v BLOB); +CREATE TABLE IF NOT EXISTS sqlitestore_metadata( + k TEXT PRIMARY KEY, + v TEXT NOT NULL +); +CREATE TABLE IF NOT EXISTS zarr( + k TEXT PRIMARY KEY, + v BLOB NOT NULL +); ``` +Readers must ignore all other tables present within the file. + ### Table `sqlitestore_metadata` Schema: | Column | Type | Description | |--------|------|-------------| -| k | `TEXT` | Primary key. The Zarr key (full path as unicode string). | -| v | `TEXT` | Binary value. The chunk or metadata content. | +| k | `TEXT` | Metadata key | +| v | `TEXT` | Metadata value string | -The `sqlitestore_metadata` table stores metadata that is useful for the SQLiteStore -implementation. +The `sqlitestore_metadata` table stores metadata that is useful for the +SQLiteStore implementation. -The `sqlitestore_metadata` table MUST include records which are marked REQUIRED in the table below. +The `sqlitestore_metadata` table must include records which are marked +*required* in the table below. Writers must not insert in the +`sqlitestore_metadata` table any record with a key that is not described here. +Readers must ignore any record with a key that is not described here. | Key | Required | Description of value | |-----|----------|----------------------| | `sqlitestore_version` | Yes | Version of the SQLiteStore file format in `MAJOR.MINOR` format. As of the present document, this should be the string "1.0". | -| `compatible_flags` | Yes | List of flag strings separated by a comma (`0x2C`) character. Reserved for future extensions to the store format. | -| `incompatible_flags` | Yes | List of flag strings separated by a comma (`0x2C`) character. Reserved for future extensions to the store format. | -| `created_by` | No | Arbitrary string describing the software that wrore the file. | +| `compatible_flags` | Yes | Comma-separated list of flag strings. Reserved for future extensions to the store format. | +| `incompatible_flags` | Yes | Comma-separated list of flag strings. Reserved for future extensions to the store format. | +| `created_by` | No | Arbitrary string describing the software that wrote the file. | +| `created_time` | No | Timestamp indicating when the file was most recently modified, formatted as a string that conforms to [RFC3339](https://datatracker.ietf.org/doc/html/rfc3339). | + +Readers shall reject files whose major version is unsupported, as specified by +the `sqlitestore_version` metadata record. Readers should ignore minor versions +greater than those they implement unless incompatible_flags contains unknown +values. + +Compatible flags may be used in future versions of SQLiteStore to declare that +the store uses optional features but may still be read or written to by an +implementation which does not implement these features. + +Incompatible flags may be used in future versions of SQLiteStore to declare that +the store uses optional features that must be supported by the implementation in +order to read or write the store. -Compatible flags are used to declare that the store uses optional features but -may still be read or written to by an implementation which does not implement -these features. +Readers shall: -Incompatible flags are used to declare that the store uses optional features -that must be supported by the implementation in order to read or write the -store. +1. ignore all compatible flags it does not recognize; -The record `created_by` MUST NOT be relied on by readers to determine the method +2. reject the file if any incompatible flag is not recognized. + +In SQLiteStore version 1.0, no flags are defined. Writers must therefore set the +values of "compatible_flags" and "incompatible_flags" to the empty string +(`""`). + +The record `created_by` must not be relied upon by readers to determine the method by which the file is to be interpreted. The file format MUST be fully described by the metadata records `sqlitestore_version`, `compatible_flags` and `incompatible_flags`. @@ -83,15 +168,38 @@ Schema: | Column | Type | Description | |--------|------|-------------| -| k | `TEXT` | Primary key. The Zarr key (full path as unicode string). | +| k | `TEXT` | Primary key. The Zarr key (full path as a Unicode string). | | v | `BLOB` | Binary value. The chunk or metadata content. | The `zarr` table acts as a key-value store for the Zarr data. Keys are stored exactly as provided by the Zarr client, without modification. -Per the *Abstract store interface* specification, keys are Unicode strings. Keys -may be stored in any of the Unicode encodings supported by SQLite (database -encoding). +Per the *Abstract store interface* specification, keys are Unicode strings. +Implementations shall preserve the logical Unicode string regardless of the +underlying SQLite database encoding. No Unicode normalization shall be +performed. Keys must be compared using SQLite's native TEXT comparison, for +example when running `SELECT` queries for key lookup. Values are stored as binary blobs (SQLite type `BLOB`) exactly as provided by the Zarr client, without modification. + +## Database journaling mode + +The default journaling mode of SQLite is `DELETE`, and this mode is recommended +for SQLiteStore files. + +Implementations may use journal mode `WAL` (write-ahead log) for performance or +concurrency reasons, especially in write-heavy use cases. When using `WAL` +journal mode, implementations should attempt to restore the journal mode to +`DELETE` or perform a WAL checkpoint is executed before the databased is closed. + +The purpose of this best-effort requirement is to ensure all data is moved by +SQLite from the WAL files (files suffixed with `-wal` and `-shm` automatically +created by SQLite) to the main database file, so that the SQLiteStore database +consists of a single self-contained file that can be safely shared. + +## Limitations + +The maximum size of a value which can be stored (for example, an array chunk) is +limited to 2147483645 bytes (3 bytes less than 2 GiB) by current SQLite +implementations. From 48cfce3c9747bc09d276863f7000207b3d520a8e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Francis=20Th=C3=A9rien?= Date: Sat, 18 Jul 2026 21:22:16 -0400 Subject: [PATCH 4/6] Edits for clarity --- SPEC.md | 39 ++++++++++++++++++++------------------- 1 file changed, 20 insertions(+), 19 deletions(-) diff --git a/SPEC.md b/SPEC.md index 5983e0b..069e65b 100644 --- a/SPEC.md +++ b/SPEC.md @@ -39,13 +39,13 @@ A SQLiteStore file is a SQLite database conforming to this specification. An *implementation* refers to library or software that is designed to read and write SQLiteStore files by implementing the operations defined by the [*Abstract store interface*](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html#abstract-store-interface). -A *reader* is a implementation which is performing read operations on a SQLiteStore +A *reader* is an implementation that is reading a SQLiteStore file. -A *writer* is a implementation which is creating, modifying or adding data to a +A *writer* is an implementation that is creating or modifying a SQLiteStore file. -A *client* refers to software which uses an *implementation*, for example +A *client* refers to software that uses an *implementation*, for example by calling the functions defined by the implementation. Examples of clients may include [zarr-python](https://zarr.readthedocs.io/en/stable/) and [zarrs]( https://zarrs.dev/). @@ -81,8 +81,8 @@ Files conforming to this specification require features introduced in SQLite The SQLite file format defines a 4-byte integer at offset 68 which can be used to identify the application-specific file type. SQLiteStore files must have this -value set to the hexadecimal integer `0x10b50760`. This requirement may achieve -this by executing the following SQL pragma statement: +value set to the hexadecimal integer `0x10b50760`. Implementations may satisfy +this requirement by executing the following SQL pragma statement: ```sql PRAGMA application_id = 0x10b50760 @@ -94,7 +94,8 @@ SQLiteStore files must contain two tables, as described below. The value columns *v* must not contain NULL values and must therefore be specified with a `NOT NULL` constraint in the table schemas. -The schema may be created by the following SQL statements: +Implementations may satisfy this requirement by executing the following SQL +statements: ```sql CREATE TABLE IF NOT EXISTS sqlitestore_metadata( @@ -128,39 +129,39 @@ Readers must ignore any record with a key that is not described here. | Key | Required | Description of value | |-----|----------|----------------------| -| `sqlitestore_version` | Yes | Version of the SQLiteStore file format in `MAJOR.MINOR` format. As of the present document, this should be the string "1.0". | +| `sqlitestore_version` | Yes | SQLiteStore file format version, formatted as `MAJOR.MINOR`. For SQLiteStore version 1.0, this value must be the string "1.0". | | `compatible_flags` | Yes | Comma-separated list of flag strings. Reserved for future extensions to the store format. | | `incompatible_flags` | Yes | Comma-separated list of flag strings. Reserved for future extensions to the store format. | | `created_by` | No | Arbitrary string describing the software that wrote the file. | | `created_time` | No | Timestamp indicating when the file was most recently modified, formatted as a string that conforms to [RFC3339](https://datatracker.ietf.org/doc/html/rfc3339). | -Readers shall reject files whose major version is unsupported, as specified by +Readers must reject files whose major version is unsupported, as specified by the `sqlitestore_version` metadata record. Readers should ignore minor versions greater than those they implement unless incompatible_flags contains unknown values. Compatible flags may be used in future versions of SQLiteStore to declare that the store uses optional features but may still be read or written to by an -implementation which does not implement these features. +implementation that does not implement these features. Incompatible flags may be used in future versions of SQLiteStore to declare that the store uses optional features that must be supported by the implementation in order to read or write the store. -Readers shall: +Readers must: -1. ignore all compatible flags it does not recognize; +1. Ignore all compatible flags it does not recognize; -2. reject the file if any incompatible flag is not recognized. +2. Reject the file if any incompatible flag is not recognized. In SQLiteStore version 1.0, no flags are defined. Writers must therefore set the values of "compatible_flags" and "incompatible_flags" to the empty string (`""`). -The record `created_by` must not be relied upon by readers to determine the method -by which the file is to be interpreted. The file format MUST be fully described -by the metadata records `sqlitestore_version`, `compatible_flags` and -`incompatible_flags`. +The `created_by` record is informational. Readers must not use it to determine +how the file is to be interpreted. The file format must be fully determined by +the metadata records sqlitestore_version, compatible_flags, and +incompatible_flags. ### Table `zarr` @@ -191,15 +192,15 @@ for SQLiteStore files. Implementations may use journal mode `WAL` (write-ahead log) for performance or concurrency reasons, especially in write-heavy use cases. When using `WAL` journal mode, implementations should attempt to restore the journal mode to -`DELETE` or perform a WAL checkpoint is executed before the databased is closed. +`DELETE` or perform a WAL checkpoint before the database is closed. The purpose of this best-effort requirement is to ensure all data is moved by SQLite from the WAL files (files suffixed with `-wal` and `-shm` automatically -created by SQLite) to the main database file, so that the SQLiteStore database +created by SQLite) to the main database file, so that the SQLiteStore file consists of a single self-contained file that can be safely shared. ## Limitations -The maximum size of a value which can be stored (for example, an array chunk) is +The maximum size of a value that can be stored (for example, an array chunk) is limited to 2147483645 bytes (3 bytes less than 2 GiB) by current SQLite implementations. From 120aeb5a8515dc653c931c56cd1555d5993bb774 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Francis=20Th=C3=A9rien?= Date: Sat, 18 Jul 2026 21:23:52 -0400 Subject: [PATCH 5/6] Add requirement of valid keys --- SPEC.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/SPEC.md b/SPEC.md index 069e65b..72634e1 100644 --- a/SPEC.md +++ b/SPEC.md @@ -181,6 +181,15 @@ underlying SQLite database encoding. No Unicode normalization shall be performed. Keys must be compared using SQLite's native TEXT comparison, for example when running `SELECT` queries for key lookup. +Writers must insert only valid keys in the zarr table. Valid keys are +those which respect the following rules: + +1. The final character of the key must not be a `/` character. +2. The first character of the key must not be a `/` character. +3. A key must not contain the substring `//`. + +The empty string is a valid key. + Values are stored as binary blobs (SQLite type `BLOB`) exactly as provided by the Zarr client, without modification. From 9f8455db4b39c2ac65daf2dd63a8cc6fba5e49a5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Francis=20Th=C3=A9rien?= Date: Mon, 20 Jul 2026 10:28:27 -0400 Subject: [PATCH 6/6] Edit key comparison requirement --- SPEC.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/SPEC.md b/SPEC.md index 72634e1..ef5e686 100644 --- a/SPEC.md +++ b/SPEC.md @@ -176,10 +176,10 @@ The `zarr` table acts as a key-value store for the Zarr data. Keys are stored exactly as provided by the Zarr client, without modification. Per the *Abstract store interface* specification, keys are Unicode strings. -Implementations shall preserve the logical Unicode string regardless of the +Implementations must preserve the logical Unicode string regardless of the underlying SQLite database encoding. No Unicode normalization shall be -performed. Keys must be compared using SQLite's native TEXT comparison, for -example when running `SELECT` queries for key lookup. +performed. Implementations must preserve the key comparison semantics defined by +the Zarr core specification. Writers must insert only valid keys in the zarr table. Valid keys are those which respect the following rules: