diff --git a/README.md b/README.md index 34e2c07..c4402cb 100644 --- a/README.md +++ b/README.md @@ -2,6 +2,13 @@ Experimental implementation of a SQLite-based store for [zarr](https://zarr.dev/) v3, in python. +SQLiteStore provides a single-file storage backend for Zarr. Key advantages over alternative single-file formats (e.g., ZipStore): + + - Support for key deletion, overwriting, and partial value writes. + - Full ACID guarantees provided by SQLite. + - High availability of SQLite implementations across programming languages + and environments. + Example usage: ```python @@ -20,3 +27,9 @@ with SQLiteStore("my_zarr_file.sqlite") as store: `SQLiteStore` otherwise behaves identically to other stores used with zarr, see the [zarr user guide](https://zarr.readthedocs.io/en/stable/user-guide/storage.html) for more information. + +## Specification + +The store format is described in the document [SPEC.md](/SPEC.md). This document +should allow the implementation of SQLiteStore for other programming languages +or Zarr libraries. \ No newline at end of file diff --git a/SPEC.md b/SPEC.md new file mode 100644 index 0000000..ef5e686 --- /dev/null +++ b/SPEC.md @@ -0,0 +1,215 @@ +# Zarr SQLiteStore specification version 1.0 + +## Scope + +This document describes **SQLiteStore**, a [Zarr (v3)](https://zarr.dev/) store +that allows the storage of a complete Zarr hierarchy inside a single file by +using the SQLite database engine. + +## Notes about the design decisions + +SQLiteStore is designed primarily for sharing moderately sized datasets by storing an entire dataset in a single file. + +Although SQLite supports databases several terabytes in size, no file size limit is defined by this specification. However, SQLiteStore is intended for datasets up to approximately 10 GB. + +SQLite was chosen to accomplish this goal for the following reasons: + +- SQLite has been in widespread use for a long time and is considered to be + highly robust software. +- Unlike many archive formats, SQLite natively supports Zarr store operations such as key deletion, overwriting values, and modifying stored values. +- SQLite libraries are available for nearly all programming languages and + computers. + +SQLiteStore was not designed to compete in read or write speed with other +stores, such as *FileSystemStore*. + +## Document conventions + +The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, +“SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be +interpreted as described in [IETF RFC +2119](https://datatracker.ietf.org/doc/html/rfc2119). + +## Definitions + +The terms *store*, *hierarchy*, *chunk* and *array* are defined by the [Zarr core specification](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html#concepts-and-terminology). + +A SQLiteStore file is a SQLite database conforming to this specification. + +An *implementation* refers to library or software that is designed to read and +write SQLiteStore files by implementing the operations defined by the [*Abstract store interface*](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html#abstract-store-interface). + +A *reader* is an implementation that is reading a SQLiteStore +file. + +A *writer* is an implementation that is creating or modifying a +SQLiteStore file. + +A *client* refers to software that uses an *implementation*, for example +by calling the functions defined by the implementation. Examples of clients +may include [zarr-python](https://zarr.readthedocs.io/en/stable/) and [zarrs]( +https://zarrs.dev/). + +## Store Capabilities + +SQLiteStore implements the [*Abstract store +interface*](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html#abstract-store-interface) +as defined by the *Zarr v3 core specification*. SQLiteStore supports the +capabilities **Readable**, **Writable** and **Listable**. However, specific +implementations of SQLiteStore may choose to support only some of these +capabilities (for example, a read-only SQLiteStore implementation may support +only Readable and Listable). + +## Canonical URI + +## File format + +Zarr SQLiteStore files must conform to the on-disk SQLite file format used by +SQLite releases 3.0.0 and later. This file format is described by the document +[*Database File Format*](https://sqlite.org/fileformat.html). Zarr SQLiteStore +files are therefore regular SQLite files which use a specific database schema, +defined by this document. + +The file extension `.zarrdb` is recommended for Zarr SQLiteStore files. + +### Minimal SQLite version + +Files conforming to this specification require features introduced in SQLite +3.7.17. + +### Application ID + +The SQLite file format defines a 4-byte integer at offset 68 which can be used +to identify the application-specific file type. SQLiteStore files must have this +value set to the hexadecimal integer `0x10b50760`. Implementations may satisfy +this requirement by executing the following SQL pragma statement: + +```sql +PRAGMA application_id = 0x10b50760 +``` + +## Database schema + +SQLiteStore files must contain two tables, as described below. The value columns +*v* must not contain NULL values and must therefore be specified with a +`NOT NULL` constraint in the table schemas. + +Implementations may satisfy this requirement by executing the following SQL +statements: + +```sql +CREATE TABLE IF NOT EXISTS sqlitestore_metadata( + k TEXT PRIMARY KEY, + v TEXT NOT NULL +); +CREATE TABLE IF NOT EXISTS zarr( + k TEXT PRIMARY KEY, + v BLOB NOT NULL +); +``` + +Readers must ignore all other tables present within the file. + +### Table `sqlitestore_metadata` + +Schema: + +| Column | Type | Description | +|--------|------|-------------| +| k | `TEXT` | Metadata key | +| v | `TEXT` | Metadata value string | + +The `sqlitestore_metadata` table stores metadata that is useful for the +SQLiteStore implementation. + +The `sqlitestore_metadata` table must include records which are marked +*required* in the table below. Writers must not insert in the +`sqlitestore_metadata` table any record with a key that is not described here. +Readers must ignore any record with a key that is not described here. + +| Key | Required | Description of value | +|-----|----------|----------------------| +| `sqlitestore_version` | Yes | SQLiteStore file format version, formatted as `MAJOR.MINOR`. For SQLiteStore version 1.0, this value must be the string "1.0". | +| `compatible_flags` | Yes | Comma-separated list of flag strings. Reserved for future extensions to the store format. | +| `incompatible_flags` | Yes | Comma-separated list of flag strings. Reserved for future extensions to the store format. | +| `created_by` | No | Arbitrary string describing the software that wrote the file. | +| `created_time` | No | Timestamp indicating when the file was most recently modified, formatted as a string that conforms to [RFC3339](https://datatracker.ietf.org/doc/html/rfc3339). | + +Readers must reject files whose major version is unsupported, as specified by +the `sqlitestore_version` metadata record. Readers should ignore minor versions +greater than those they implement unless incompatible_flags contains unknown +values. + +Compatible flags may be used in future versions of SQLiteStore to declare that +the store uses optional features but may still be read or written to by an +implementation that does not implement these features. + +Incompatible flags may be used in future versions of SQLiteStore to declare that +the store uses optional features that must be supported by the implementation in +order to read or write the store. + +Readers must: + +1. Ignore all compatible flags it does not recognize; + +2. Reject the file if any incompatible flag is not recognized. + +In SQLiteStore version 1.0, no flags are defined. Writers must therefore set the +values of "compatible_flags" and "incompatible_flags" to the empty string +(`""`). + +The `created_by` record is informational. Readers must not use it to determine +how the file is to be interpreted. The file format must be fully determined by +the metadata records sqlitestore_version, compatible_flags, and +incompatible_flags. + +### Table `zarr` + +Schema: + +| Column | Type | Description | +|--------|------|-------------| +| k | `TEXT` | Primary key. The Zarr key (full path as a Unicode string). | +| v | `BLOB` | Binary value. The chunk or metadata content. | + +The `zarr` table acts as a key-value store for the Zarr data. + +Keys are stored exactly as provided by the Zarr client, without modification. +Per the *Abstract store interface* specification, keys are Unicode strings. +Implementations must preserve the logical Unicode string regardless of the +underlying SQLite database encoding. No Unicode normalization shall be +performed. Implementations must preserve the key comparison semantics defined by +the Zarr core specification. + +Writers must insert only valid keys in the zarr table. Valid keys are +those which respect the following rules: + +1. The final character of the key must not be a `/` character. +2. The first character of the key must not be a `/` character. +3. A key must not contain the substring `//`. + +The empty string is a valid key. + +Values are stored as binary blobs (SQLite type `BLOB`) exactly as provided by +the Zarr client, without modification. + +## Database journaling mode + +The default journaling mode of SQLite is `DELETE`, and this mode is recommended +for SQLiteStore files. + +Implementations may use journal mode `WAL` (write-ahead log) for performance or +concurrency reasons, especially in write-heavy use cases. When using `WAL` +journal mode, implementations should attempt to restore the journal mode to +`DELETE` or perform a WAL checkpoint before the database is closed. + +The purpose of this best-effort requirement is to ensure all data is moved by +SQLite from the WAL files (files suffixed with `-wal` and `-shm` automatically +created by SQLite) to the main database file, so that the SQLiteStore file +consists of a single self-contained file that can be safely shared. + +## Limitations + +The maximum size of a value that can be stored (for example, an array chunk) is +limited to 2147483645 bytes (3 bytes less than 2 GiB) by current SQLite +implementations.