Conversation
- Clarifications - Add description of which metadata records are required - Add created-by record to metadata table (not required) - Add link to the spec in README
|
@newville I would appreciate your review and comments. |
| files are therefore regular SQLite files which use a specific database schema, | ||
| defined by this document. | ||
|
|
||
| The file extension `.zarrdb` is **recommended** for Zarr SQLiteStore files. |
There was a problem hiding this comment.
is this an extension that sqlite tools will recognize? something like .zarr.db might work better if not
There was a problem hiding this comment.
I don't know that there is really a standard file extension.
There was a problem hiding this comment.
My preference is for a single extension, it's easier to work with using splitext and similar functions. There is no standard extensions for sqlite, .db and .sqlite are somewhat common, but it is also common to see app-specific extensions for their application.
|
@auxym Thanks very much. I think this looks really good. But I will also readily admit that I know very little about the ZEP process or expectations. I might suggest stating that this schema is "version 1.0" and that other schema (perhaps "table per group") are conceivable in the future, but would need an update to the spec and updates to supporting libraries. For metadata fields, I agree that "created-by" is more general and suitable for the general spec. I might say that some value like "created_datetime" could be listed as "optional metadata", as original timestamps can be lost. If we are bike-shedding the schema, I might have a slight preference for "key" and "value" or "val" over "k" and "v", but that is minor.
'.db' is used by many tools. It is used for many sqlite3 databases, but, '.db3', '.sqlite' and '.sqlite3' are also common. I don't know of any sqlite tools that enforce ".db". I think '.zarrdb' is a fine default, but agree most strongly with the "recommended" here. |
|
|
||
| | Key | Required | Description of value | | ||
| |-----|----------|----------------------| | ||
| | `sqlitestore_version` | Yes | Version of the SQLiteStore file format in `MAJOR.MINOR` format. As of the present document, this should be the string "1.0". | |
There was a problem hiding this comment.
This could instead be stored as pragma user_version.
There was a problem hiding this comment.
The Spec here conveniently specifies finite rules for the tables (and exactly two of them) to used by Zarr. It does not forbid other tables in the same database. That might be very convenient for application-specific files that want to store some data in "chunked, compressed numerical data" but also store other data in multi-column tables, maybe even really with relations (or just prefer to store a list of strings or json structures using "plain" datatypes instead of as Zarr arrays). Zarr itself would not see those other tables, but a downstream app could.
I think that the sqlite3 pragmas for user_version or application_data are then probably best left to the users and applications, not to the Zarr spec.
There was a problem hiding this comment.
That is an interesting consideration --- do you anticipate that to be a likely use case?
An advantage of user_version and application_id is that they can be used for file-format auto-detection quite easily, because they are at fixed offsets from the start of the file. E.g. the standard file tool could easily be updated to indicate that it is a zarr sqlite store. Potentially they could be recommended for format auto-detection but not required.
If the intent is to allow both zarr and non-zarr data in the same database you might also consider using a clearer name than sqlitestore for the metadata table, e.g. zarr_sqlite_metadata.
There was a problem hiding this comment.
Excellent idea for the application ID, I was not aware of this feature and I will look into it.
For the user_version, I'm a bit conflicted. On one hand, it's a fixed offset value and makes it easy to determine the schema version without having to call into sqlite. On the other hand, I liked having "everything in one place" (the metadata table).
@newville concerning the possibility of adding extra tables -- I think it would probably be best to just leave that unspecified. If people want to do it, it won't break readers, but they should know they are doing something non-standard. Future versions of the spec may define feature flags that could signal, for example, other tables with whatever in them.
| files are therefore regular SQLite files which use a specific database schema, | ||
| defined by this document. | ||
|
|
||
| The file extension `.zarrdb` is **recommended** for Zarr SQLiteStore files. |
There was a problem hiding this comment.
It would be nice to get an assigned application_id for this format. https://sqlite.org/pragma.html#pragma_application_id
| - Full ACID guarantees provided by SQLite. | ||
| - High availability of SQLite implementations across programming languages | ||
| and environments. | ||
|
|
There was a problem hiding this comment.
A key disadvantage to be aware of is that the standard sqlite c implementation performs all i/o sequentially and reads a page at a time Therefore the read throughput will be quite low for high latency storage, like s3, gcs, etc. although you can try to mitigate that with readahead heuristics or use an aggregative alternative implementation.
There was a problem hiding this comment.
sorry, but "a key disadvantage" compared to what?
There was a problem hiding this comment.
I am really not nowledgeable enough on sqlite to comment on this. I will however update the spec to have a more neutral tone. This information (advantages/disadvantages) would better go in the README, I think.
For s3/blob storage, zarr already supports native storage backend that will likely be more performant. The goal of SQLiteStore, for me, was never really about better performance (compared to zip, s3).
|
Another important caveat with sqlite format is that individual large rows, including blobs are inherently stored as a linked list of overflow pages. Therefore reading a large value may be slow on high latency storage, because each page will have to be read sequentially, and byte range reads of individual values will also not be particularly efficient. |
|
The largest binary blob size is 2 GiB in sqlite, but we will likely want to keep the blobs relatively small due to the overflow page mechanism. I also suggest that the specification include some recommendations about setting the page size and some consideration about aligning the Zarr chunk size and the sqlite page size when compression is not being used. I think we would want the Zarr chunk size to be a small multiple of the page size. https://www.sqlite.org/pragma.html#pragma_page_size It seems to me that we would want to set the page size to be the largest value in most cases. With 64 KiB pages, a 1 MB chunk would only consist of at most 16 overflow pages. Another consequence of this paging structure is that we should recommend against using sharding within a sqlite file. |
This may be tricky because some portion of the value will be stored inline and the rest in the overflow page.
A downside of a large page size is more wasted space, because an overflow page can only hold a single row. Therefore, the space consumed by the overflowing portion of the value is rounded up to a multiple of the page size.
Yes, indeed. |
|
This specification is mainly written as a specifiation to store a Zarr array into a single sqlite file. That seems appropriate for arrays of a certain scale. For arrays of a large scale, we may want to partition (shard) the array among several sqlite files. At minimum, I think we should add some scope statements to the specification that as written as single file this intended for arrays of several gigabytes at most. I also see some need for cloud-optimization should the sqlite files be stored in remote cloud store like S3. It may make sense to retrieve and cache information from the dbstat table. For example, we may want to cache the byte offsets of pages and figure out how to implement a virtual file system layer to aggregate small partial reads into larger partial reads. This could be used so that reading a single chunk can be done via a single S3/HTTP request which would mitigate the overflow paging issues discussed above. That said, I would like to see the multiple file case addressed at some point, but this may be a distinct specification for a codec analogous to the current core sharding_indexed codec. Perhaps here it could just be stated as potential alternative that is out of scope for this specification. |
|
As I see it, this sqlite store is best suited for moderate size zarr hierarchies (total size less than ~10GB) intended for read/write use on a local filesystem or low-latency network filesystem. For cloud storage, or large datasets that you would want to split over large files, I don't think this is a great fit --- something like icechunk or OCDBT (supported by tensorstore and neuroglancer) would almost certainly be better. |
|
It is great that SQLite pragmas can be adjusted to improve efficiencies, but I would be reluctant to have these hard-coded in the Spec. It might be OK for the spec to state which pragmas should be settable from the API when creating a SQLiteStore, and maybe recommend defaults. Having guidance in user-level documentation might be good enough. I don't see such options for Zip or LocalFile storage. There is general guidance on "how to think about setting chunk sizes" (though maybe I am confusing it with HDF5), but that seems mostly independent of storage. I agree that "50 GB or smaller" is a fine assumption for SQLiteStore (and probably ZipStore too). It's awesome that Zarr can handle much larger datasets and supports access over cloud storage with fsspec, etc, but that might not be the main use case for SQLiteStore. As a data producer, and supporting analysis tools for such datasets, I'm assuming that data-producing and data-analyzing tools would create SQLiteStore files on behalf of users (basically, in place of HDF5, also in the ~50GB or smaller range). Such applications are probably the best place to make assumptions about array sizes, and the number of distinct groups and arrays that would be stored, and so are probably the right place to set such options. Making those options clear and giving guidance on setting values would be great. |
Agreed, I will add this mention to the spec. I wonder, if a future version adds an "alternative schema", would this be defined by a flag? Or maybe we should add a new record "schema_type" or similar to the metadata table?
Yeah I don't see a big downside to adding an optional datetime record to metadata, I'll add it.
I agree but since it is mostly inconsequential, I thought keeping some compabibility with the zarr-v2 implementation (which used k/v) was preferable. |
I agree, I will make this change. Regarding various performance optimizations being discussed (page size and whatnot): since they do not break file-level compatibility between implementations, I think they should be left unspecified and implementation specific. I am open to future revisions of the spec including some performance recommendations. |
|
@auxym Thanks, yes I agree that pragmas probably do not change the spec. That is, unless one wants to spell out which pragmas must be exposed at db creation. But I think it is okay to decide that the spec leaves pragmas up to downstream apps and users. I think it might be enough to say that a valid SQLiteStore v1 MUST have a table named 'sqlitestore_metadata' table, which MUST have a value for key= |
|
|
I played with an sqlite store before and thought about splitting it across 2 tables, one mapping zarr chunk keys to content hashes and then the chunks themselves in a table keyed on the content hash. That way you get deduplication, if that's of interest, and your key table rows are of more predictable size which might make listing and existence checks smoother. |
|
@mkitti Thanks for the reference to the OME-Zarr spec. |
|
@clbarnes I think that could be very interesting. Do you have any suggestions for how the Spec (and API) could be altered for that? Performance info and tuning would be great, but probably needs realistic example datasets. Perhaps asking for too much (or sorry for the ignorance if it already exists), but is there an obvious set of examples or test data sets? That seems like it would be very helpful for comparing FileStore, ZipStore, and SQLiteStore, and choices for the schema of SQLiteStore, different filesystems for FileStore, or the available options for ZipStore. That's slightly off-topic for this Spec, but it seems like it would make much of the discussion here more concrete. |
That's an interesting idea. I see two drawbacks:
Is de-duplication an appreciable gain? How common are duplicate chunks in zarr datasets? My intuition says "probably very rare", except maybe for trivial data (full array of 0's) which compresses very well anyways. Concerning the gain on row size, unfortunately I'm not knowledgeable enough of sqlite internals to judge. If someone if interested in this, maybe they could create a few benchmarks? I'm not against going in this direction if it demonstrates measurable performance gains. Otherwise, I would propose keeping this idea as an alternative format/schema which could be added to a future version of the spec. |
|
I agree that duplicated chunks would be rare and that hashing could be expensive, but the sqlite store would probably be best suited for small chunks (=cheaper hashes and more likely duplication). The implementation complexity is pretty minor, but still likely not worth bothering with. |
|
I would suggest against adding something like hashing/deduplication to this store. It is the kind of thing that could be achieved though a storage adapter (akin to icechunk) or even better a storage transformer. |
|
probably simplicity is a virtue for the initial spec. unless there are use cases that are doomed without hashing / deduplication / other elaborations, maybe this spec should just commit to the simplest thing for starters |
- Replace the Context section with a Scope section with more neutral terminology. - Add a section for notes about design decisions - Add document conventions with a reference to RFC-2119 - Add a definitions section - Add application_id and require sqlite 3.7.17 (released 2013-05-20) - Specify explicitly how readers must behave in various places - Add a NOT NULL constraint to the schema - Add an optional created_time metadata record - Specify explicitly how readers should handle versions and flags - Add a recommendation on journaling mode - Add an informative section on limitations.
|
Hi all, thank you for your comments and suggestions. I have made a commit with many modifications to the spec.
I think we are converging on something I can merge soon. I'd once again appreciate your review, comments and recommendations. |
|
I don't have anything technical to add but I just want to say that this direction looks great, thanks to everyone for working on this |
|
Oh! Also I forgot, I would need guidance from zarr devs (@d-v-b @LDeakin @jbms) on the Canonical URI section, which I added but left empty. Sqlite itself natively supports I did notice that the ZipStore proposal includes the URI prefix |
I don't know that a canonical URL is required, but according to my URL pipeline proposal zip should be: file:///path/to/archive.zip|zip:path/within/archive and similarly for this proposal it could be: file:///path/to/database.zarrdb|sqlite:path/within/sqlitestore |
|
Regarding the URI, I was referring to this section of the zarr core spec:
|
|
Thanks -- I think URI is a good discussion point.
I don't see that added backslash at ZipStore proposal. Aside from a knee-jerk "yuck" at mixing forward and backslashes, I suggest using actual URI conventions Following the suggestion from @auxym, something like would probably be a reasonable option. Using would be a fine option too. With that approach, it is imaginable that in an idealized future, |
|
That isn't a backslash, it is a vertical bar character. |
|
Another question for the zarr devs: The core spec is somewhat ambiguous on this topic but appears to imply that If so, we should probably add to the sqlitestore spec a key normalization step like:
|
|
This is what In regards to the canonical URL, I think we should relax the spec there and make it not required. The That Zip store spec hasn't landed yet, but it only formalises what is in I've used the "storage adapter" terminology in |
|
@LDeakin thank you for that info on zarrs's handling of leading slashes. I think it is actually a better idea to implement these same restrictions in the spec, rather than describe a key normalization algorithm (even if simple). |
I rest my case for using the standard https://en.wikipedia.org/wiki/Uniform_Resource_Identifier#Syntax. See also https://peps.python.org/pep-0020/, in particular "Readability counts". |
|
The URL situation is quite confusing because we need to describe at least three distinct concepts and their composition:
In isolation, all of these have been used as the "scheme" part of a URL in various contexts. The URL syntax issue is another long running discussion. See the draft ZEP 8: Pragmatically, I think the URL syntax is a larger discussion that probably belongs its own specification (e.g. ZEP8) and cannot be easily settled here. I would just add an addendum here contigent on composition with ZEP 8 that the "sqlite" scheme is reserved. That said, the ZEP8 URL pipeline scheme does have significant traction with implementations across multiple clients now: |
|
In the context of my url-pipeline specification, I realized that I overlooked the fact that the url-pipeline specification is intended to apply to multiple data formats, not just zarr, and therefore it would not be appropriate in that context to reserve "sqlite" for a zarr-specific sqlite kvstore adapter. Instead it should be something that explicitly includes zarr, like |
Agreed in general, but in my url-pipeline specification I instead distinguish between "root schemes" and "adapter schemes". Root schemes: file, http, s3, s3+https, gs, webdav, etc. Adapter schemes: icechunk, ocdbt, zip, sqlite, zarr, n5, neuroglancer-precomputed While the s3 protocol is a http-based protocol, in the url-pipeline specification I don't expose that layering beyond the fact that there is an icechunk is layered on top of an arbitrary key-value store, while the s3 protocol is specifically an http-based protocol. |
Yes, it does look to be complex. Still, staying within syntactic conventions of the URL is probably wise. Use schema with only the well-supported registered values like "https", "file" -- these are supposed to be registered. I can believe that "s3" or "s3+https" is registered. Stating storage as query key-value pairs might be the best approach (expandable and also optional). Using the fragment (with no imposed syntax) for the data_access and slices is probably the most flexible, since all of "/", ";", and ":" would be common here, which would not work well as part of path or query.
I don't see a ZEP 8 listed as an active or draft ZEP at https://zarr.dev/zeps/. The list of ZEP meetings there ends in 2024, which might be related. The ZEPs listed only give a number, making it difficult to know what the subject is. |
I would refer this matter to @zarr-developers/steering-council . As I mentioned previously, the most recent zip file related efforts are pursuing specification via OME-Zarr. |
|
For zip a specification is not as important because the default thing to do, just treat the zip contents like any other directory, works fine. For sqlite something like this specification is required since sqlite is not natively a container of a directory tree. |
|
OK, so concerning the URI issue, I conclude that we should probably leave that unspecified for now. |
7 participants
Here is a draft spec for the SQLiteStore. It is quite simple, but I figure we should at least start with something. I think we should agree on a somewhat stable spec before this package is published to PyPI and files start profilerating.
Metadata table.
Similar to what was proposed in #2, this spec adds a second table which is used exclusively to write data which is necessary for the interpretation of the store format.
I decided against storing the following fields:
The metadata table is not yet implemented in the python lib.
I am open to any comments, modifications, additions. I will merge this PR and publish a 1.0 version of this package to PyPI once we reach a sort of consensus on the spec.