Measured while building the #79 benchmark store: a 21M x 768d vector-only store ingests in 82 min (4,270 rows/s) and then spends 102 min inside a single persist (32 vCPU, local NVMe). The .tvim base write is a few GB of buffered sequential IO; the dominant term is building and canonicalizing the JSON state for 21M documents (single-threaded, multi-GB intermediate tree plus a ~5.5 GB output string) and hashing it.
Until #84's follow-up (#87) released the GIL around persist, this also starved every other Python thread in the host process for the full duration, which is how it was found.
Directions:
- A binary or streamed state representation for the chunk table (the commit manifest already carries checksums; the JSON state is the scaling bottleneck).
- Incremental state: persist only the delta against the previous state body at commit time.
- Parallel canonicalization/hashing if JSON stays.
Store shape for reproduction is committed in the #85 harness (21M exact_bw4).
Measured while building the #79 benchmark store: a 21M x 768d vector-only store ingests in 82 min (4,270 rows/s) and then spends 102 min inside a single persist (32 vCPU, local NVMe). The .tvim base write is a few GB of buffered sequential IO; the dominant term is building and canonicalizing the JSON state for 21M documents (single-threaded, multi-GB intermediate tree plus a ~5.5 GB output string) and hashing it.
Until #84's follow-up (#87) released the GIL around persist, this also starved every other Python thread in the host process for the full duration, which is how it was found.
Directions:
Store shape for reproduction is committed in the #85 harness (21M exact_bw4).