Zstd: append values in bulk and harden the frame metadata reads - #9099
Open
robert3005 wants to merge 1 commit into
Open
Zstd: append values in bulk and harden the frame metadata reads#9099robert3005 wants to merge 1 commit into
robert3005 wants to merge 1 commit into
Conversation
`append_to_builder` walked the decompressed frames one value at a time, appending each through a single-value call and interleaving null runs by hand. It now derives the value byte range from the slice's own metadata and hands the whole region to `append_value_slices`, which sizes the offsets, byte storage and validity once — one offset store plus one `memcpy` per value. Adds a `VarBinViewBuilder` path as well: the frames already hold the values contiguously, so the views can reference them in place instead of going through the canonical array and rewriting every view a second time to rebase its buffer index. `reconstruct_views` therefore takes the buffer index the views should start at. Frame metadata comes straight off disk, so the arithmetic it drives is now checked and surfaced as errors rather than panics or wrapped lengths: value counts, frame sizes, length prefixes and the offsets they walk to. The missing `n_values` fallback no longer reads a byte count as a value count for variable-width values, where that mis-attributes values to frames — it is accepted only for the single-frame case that is still recoverable. Decompression also writes through `WriteBuf` into uninitialized spare capacity rather than a `&mut [u8]` over memory nothing has written yet. Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
robert3005
commented
Jul 31, 2026
| row = valid_index + 1; | ||
| // Null rows carry an empty view, so scatter the stored values into their rows. | ||
| let mut views = BufferMut::<BinaryView>::zeroed(slice.n_rows); | ||
| for (view, index) in valid_views.into_iter().zip_eq(valid_indices) { |
Contributor
Author
There was a problem hiding this comment.
we should use for_each_valid_index if possible
robert3005
commented
Jul 31, 2026
Comment on lines
293
to
298
| array | ||
| .array() | ||
| .clone() | ||
| .execute::<Canonical>(ctx)? | ||
| .into_array() | ||
| .append_to_builder(builder, ctx) |
Contributor
Author
There was a problem hiding this comment.
this shouldn't be necessary
robert3005
marked this pull request as ready for review
July 31, 2026 13:53
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Write zstd values directly into the builder without additional copy