Introduce array_slots macro for type-safe array slot access - #8912
Merged
Conversation
Contributor
Benchmarks: Vortex queries 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.995x ➖, 0↑ 0↓)
datafusion / parquet (0.993x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (0.976x ➖, 0↑ 0↓)
duckdb / parquet (0.996x ➖, 0↑ 0↓)
No file size changes detected. |
Merging this PR will not alter performance
|
Migrate every array encoding with a fixed set of child slots to the #[array_slots] proc macro, replacing hand-written slot index constants (*_SLOT/NUM_SLOTS/SLOT_NAMES), from_slots/into_slots plumbing, and per-slot ext-trait accessors with the generated Slots struct, SlotsView, and *ArraySlotsExt trait. Newly ported arrays: - vortex-array: Primitive, Bool, Decimal, VarBin, VarBinView, FixedSizeList, List, ListView, Filter, Slice, Extension, Shared, Variant - fastlanes: FoR, Delta, RLE - encodings: RunEnd, ZigZag, FSST, ALPRD, Pco, ByteBool, Zstd, DecimalByteParts, ParquetVariant Slot field order everywhere matches the previous slot indices, and slot names match the previous SLOT_NAMES, so serialization is unchanged. Validity-style slots are Option<ArrayRef>; Validity-returning helpers that would collide with generated accessors follow the Masked pattern (e.g. bytebool_validity, parquet_variant_validity, unsliced_validity). Ext traits that kept non-slot methods are now supertraited on the generated *ArraySlotsExt; traits that only had slot accessors were removed. Downstream call sites across vortex-cuda, vortex-arrow, vortex-btrblocks, vortex-compressor, vortex-duckdb, vortex-row, vortex-geo, vortex-tensor, fuzz, and benches import the generated traits. Arrays with variable-length children (Chunked, Struct, Union, Interleave, ScalarFn, ZstdBuffers) and zero-slot arrays (Null, Constant, Sequence) are intentionally not migrated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BGk7kM4exZVJ5STJxt4dxK Signed-off-by: Claude <noreply@anthropic.com>
joseph-isaacs
force-pushed
the
claude/array-slots-macro-migration-btzeou
branch
from
July 23, 2026 11:22
50971c5 to
bb63034
Compare
…-macro-migration-btzeou Signed-off-by: Claude <noreply@anthropic.com> # Conflicts: # vortex-array/src/arrays/masked/execute.rs # vortex-compressor/src/compressor/cascade.rs
joseph-isaacs
added
the
action/benchmark
github-actions
Bot
removed
the
action/benchmark
joseph-isaacs
added
action/benchmark
Claude
github-actions
Bot
removed
the
action/benchmark
Contributor
Polar Signals Profiling ResultsLatest Run
Previous Runs (1)
Powered by Polar Signals Cloud |
Contributor
Benchmarks: PolarSignals Profiling 📖Vortex (geomean): 1.015x ➖ How to read Verdict and Engines
datafusion / vortex-file-compressed (1.015x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: FineWeb NVMe 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.000x ➖, 1↑ 1↓)
datafusion / vortex-compact (0.999x ➖, 0↑ 1↓)
datafusion / parquet (0.987x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (0.990x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.002x ➖, 0↑ 0↓)
duckdb / parquet (0.999x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=1 on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.043x ➖, 0↑ 2↓)
datafusion / vortex-compact (1.025x ➖, 0↑ 0↓)
datafusion / parquet (1.017x ➖, 1↑ 0↓)
datafusion / arrow (1.012x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed (1.009x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.019x ➖, 0↑ 0↓)
duckdb / parquet (1.011x ➖, 0↑ 2↓)
duckdb / duckdb (1.010x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-DS SF=1 on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.064x ➖, 0↑ 21↓)
datafusion / vortex-compact (1.047x ➖, 0↑ 7↓)
datafusion / parquet (1.054x ➖, 0↑ 16↓)
duckdb / vortex-file-compressed (1.051x ➖, 0↑ 11↓)
duckdb / vortex-compact (1.037x ➖, 0↑ 7↓)
duckdb / parquet (1.028x ➖, 0↑ 1↓)
duckdb / duckdb (1.033x ➖, 1↑ 6↓)
No file size changes detected. |
Contributor
Benchmarks: FineWeb S3 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.993x ➖, 1↑ 2↓)
datafusion / vortex-compact (0.968x ➖, 0↑ 0↓)
datafusion / parquet (1.018x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed (1.112x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.102x ➖, 0↑ 0↓)
duckdb / parquet (1.032x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: Clickbench Sorted on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.012x ➖, 1↑ 2↓)
datafusion / parquet (1.004x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (1.007x ➖, 0↑ 1↓)
duckdb / parquet (0.986x ➖, 1↑ 0↓)
duckdb / duckdb (0.993x ➖, 0↑ 0↓)
File Size Changes (201 files changed, -0.0% overall, 97↑ 104↓)
Totals:
|
Contributor
Benchmarks: Statistical and Population Genetics 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
duckdb / vortex-file-compressed (0.851x ✅, 9↑ 0↓)
duckdb / vortex-compact (0.891x ✅, 7↑ 0↓)
duckdb / parquet (0.905x ➖, 4↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: Random Access 📖Vortex (geomean): 0.952x ➖ How to read Verdict and Engines
unknown / unknown (0.959x ➖, 7↑ 0↓)
|
Contributor
Benchmarks: TPC-H SF=1 on S3 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.035x ➖, 0↑ 1↓)
datafusion / vortex-compact (1.049x ➖, 0↑ 1↓)
datafusion / parquet (0.972x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (1.044x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.043x ➖, 0↑ 0↓)
duckdb / parquet (1.040x ➖, 0↑ 0↓)
|
…-macro-migration-btzeou
Contributor
Benchmarks: TPC-H SF=10 on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.994x ➖, 0↑ 0↓)
datafusion / vortex-compact (0.992x ➖, 0↑ 0↓)
datafusion / parquet (1.000x ➖, 0↑ 0↓)
datafusion / arrow (1.072x ➖, 0↑ 8↓)
duckdb / vortex-file-compressed (0.999x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.002x ➖, 0↑ 0↓)
duckdb / parquet (0.991x ➖, 0↑ 0↓)
duckdb / duckdb (1.026x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: Clickbench on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.020x ➖, 0↑ 0↓)
datafusion / parquet (1.011x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed (0.995x ➖, 2↑ 1↓)
duckdb / parquet (1.005x ➖, 0↑ 0↓)
duckdb / duckdb (1.008x ➖, 0↑ 1↓)
File Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
Contributor
Benchmarks: Appian on NVME 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.072x ➖, 0↑ 3↓)
datafusion / parquet (1.074x ➖, 0↑ 4↓)
duckdb / vortex-file-compressed (1.002x ➖, 0↑ 0↓)
duckdb / parquet (1.000x ➖, 0↑ 0↓)
duckdb / duckdb (1.000x ➖, 0↑ 0↓)
File Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
Contributor
Benchmarks: TPC-H SF=10 on S3 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.979x ➖, 0↑ 0↓)
datafusion / vortex-compact (0.932x ➖, 0↑ 0↓)
datafusion / parquet (1.051x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (0.971x ➖, 0↑ 0↓)
duckdb / vortex-compact (0.985x ➖, 0↑ 0↓)
duckdb / parquet (1.035x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: Compression 📖Vortex (geomean): 0.999x ➖ How to read Verdict and Engines
unknown / unknown (0.994x ➖, 0↑ 0↓)
|
joseph-isaacs
marked this pull request as ready for review
July 24, 2026 12:31
Contributor
Author
|
@codex can you review this pr, lookinng for any behaviour changes other than using auto gen slots |
|
Codex Review: Didn't find any major issues. Keep them coming! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
joseph-isaacs
added
the
action/benchmark
github-actions
Bot
removed
the
action/benchmark
robert3005
approved these changes
Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
This PR introduces a new
#[array_slots]procedural macro that provides type-safe, ergonomic access to array child slots. Previously, array implementations used manual constant definitions (e.g.,VALIDITY_SLOT = 0,METADATA_SLOT = 1) and error-prone index-based slot access viaslots[idx]. This approach was repetitive, error-prone, and lacked compile-time safety.What changes are included in this PR?
ParquetVariantArray(parquet-variant encoding)FSSTArray(fsst encoding)ALPRDArray(alp encoding)ListArray,ListViewArray,FixedSizeListArray(core arrays)VarBinArray,VarBinViewArray(core arrays)BoolArray,DecimalArray,PrimitiveArray(core arrays)VariantArray(core array)RLEArray,DeltaArray,FoRArray(fastlanes encodings)RunEndArray(runend encoding)ZigZagArray(zigzag encoding)ZstdArray(zstd encoding)PcoArray(pco encoding)DecimalBytePartsArray(decimal-byte-parts encoding)