Skip to content

Add fixed-size binary support - #8865

Draft
connortsui20 wants to merge 6 commits into
ct/fixed-width-primitive-decimalfrom
ct/fixed-width-encoding
Draft

Add fixed-size binary support#8865
connortsui20 wants to merge 6 commits into
ct/fixed-width-primitive-decimalfrom
ct/fixed-width-encoding

Conversation

@connortsui20

@connortsui20 connortsui20 commented Jul 20, 2026

Copy link
Copy Markdown
Member

Stacked on #8864.

Why

FixedSizeBinary has the same one-buffer fixed-width structure, so it should inherit the shared structural compute rather than introduce a third take/filter implementation. It remains a distinct logical type and stores an explicit row count because zero-width values cannot derive length from the values buffer.

What

  • add dtype, scalar, canonical array, builder, serialization, Arrow, Python, FFI, CUDA, file, fuzz, and conformance support
  • implement the same three-hook FixedWidthArray adapter used by Primitive and Decimal; FSB has no local take or filter pipeline
  • reuse byte-compress filtering through 4 bytes, typed filtering through 32 bytes, and the common runtime-width fallback
  • reuse AVX2 takes for 4- and 8-byte records and common scalar take paths for other widths
  • validate physical buffer lengths and preserve zero-width row counts through Vortex and Arrow

Issue provenance

  • Existing and fixed: compact Serde decoding had stale numeric positions for Extension and Union.
  • Introduced during development and fixed: compact-tag placement, missing ArrayParts validation, zero-width Arrow handling, and full-source cloning for uncommon-width filters.

Validation

  • cargo test -p vortex-array --lib (3,064 passed, 1 ignored)
  • cargo clippy -p vortex-array --all-targets --all-features
  • cargo test -p vortex-arrow fixed_size_binary
  • cargo test -p vortex-file test_round_trip_many_types
  • cargo test --no-run --target x86_64-apple-darwin -p vortex-array --lib
  • git diff --check

@codspeed-hq

codspeed-hq Bot commented Jul 20, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 1884 untouched benchmarks
⏩ 12 skipped benchmarks1


Comparing ct/fixed-width-encoding (3935ac9) with ct/fixed-width-primitive-decimal (dfc6a40)

Open in CodSpeed

Footnotes

  1. 12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@connortsui20
connortsui20 force-pushed the ct/fixed-width-encoding branch from dde9039 to e13e63c Compare July 20, 2026 19:26
@connortsui20
connortsui20 force-pushed the ct/fixed-width-encoding branch from e13e63c to 5dd799a Compare July 20, 2026 19:50
@github-actions

github-actions Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Polar Signals Profiling Results

Latest Run

Status Commit Job Attempt Link
🟢 Done 460d6dc 1 Explore Profiling Data
Previous Runs (9)
Status Commit Job Attempt Link
🟢 Done 935e86c 1 Explore Profiling Data
🟢 Done 11a4e4f 1 Explore Profiling Data
🟢 Done d36c772 1 Explore Profiling Data
🟢 Done c924a06 1 Explore Profiling Data
🟢 Done 97c8e02 1 Explore Profiling Data
🟢 Done 0592a4f 1 Explore Profiling Data
🟢 Done 8dedc68 1 Explore Profiling Data
🟢 Done 8ef7bb2 1 Explore Profiling Data
🟢 Done 5dd799a 1 Explore Profiling Data

Powered by Polar Signals Cloud

@github-actions

github-actions Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Benchmarks: Vortex queries 📖

Verdict: No clear signal (low confidence)
Attributed Vortex impact: +2.0%
Engines: DataFusion No clear signal (+6.1%, low confidence) · DuckDB No clear signal (-1.9%, low confidence)
Vortex (geomean): 1.026x ➖
Parquet (geomean): 1.016x ➖
Shifts: Parquet (control) +1.6% · Median polish +1.6%

How to read Verdict and Engines
  • Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
  • Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
  • Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

datafusion / vortex-file-compressed (1.085x ➖, 0↑ 1↓)
name PR 460d6dc (ns) base 57962f4 (ns) ratio (PR/base)
vortex_q00/datafusion:vortex-file-compressed 8648179 8197567 1.05
vortex_q01/datafusion:vortex-file-compressed 🚨 5452852 4885078 1.12
datafusion / parquet (1.023x ➖, 0↑ 0↓)
name PR 460d6dc (ns) base 57962f4 (ns) ratio (PR/base)
vortex_q00/datafusion:parquet 20848141 20638806 1.01
vortex_q01/datafusion:parquet 4744100 4580085 1.04
duckdb / vortex-file-compressed (0.989x ➖, 0↑ 0↓)
name PR 460d6dc (ns) base 57962f4 (ns) ratio (PR/base)
vortex_q00/duckdb:vortex-file-compressed 10192401 10061599 1.01
vortex_q01/duckdb:vortex-file-compressed 5960426 6167184 0.97
duckdb / parquet (1.009x ➖, 0↑ 0↓)
name PR 460d6dc (ns) base 57962f4 (ns) ratio (PR/base)
vortex_q00/duckdb:parquet 23424626 23195009 1.01
vortex_q01/duckdb:parquet 9413805 9340266 1.01

File Size Changes (1 files changed, +0.0% overall, 1↑ 0↓)
File Scale Format Base HEAD Change %
test.vortex 1.0 vortex-file-compressed 80.59 MB 80.59 MB +48 B +0.0%

Totals:

  • vortex-file-compressed: 80.85 MB → 80.85 MB (+0.0%)

@connortsui20
connortsui20 force-pushed the ct/fixed-width-encoding branch from 5dd799a to 8dedc68 Compare July 20, 2026 21:35
@connortsui20
connortsui20 force-pushed the ct/fixed-width-encoding branch from 8dedc68 to 8ef7bb2 Compare July 20, 2026 22:05
@connortsui20
connortsui20 force-pushed the ct/fixed-width-encoding branch from 8ef7bb2 to 8dedc68 Compare July 20, 2026 22:20
@codecov

codecov Bot commented Jul 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 64.16235% with 415 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.17%. Comparing base (fa4beb4) to head (cc89366).

Files with missing lines Patch % Lines
...rtex-array/src/arrays/fixed_size_binary/compute.rs 30.28% 99 Missing ⚠️
vortex-array/src/arrays/fixed_size_binary/mod.rs 77.05% 53 Missing ⚠️
vortex-array/src/builders/fixed_size_binary.rs 54.16% 44 Missing ⚠️
encodings/sparse/src/canonical.rs 0.00% 41 Missing ⚠️
...rtex-array/src/arrays/constant/vtable/canonical.rs 0.00% 17 Missing ⚠️
...x-array/src/scalar_fn/fns/binary/compare/nested.rs 0.00% 17 Missing ⚠️
vortex-array/src/arrays/dict/execute.rs 0.00% 14 Missing ⚠️
vortex-datafusion/src/convert/scalars.rs 33.33% 14 Missing ⚠️
vortex-array/src/dtype/serde/serde.rs 53.57% 13 Missing ⚠️
.../aggregate_fn/fns/is_constant/fixed_size_binary.rs 0.00% 12 Missing ⚠️
... and 26 more

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@connortsui20
connortsui20 force-pushed the ct/fixed-width-encoding branch from 8dedc68 to 0592a4f Compare July 20, 2026 22:33
@connortsui20
connortsui20 force-pushed the ct/fixed-width-encoding branch from 0592a4f to 97c8e02 Compare July 20, 2026 22:42
@connortsui20
connortsui20 force-pushed the ct/fixed-width-encoding branch from 97c8e02 to c924a06 Compare July 21, 2026 17:26
@connortsui20
connortsui20 force-pushed the ct/fixed-width-encoding branch from c924a06 to d36c772 Compare July 21, 2026 17:30
@connortsui20
connortsui20 force-pushed the ct/fixed-width-encoding branch from d36c772 to 11a4e4f Compare July 22, 2026 18:38
@connortsui20
connortsui20 force-pushed the ct/fixed-width-encoding branch 4 times, most recently from 460d6dc to 14e1fb1 Compare July 23, 2026 14:40
@connortsui20
connortsui20 force-pushed the ct/fixed-width-encoding branch 2 times, most recently from 758c3e0 to 5b1b039 Compare July 23, 2026 15:40
Comment thread vortex-arrow/src/dtype.rs
}
DType::Utf8(_) => DataType::Utf8View,
DType::Binary(_) => DataType::BinaryView,
DType::FixedSizeBinary(byte_width, _) => DataType::FixedSizeBinary(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please no new DType

@connortsui20
connortsui20 force-pushed the ct/fixed-width-encoding branch from 5b1b039 to 1767981 Compare July 23, 2026 16:33
@connortsui20
connortsui20 force-pushed the ct/fixed-width-encoding branch 2 times, most recently from 39ea0b9 to c645586 Compare July 23, 2026 19:15
@connortsui20
connortsui20 force-pushed the ct/fixed-width-encoding branch from c645586 to 53a4c9d Compare July 23, 2026 19:30
@connortsui20
connortsui20 force-pushed the ct/fixed-width-encoding branch from 53a4c9d to cc89366 Compare July 23, 2026 20:15
@connortsui20
connortsui20 force-pushed the ct/fixed-width-encoding branch 2 times, most recently from 3fd82c6 to e364d90 Compare July 27, 2026 18:25
@connortsui20
connortsui20 force-pushed the ct/fixed-width-encoding branch 3 times, most recently from 7dbc50b to 12792b7 Compare July 28, 2026 16:47
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 force-pushed the ct/fixed-width-encoding branch from 12792b7 to 3935ac9 Compare July 30, 2026 19:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants