Exectuion and optimization tracing harness for testing - #7814
Open
AdamGS wants to merge 15 commits into
Open
Conversation
AdamGS
force-pushed
the
adamg/execution-tests
branch
from
May 6, 2026 14:41
1520af8 to
51759b3
Compare
Merging this PR will not alter performance
Comparing 
Footnotes
|
AdamGS
force-pushed
the
adamg/execution-tests
branch
7 times, most recently
from
May 11, 2026 11:28
2376559 to
e9f08fe
Compare
AdamGS
marked this pull request as ready for review
May 11, 2026 11:56
joseph-isaacs
added
the
action/benchmark
github-actions
Bot
removed
the
action/benchmark
Contributor
Polar Signals Profiling ResultsLatest Run
Previous Runs (2)
Powered by Polar Signals Cloud |
Contributor
Benchmarks: PolarSignals ProfilingVortex (geomean): 1.106x ❌ How to read Verdict and Engines
datafusion / vortex-file-compressed (1.106x ❌, 0↑ 5↓)
No file size changes detected. |
Contributor
File Sizes: PolarSignals ProfilingNo file size changes detected. |
Contributor
Benchmarks: FineWeb NVMeVerdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.985x ➖, 0↑ 0↓)
datafusion / vortex-compact (0.992x ➖, 0↑ 0↓)
datafusion / parquet (0.994x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (0.958x ➖, 2↑ 0↓)
duckdb / vortex-compact (0.989x ➖, 0↑ 0↓)
duckdb / parquet (1.003x ➖, 0↑ 0↓)
File Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
Contributor
File Sizes: FineWeb NVMeNo file size changes detected. |
Contributor
Benchmarks: TPC-H SF=1 on NVMEVerdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.082x ➖, 0↑ 4↓)
datafusion / vortex-compact (1.057x ➖, 0↑ 1↓)
datafusion / parquet (1.041x ➖, 0↑ 1↓)
datafusion / arrow (1.109x ❌, 0↑ 10↓)
duckdb / vortex-file-compressed (1.062x ➖, 0↑ 1↓)
duckdb / vortex-compact (1.061x ➖, 0↑ 3↓)
duckdb / parquet (1.032x ➖, 1↑ 2↓)
duckdb / duckdb (1.048x ➖, 0↑ 2↓)
File Size Changes (9 files changed, -0.0% overall, 3↑ 6↓)
Totals:
|
Contributor
File Sizes: TPC-H SF=1 on NVMENo file size changes detected. |
Contributor
Benchmarks: TPC-DS SF=1 on NVMEVerdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.992x ➖, 1↑ 0↓)
datafusion / vortex-compact (0.985x ➖, 2↑ 1↓)
datafusion / parquet (0.987x ➖, 5↑ 0↓)
duckdb / vortex-file-compressed (0.996x ➖, 1↑ 0↓)
duckdb / vortex-compact (0.995x ➖, 1↑ 3↓)
duckdb / parquet (1.004x ➖, 0↑ 1↓)
duckdb / duckdb (0.990x ➖, 2↑ 1↓)
File Size Changes (6 files changed, -0.0% overall, 3↑ 3↓)
Totals:
|
Contributor
File Sizes: TPC-DS SF=1 on NVMENo file size changes detected. |
Contributor
Benchmarks: FineWeb S3Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.167x ➖, 0↑ 2↓)
datafusion / vortex-compact (1.077x ➖, 1↑ 2↓)
datafusion / parquet (1.122x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (1.073x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.129x ➖, 0↑ 1↓)
duckdb / parquet (1.089x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: Statistical and Population GeneticsVerdict: No clear signal (low confidence) How to read Verdict and Engines
duckdb / vortex-file-compressed (1.016x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.021x ➖, 0↑ 0↓)
duckdb / parquet (1.028x ➖, 0↑ 0↓)
File Size Changes (1 files changed, +0.0% overall, 1↑ 0↓)
Totals:
|
Contributor
File Sizes: Statistical and Population GeneticsNo file size changes detected. |
Contributor
Benchmarks: TPC-H SF=10 on NVMEVerdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.093x ➖, 0↑ 9↓)
datafusion / vortex-compact (1.078x ➖, 0↑ 5↓)
datafusion / parquet (1.105x ❌, 0↑ 9↓)
datafusion / arrow (1.133x ❌, 0↑ 14↓)
duckdb / vortex-file-compressed (1.094x ➖, 0↑ 7↓)
duckdb / vortex-compact (1.032x ➖, 0↑ 1↓)
duckdb / parquet (1.029x ➖, 0↑ 1↓)
duckdb / duckdb (1.007x ➖, 0↑ 0↓)
File Size Changes (27 files changed, -0.0% overall, 14↑ 13↓)
Totals:
|
Contributor
File Sizes: TPC-H SF=10 on NVMENo file size changes detected. |
Contributor
|
This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days |
robert3005
pushed a commit
that referenced
this pull request
Jul 30, 2026
…mpressed arrays (#8828) ## Summary Stacked on #7814 (the `trace_op` execution/optimization tracing harness). Builds out trace-test coverage for how take/filter/compare/like reduce and execute over complex arrays, using the harness from the base PR: **vortex-array** (promotes `test_harness/trace.rs` to a directory module `trace/mod.rs` + `trace/tests.rs`; harness source unchanged): - take (`DictArray`) over a chunked array — no reduce rule fires; execution canonicalizes the chunked values chunk-by-chunk before the dict take kernel runs. - filter over a struct with complex children (dict-of-strings + chunked fields) — `FilterStructRule` pushes the filter into each field, `FilterReduceAdaptor(Dict)` rewrites the dict field at optimize time. - compare (`Binary` Eq vs constant) over a dict — `DictionaryScalarFnValuesPushDownRule` reduces the compare into the dictionary values. - like over a dict-of-strings — `LikeReduceAdaptor(Dict)` reduces into the dictionary values. **vortex-runend** (new `trace_tests.rs`, `insta` added as dev-dependency): - compare vs constant reduces via `RunEndScalarFnRule` at optimize time; filter and take execute via the `FilterExecuteAdaptor`/`TakeExecuteAdaptor` parent kernels. **vortex-btrblocks** (new `trace_tests.rs`; `insta`, `tpchgen`, `tpchgen-arrow`, `arrow-array` added as dev-dependencies): generates TPC-H lineitem (SF 0.001, 4096 rows, deterministic — tpchgen and the compressor's sampling seed are both fixed), compresses it with `BtrBlocksCompressor`, and traces TPC-H-style scan predicates over the resulting encodings: - `l_shipdate >= const` over `ext(date) → for → bitpacked` — extension compare kernel at execution time, with `CastReduceAdaptor(FoR)/(BitPacked)` reductions inside the arrow fallback. - `l_quantity < const` over `decimal_byte_parts → dict → bitpacked` — the decimal_byte_parts compare kernel pushes into the byte-parts dictionary, where `DictionaryScalarFnValuesPushDownRule` reduces the compare to the 50 dictionary values before decode. - `l_shipmode = const` over dict-of-FSST — dict pushdown at optimize time, FSST compare kernel + dict decode at execution time. - `l_comment LIKE` over FSST — the FSST like kernel matches in compressed space. - filter over a struct of compressed columns — `FilterStructRule` plus per-encoding pushdown via `DecimalBytePartsFilterPushDownRule`, `ExtensionFilterPushDownRule`, `FoRFilterPushDownRule`, and `FilterReduceAdaptor(Dict)`, leaving execution a no-op. - take over a struct of compressed columns — `TakeReduceAdaptor(Struct)` absorbs the take entirely at optimize time. Each btrblocks test also asserts the canonical result matches running the same operation over the uncompressed column. A 1MiB TPC-H text pool replaces the spec-default 300MiB one, cutting per-test setup from ~9s to ~0.5s without changing the traces. ## Testing This PR is tests-only; no production code changes. - `cargo nextest run -p vortex-array` — 2961 passed (includes the 8 trace tests, run in parallel to confirm the thread-local recorder is deterministic). - `cargo nextest run -p vortex-runend` — 61 passed. - `cargo nextest run -p vortex-btrblocks` — 43 passed. - `cargo clippy --all-targets` on the three touched crates — clean. - `cargo +nightly fmt --all` and `git diff --check` — clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01RMBcCiwpC3dbGLXRbK7UuR --- _Generated by [Claude Code](https://claude.ai/code/session_01RMBcCiwpC3dbGLXRbK7UuR)_ --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
robert3005
pushed a commit
that referenced
this pull request
Jul 30, 2026
…mpressed arrays (#8828) Stacked on #7814 (the `trace_op` execution/optimization tracing harness). Builds out trace-test coverage for how take/filter/compare/like reduce and execute over complex arrays, using the harness from the base PR: **vortex-array** (promotes `test_harness/trace.rs` to a directory module `trace/mod.rs` + `trace/tests.rs`; harness source unchanged): - take (`DictArray`) over a chunked array — no reduce rule fires; execution canonicalizes the chunked values chunk-by-chunk before the dict take kernel runs. - filter over a struct with complex children (dict-of-strings + chunked fields) — `FilterStructRule` pushes the filter into each field, `FilterReduceAdaptor(Dict)` rewrites the dict field at optimize time. - compare (`Binary` Eq vs constant) over a dict — `DictionaryScalarFnValuesPushDownRule` reduces the compare into the dictionary values. - like over a dict-of-strings — `LikeReduceAdaptor(Dict)` reduces into the dictionary values. **vortex-runend** (new `trace_tests.rs`, `insta` added as dev-dependency): - compare vs constant reduces via `RunEndScalarFnRule` at optimize time; filter and take execute via the `FilterExecuteAdaptor`/`TakeExecuteAdaptor` parent kernels. **vortex-btrblocks** (new `trace_tests.rs`; `insta`, `tpchgen`, `tpchgen-arrow`, `arrow-array` added as dev-dependencies): generates TPC-H lineitem (SF 0.001, 4096 rows, deterministic — tpchgen and the compressor's sampling seed are both fixed), compresses it with `BtrBlocksCompressor`, and traces TPC-H-style scan predicates over the resulting encodings: - `l_shipdate >= const` over `ext(date) → for → bitpacked` — extension compare kernel at execution time, with `CastReduceAdaptor(FoR)/(BitPacked)` reductions inside the arrow fallback. - `l_quantity < const` over `decimal_byte_parts → dict → bitpacked` — the decimal_byte_parts compare kernel pushes into the byte-parts dictionary, where `DictionaryScalarFnValuesPushDownRule` reduces the compare to the 50 dictionary values before decode. - `l_shipmode = const` over dict-of-FSST — dict pushdown at optimize time, FSST compare kernel + dict decode at execution time. - `l_comment LIKE` over FSST — the FSST like kernel matches in compressed space. - filter over a struct of compressed columns — `FilterStructRule` plus per-encoding pushdown via `DecimalBytePartsFilterPushDownRule`, `ExtensionFilterPushDownRule`, `FoRFilterPushDownRule`, and `FilterReduceAdaptor(Dict)`, leaving execution a no-op. - take over a struct of compressed columns — `TakeReduceAdaptor(Struct)` absorbs the take entirely at optimize time. Each btrblocks test also asserts the canonical result matches running the same operation over the uncompressed column. A 1MiB TPC-H text pool replaces the spec-default 300MiB one, cutting per-test setup from ~9s to ~0.5s without changing the traces. This PR is tests-only; no production code changes. - `cargo nextest run -p vortex-array` — 2961 passed (includes the 8 trace tests, run in parallel to confirm the thread-local recorder is deterministic). - `cargo nextest run -p vortex-runend` — 61 passed. - `cargo nextest run -p vortex-btrblocks` — 43 passed. - `cargo clippy --all-targets` on the three touched crates — clean. - `cargo +nightly fmt --all` and `git diff --check` — clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01RMBcCiwpC3dbGLXRbK7UuR --- _Generated by [Claude Code](https://claude.ai/code/session_01RMBcCiwpC3dbGLXRbK7UuR)_ --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
robert3005
force-pushed
the
adamg/execution-tests
branch
from
July 30, 2026 15:25
541c583 to
4a18b84
Compare
…mpressed arrays (#8828) Stacked on #7814 (the `trace_op` execution/optimization tracing harness). Builds out trace-test coverage for how take/filter/compare/like reduce and execute over complex arrays, using the harness from the base PR: **vortex-array** (promotes `test_harness/trace.rs` to a directory module `trace/mod.rs` + `trace/tests.rs`; harness source unchanged): - take (`DictArray`) over a chunked array — no reduce rule fires; execution canonicalizes the chunked values chunk-by-chunk before the dict take kernel runs. - filter over a struct with complex children (dict-of-strings + chunked fields) — `FilterStructRule` pushes the filter into each field, `FilterReduceAdaptor(Dict)` rewrites the dict field at optimize time. - compare (`Binary` Eq vs constant) over a dict — `DictionaryScalarFnValuesPushDownRule` reduces the compare into the dictionary values. - like over a dict-of-strings — `LikeReduceAdaptor(Dict)` reduces into the dictionary values. **vortex-runend** (new `trace_tests.rs`, `insta` added as dev-dependency): - compare vs constant reduces via `RunEndScalarFnRule` at optimize time; filter and take execute via the `FilterExecuteAdaptor`/`TakeExecuteAdaptor` parent kernels. **vortex-btrblocks** (new `trace_tests.rs`; `insta`, `tpchgen`, `tpchgen-arrow`, `arrow-array` added as dev-dependencies): generates TPC-H lineitem (SF 0.001, 4096 rows, deterministic — tpchgen and the compressor's sampling seed are both fixed), compresses it with `BtrBlocksCompressor`, and traces TPC-H-style scan predicates over the resulting encodings: - `l_shipdate >= const` over `ext(date) → for → bitpacked` — extension compare kernel at execution time, with `CastReduceAdaptor(FoR)/(BitPacked)` reductions inside the arrow fallback. - `l_quantity < const` over `decimal_byte_parts → dict → bitpacked` — the decimal_byte_parts compare kernel pushes into the byte-parts dictionary, where `DictionaryScalarFnValuesPushDownRule` reduces the compare to the 50 dictionary values before decode. - `l_shipmode = const` over dict-of-FSST — dict pushdown at optimize time, FSST compare kernel + dict decode at execution time. - `l_comment LIKE` over FSST — the FSST like kernel matches in compressed space. - filter over a struct of compressed columns — `FilterStructRule` plus per-encoding pushdown via `DecimalBytePartsFilterPushDownRule`, `ExtensionFilterPushDownRule`, `FoRFilterPushDownRule`, and `FilterReduceAdaptor(Dict)`, leaving execution a no-op. - take over a struct of compressed columns — `TakeReduceAdaptor(Struct)` absorbs the take entirely at optimize time. Each btrblocks test also asserts the canonical result matches running the same operation over the uncompressed column. A 1MiB TPC-H text pool replaces the spec-default 300MiB one, cutting per-test setup from ~9s to ~0.5s without changing the traces. This PR is tests-only; no production code changes. - `cargo nextest run -p vortex-array` — 2961 passed (includes the 8 trace tests, run in parallel to confirm the thread-local recorder is deterministic). - `cargo nextest run -p vortex-runend` — 61 passed. - `cargo nextest run -p vortex-btrblocks` — 43 passed. - `cargo clippy --all-targets` on the three touched crates — clean. - `cargo +nightly fmt --all` and `git diff --check` — clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01RMBcCiwpC3dbGLXRbK7UuR --- _Generated by [Claude Code](https://claude.ai/code/session_01RMBcCiwpC3dbGLXRbK7UuR)_ --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
robert3005
force-pushed
the
adamg/execution-tests
branch
from
July 30, 2026 21:24
4a18b84 to
8ead1e8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces an internal tracing API that enables testing execution and optimization flows with snapshot testing! Just wrap your code into a callaback and call
trace_op, and you get back both the result and a snapshot you can assert.For example:
The API for tests is pretty nice, internally there are still improvements that can be made here, but this is a good step towards building more confidence and making the execution model clearer.