feat(vortex-file): gate the file writer behind editions - #8877
Closed
joseph-isaacs wants to merge 12 commits into
Closed
feat(vortex-file): gate the file writer behind editions#8877joseph-isaacs wants to merge 12 commits into
joseph-isaacs wants to merge 12 commits into
Conversation
joseph-isaacs
marked this pull request as draft
July 21, 2026 12:41
joseph-isaacs
force-pushed
the
claude/writer-editions-gating-8qy1fv
branch
from
July 21, 2026 13:54
bf7ca36 to
2da25df
Compare
Contributor
Polar Signals Profiling ResultsLatest Run
Previous Runs (6)
Powered by Polar Signals Cloud |
Contributor
Benchmarks: Vortex queries 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.150x ✅, 2↑ 0↓)
datafusion / parquet (0.160x ✅, 2↑ 0↓)
duckdb / vortex-file-compressed (0.167x ✅, 2↑ 0↓)
duckdb / parquet (0.141x ✅, 2↑ 0↓)
File Size Changes (1 files changed, -90.0% overall, 0↑ 1↓)
Totals:
|
…at layout to allow composing encoding validation with different layout strategies This will be used to adding edition encoding filtering Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
joseph-isaacs
force-pushed
the
claude/writer-editions-gating-8qy1fv
branch
from
July 22, 2026 10:16
c84896c to
58776e3
Compare
…ions-gating-8qy1fv Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> # Conflicts: # vortex-file/src/strategy.rs
joseph-isaacs
added
the
action/benchmark
github-actions
Bot
removed
the
action/benchmark
Contributor
Benchmarks: PolarSignals Profiling 📖Vortex (geomean): 0.895x ✅ How to read Verdict and Engines
datafusion / vortex-file-compressed (0.895x ✅, 5↑ 0↓)
File Size Changes (1 files changed, +0.0% overall, 1↑ 0↓)
Totals:
|
Contributor
Benchmarks: TPC-H SF=1 on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.983x ➖, 0↑ 0↓)
datafusion / vortex-compact (0.993x ➖, 0↑ 0↓)
datafusion / parquet (1.026x ➖, 0↑ 4↓)
datafusion / arrow (1.000x ➖, 1↑ 1↓)
duckdb / vortex-file-compressed (0.999x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.042x ➖, 0↑ 3↓)
duckdb / parquet (0.999x ➖, 2↑ 2↓)
duckdb / duckdb (1.039x ➖, 0↑ 1↓)
File Size Changes (18 files changed, +0.0% overall, 18↑ 0↓)
Totals:
|
Contributor
Benchmarks: FineWeb NVMe 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.027x ➖, 1↑ 1↓)
datafusion / vortex-compact (1.070x ➖, 0↑ 2↓)
datafusion / parquet (1.046x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (0.975x ➖, 1↑ 0↓)
duckdb / vortex-compact (0.985x ➖, 1↑ 0↓)
duckdb / parquet (1.059x ➖, 0↑ 3↓)
File Size Changes (2 files changed, +0.0% overall, 2↑ 0↓)
Totals:
|
Contributor
Benchmarks: TPC-DS SF=1 on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.079x ➖, 1↑ 38↓)
datafusion / vortex-compact (1.069x ➖, 0↑ 25↓)
datafusion / parquet (1.074x ➖, 1↑ 34↓)
duckdb / vortex-file-compressed (1.074x ➖, 0↑ 34↓)
duckdb / vortex-compact (1.059x ➖, 1↑ 21↓)
duckdb / parquet (1.046x ➖, 0↑ 10↓)
duckdb / duckdb (1.066x ➖, 0↑ 19↓)
File Size Changes (48 files changed, +0.0% overall, 48↑ 0↓)
Totals:
|
Contributor
Benchmarks: FineWeb S3 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.098x ➖, 0↑ 1↓)
datafusion / vortex-compact (0.947x ➖, 1↑ 0↓)
datafusion / parquet (0.985x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (1.067x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.029x ➖, 0↑ 0↓)
duckdb / parquet (1.009x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: Statistical and Population Genetics 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
duckdb / vortex-file-compressed (0.964x ➖, 0↑ 0↓)
duckdb / vortex-compact (0.985x ➖, 0↑ 0↓)
duckdb / parquet (0.975x ➖, 0↑ 0↓)
File Size Changes (2 files changed, +0.0% overall, 2↑ 0↓)
Totals:
|
Contributor
Benchmarks: Clickbench Sorted on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.093x ➖, 0↑ 5↓)
datafusion / parquet (0.988x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed (0.986x ➖, 0↑ 0↓)
duckdb / parquet (1.023x ➖, 0↑ 1↓)
duckdb / duckdb (0.982x ➖, 0↑ 0↓)
File Size Changes (201 files changed, +0.1% overall, 114↑ 87↓)
Totals:
|
Contributor
Benchmarks: TPC-H SF=10 on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.999x ➖, 0↑ 0↓)
datafusion / vortex-compact (0.994x ➖, 0↑ 0↓)
datafusion / parquet (0.992x ➖, 0↑ 0↓)
datafusion / arrow (1.010x ➖, 0↑ 2↓)
duckdb / vortex-file-compressed (1.001x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.000x ➖, 0↑ 0↓)
duckdb / parquet (0.997x ➖, 0↑ 1↓)
duckdb / duckdb (1.026x ➖, 0↑ 0↓)
File Size Changes (48 files changed, +0.0% overall, 48↑ 0↓)
Totals:
|
Contributor
Benchmarks: Clickbench on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.858x ✅, 32↑ 0↓)
datafusion / parquet (0.862x ✅, 36↑ 0↓)
duckdb / vortex-file-compressed (0.868x ✅, 29↑ 0↓)
duckdb / parquet (0.936x ➖, 6↑ 0↓)
duckdb / duckdb (0.933x ➖, 9↑ 0↓)
File Size Changes (201 files changed, -0.0% overall, 200↑ 1↓)
Totals:
|
Contributor
Benchmarks: TPC-H SF=1 on S3 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.966x ➖, 0↑ 0↓)
datafusion / vortex-compact (1.002x ➖, 0↑ 1↓)
datafusion / parquet (1.023x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed (0.955x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.017x ➖, 0↑ 1↓)
duckdb / parquet (1.021x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: Random Access 📖Vortex (geomean): 1.039x ➖ How to read Verdict and Engines
unknown / unknown (1.030x ➖, 1↑ 6↓)
|
Contributor
Benchmarks: Appian on NVME 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.979x ➖, 1↑ 0↓)
datafusion / parquet (0.918x ➖, 3↑ 0↓)
duckdb / vortex-file-compressed (0.974x ➖, 1↑ 0↓)
duckdb / parquet (0.985x ➖, 0↑ 0↓)
duckdb / duckdb (0.990x ➖, 0↑ 0↓)
File Size Changes (19 files changed, -0.0% overall, 18↑ 1↓)
Totals:
|
Contributor
Benchmarks: TPC-H SF=10 on S3 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.994x ➖, 0↑ 0↓)
datafusion / vortex-compact (1.042x ➖, 0↑ 1↓)
datafusion / parquet (1.007x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed (0.972x ➖, 0↑ 0↓)
duckdb / vortex-compact (0.994x ➖, 0↑ 0↓)
duckdb / parquet (1.017x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: Compression 📖Vortex (geomean): 0.986x ➖ How to read Verdict and Engines
unknown / unknown (0.980x ➖, 7↑ 0↓)
|
joseph-isaacs
force-pushed
the
claude/writer-editions-gating-8qy1fv
branch
from
July 23, 2026 10:21
c85b1c3 to
a388e2e
Compare
…ions-gating-8qy1fv Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> # Conflicts: # vortex-file/src/writer.rs
Merging this PR will degrade performance by 0.17%
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing 
Footnotes
|
…nks, Python goldens The develop merge brought in the file-level metadata tests (#8740), which write with bare sessions that enable no editions and now fail validation. Enable the crate's test edition in those sessions, matching the other vortex-file unit tests. Also re-qualify the vortex::editions module-doc links (module docs resolve at the lib.rs `pub mod` site) and update the Python doctest golden file sizes for the edition-derived footer context. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CK2nCuXnyd2g3mNHQC8Lz2
The writer now rejects encodings outside the session's enabled editions regardless of the configured strategy, so vortex-jni's strategy-level allow-set override for `vortex.parquet.variant` no longer worked. JNI sessions instead enable the latest unstable edition (the family that encoding belongs to), which restores variant writability through the session policy. The schema-conditional strategy override is removed: parquet-variant arrays only arise for variant-typed columns, so the session-wide policy is equivalent. Sibling unstable encodings are not registered in the JNI build, so the footer's pre-populated context may advertise enabled-but-unregistered ids; a new integration test pins that such files stay readable. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CK2nCuXnyd2g3mNHQC8Lz2
joseph-isaacs
force-pushed
the
claude/writer-editions-gating-8qy1fv
branch
from
July 23, 2026 11:44
c5aff5c to
a5e0c3d
Compare
joseph-isaacs
added
the
action/benchmark
github-actions
Bot
removed
the
action/benchmark
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Follow-up to the editions implementation (#8871): the edition declarations existed but nothing enforced them. This PR gates the file writer behind editions. The session is the source of truth: it stores the set of enabled editions alongside the declarations, and exposes the resolved set of all enabled encodings. The default session enables
core2026.07.0(EditionId::new("core", 2026, 7, 0)), so by default:What changes are included in this PR?
vortex-edition:EditionSessionnow stores the enabled editions next to the declarations:enable()(must be declared; at most one edition per family),enabled(), andenabled_encodings()— the resolved, sorted union of the enabled editions' encoding sets (Nonewhen nothing is enabled). The first-party declarations stay in thevortexfacade; this crate remains types + session variable + test harness.vortex(facade):register_default_editionsnow also enablesDEFAULT_WRITE_EDITIONS(core2026.07.0, plus the newestunstabledraft when theunstable_encodingsfeature opts in) on the session. New tests: a golden pin of the frozencore2026.07.0encoding set, per-edition harness validation, subset properties, and a check that every default compression scheme's produced encodings are covered by the default write editions.vortex-compressor: new requiredScheme::produced_encodings(&self) -> Vec<ArrayId>trait method declaring the array encodings a scheme may emit (empty = only canonical arrays; cascaded children are covered by their own schemes).CascadingCompressor::with_enabled_editions_gatingmakes scheme selection resolve the session's enabled encodings from the execution context at compression time and skip schemes whose outputs are not all allowed; sessions with no enabled editions are not gated. Unit-tested with a synthetic edition/scheme.vortex-btrblocks: every scheme declares its produced encodings (including the env-gatedPatchedoutput of BitPacking/ALP).BtrBlocksCompressorBuildergainsgate_by_enabled_editions()(session gating pass-through) andretain_allowed_encodings()(static filtering for explicit allow sets).vortex-layout:FlatLayoutStrategy::with_enabled_editions_gatingresolves the allowed encodings from the session's enabled editions atwrite_streamtime — an encoding outside them fails the write via the existing normalize-with-Errorvalidation — falling back to the explicitallowed_encodingsset when the session enables no editions.vortex-file: the defaultWriteStrategyBuilderwires all of this up: flat writer gated by enabled editions with the staticALLOWED_ENCODINGSregistry as fallback, and both the data and stats compressors gated at compression time. An explicitwith_allow_encodingsset overrides the session gate everywhere (unchanged behavior forvortex-jni/vortex-parquet-variant). The writer's deterministic pre-populatedArrayContextlikewise comes from the session's enabled encodings. Experimental patches remain a runtime opt-out:Patchedis only writable when the experimental environment variable is set, mirroring when compression produces it.Checks run (after rebasing onto
developat #8871):cargo nextest run -p vortex-edition -p vortex-compressor -p vortex-btrblocks -p vortex-layout -p vortex-file -p vortex(411 passed; also with--all-featuresfor file/btrblocks/compressor),cargo test --docfor the touched crates,cargo +nightly fmt --all, andcargo clippy --all-targets --all-featuresfor the touched crates plusvortex-tensor,vortex-jni, andvortex-parquet-variant(clean). Workspace-wide clippy could not be run:vortex-duckdb's build script cannot download DuckDB sources in this sandbox.What APIs are changed? Are there any user-facing changes?
Schemehas a new required methodproduced_encodings() -> Vec<ArrayId>— a breaking change for third-party schemes, which must now declare their outputs.EditionSession::{enable, enabled, enabled_encodings},CascadingCompressor::with_enabled_editions_gating,BtrBlocksCompressorBuilder::{gate_by_enabled_editions, retain_allowed_encodings},FlatLayoutStrategy::with_enabled_editions_gating, andvortex::editions::DEFAULT_WRITE_EDITIONSare new.vortexfacade: encodings outside the enabled editions (fastlanes.delta,vortex.onpair,vortex.zstd_bufferswithout theunstable_encodingsfeature,vortex.patchedwithout the env opt-in) are no longer writable — writes containing them fail at the flat-layout validation instead of producing files without the read-forever guarantee. Sessions that enable no editions keep the previous staticALLOWED_ENCODINGSbehavior.🤖 Generated with Claude Code
https://claude.ai/code/session_01CK2nCuXnyd2g3mNHQC8Lz2