Skip to content

feat(table): variant predicate pushdown and reject variant partition source - #1607

Open
nssalian wants to merge 2 commits into
apache:mainfrom
nssalian:variant-predicate-pushdown
Open

feat(table): variant predicate pushdown and reject variant partition source#1607
nssalian wants to merge 2 commits into
apache:mainfrom
nssalian:variant-predicate-pushdown

Conversation

@nssalian

Copy link
Copy Markdown
Contributor

Adds predicate pushdown for shredded variant fields and rejects Variant as a partition
source. Closes #1016.

Changes

  1. Extract expressions - extract(variant, path, type) referencing a field inside a variant column. RFC-9535 path parse (variant_path.go), bind against the shredded schema (variant_extract.go), cast to the target type (variant_cast.go). Follows Java Expressions.extract / PathUtil.parse / VariantExpressionUtil.castTo.

  2. File pruning - inclusive/strict metrics evaluators use a shredded field's stored bounds for an extract term (table/evaluators.go); the strict path never prunes on extract. Bounds are split via arrow-go Metadata.SizeBytes()
    (table/internal/variant_bounds.go). Follows Java InclusiveMetricsEvaluator /StrictMetricsEvaluator.

  3. Residual filtering (columnar) - surviving extract terms become synthetic derived columns evaluated through the existing arrow-go compute record-filter path, not row-by-row (scanTranslator in visitors.go, table/variant_residual.go,table/arrow_scanner.go).

  4. Reject Variant partition source - addSpecFieldInternal (partitions.go) rejects a Variant source for any transform. Builds on #1206 (which added Variant to IdentityTransform's deny-list); this path does not consult CanTransform, so it closes the gap for sources added via AddPartitionFieldBySourceID.

Testing

  • Unit tests for path parse, cast, extract binding, residual eval, metrics pruning, and partition rejection; end-to-end shredded-write + pushdown test (table/variant_shredded_write_test.go).

@nssalian
nssalian marked this pull request as ready for review July 29, 2026 23:05
@nssalian
nssalian requested a review from zeroshade as a code owner July 29, 2026 23:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(table): predicate evaluation and partition-transform behavior on variant fields

1 participant