Core: Read content stats from v4 Manifest - #6
Closed
nastra wants to merge 41 commits into
Closed
Conversation
…6770) Co-authored-by: Vova Kolmakov <wombatukun@apache.org> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bumps [actions/checkout](https://github.com/actions/checkout) from 7.0.0 to 7.0.1. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@9c091bb...3d3c42e) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 7.0.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 6.3.0 to 7.0.0. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@ece7cb0...5fda3b9) --- updated-dependencies: - dependency-name: actions/setup-python dependency-version: 7.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/labeler](https://github.com/actions/labeler) from 6.2.0 to 7.0.0. - [Release notes](https://github.com/actions/labeler/releases) - [Commits](actions/labeler@b8dd2d9...bf12e9b) --- updated-dependencies: - dependency-name: actions/labeler dependency-version: 7.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps the codeql-action group with 3 updates: [github/codeql-action/init](https://github.com/github/codeql-action), [github/codeql-action/analyze](https://github.com/github/codeql-action) and [github/codeql-action/upload-sarif](https://github.com/github/codeql-action). Updates `github/codeql-action/init` from 4.37.0 to 4.37.1 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@99df26d...7188fc3) Updates `github/codeql-action/analyze` from 4.37.0 to 4.37.1 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@99df26d...7188fc3) Updates `github/codeql-action/upload-sarif` from 4.37.0 to 4.37.1 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@99df26d...7188fc3) --- updated-dependencies: - dependency-name: github/codeql-action/init dependency-version: 4.37.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: codeql-action - dependency-name: github/codeql-action/analyze dependency-version: 4.37.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: codeql-action - dependency-name: github/codeql-action/upload-sarif dependency-version: 4.37.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: codeql-action ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps `nessie` from 0.108.1 to 0.108.2. Updates `org.projectnessie.nessie:nessie-client` from 0.108.1 to 0.108.2 - [Release notes](https://github.com/projectnessie/nessie/releases) - [Changelog](https://github.com/projectnessie/nessie/blob/main/CHANGELOG.md) - [Commits](projectnessie/nessie@nessie-0.108.1...nessie-0.108.2) Updates `org.projectnessie.nessie:nessie-jaxrs-testextension` from 0.108.1 to 0.108.2 - [Release notes](https://github.com/projectnessie/nessie/releases) - [Changelog](https://github.com/projectnessie/nessie/blob/main/CHANGELOG.md) - [Commits](projectnessie/nessie@nessie-0.108.1...nessie-0.108.2) Updates `org.projectnessie.nessie:nessie-versioned-storage-inmemory-tests` from 0.108.1 to 0.108.2 - [Release notes](https://github.com/projectnessie/nessie/releases) - [Changelog](https://github.com/projectnessie/nessie/blob/main/CHANGELOG.md) - [Commits](projectnessie/nessie@nessie-0.108.1...nessie-0.108.2) Updates `org.projectnessie.nessie:nessie-versioned-storage-testextension` from 0.108.1 to 0.108.2 - [Release notes](https://github.com/projectnessie/nessie/releases) - [Changelog](https://github.com/projectnessie/nessie/blob/main/CHANGELOG.md) - [Commits](projectnessie/nessie@nessie-0.108.1...nessie-0.108.2) --- updated-dependencies: - dependency-name: org.projectnessie.nessie:nessie-client dependency-version: 0.108.2 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: org.projectnessie.nessie:nessie-jaxrs-testextension dependency-version: 0.108.2 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: org.projectnessie.nessie:nessie-versioned-storage-inmemory-tests dependency-version: 0.108.2 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: org.projectnessie.nessie:nessie-versioned-storage-testextension dependency-version: 0.108.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…pache#17368) Bumps [com.google.cloud:libraries-bom](https://github.com/googleapis/google-cloud-java) from 26.85.0 to 26.85.1. - [Release notes](https://github.com/googleapis/google-cloud-java/releases) - [Changelog](https://github.com/googleapis/google-cloud-java/blob/main/CHANGELOG.md) - [Commits](googleapis/google-cloud-java@libraries-bom/v26.85.0...libraries-bom/v26.85.1) --- updated-dependencies: - dependency-name: com.google.cloud:libraries-bom dependency-version: 26.85.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Uros Bojanic <uros.bojanic@databricks.com>
The `triage` job started failing on all PRs with:
YAMLException: deficient indentation (24:7)
actions/labeler v7.0.0 (apache#17371) upgraded its bundled js-yaml from 4.x to
5.x, whose stricter YAML 1.2 parser rejects the multi-line flow sequences
(`[ ... ]`) in this file, where the list entries were indented to the same
column as their parent key rather than deeper.
Convert the flow sequences to block sequences, which parse cleanly under
the new parser. This is a pure syntax change: the resulting configuration
is byte-for-byte equivalent (same labels, same globs). Also drops a stray
trailing comma in the HIVE block.
…e#17379) Co-authored-by: Isaac Co-authored-by: Uros Bojanic <uros.bojanic@databricks.com>
* [docs] Update write.metadata.path in view configuration * Update docs/docs/view-configuration.md Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com> --------- Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com>
* Update labeler.yml Have failures on my integration iceberg PR, see https://github.com/Teradata/iceberg/actions/runs/14752592051/job/41413291668?pr=4 Found resolution here, googleapis/release-please-action#1105 (comment) * Revert "Update labeler.yml" * add Teradata to vendor.md * remove trailing space --------- Co-authored-by: jenbaldwin <jenbaldwin3@yahoo.com> Co-authored-by: lk255018 <lk255018@teradata.com> Co-authored-by: Jennifer Baldwin <jennifer.baldwin@teradata.com>
Co-authored-by: Isaac Co-authored-by: Uros Bojanic <uros.bojanic@databricks.com>
…pache#17138) * OpenAPI: Update REST spec expressions to match new expressions spec Adds UnaryPredicate, ComparisonPredicate, and SetPredicate schemas that use 'child' and 'left'/'right' value expression references, matching the JSON serialization defined in Appendix B of the Iceberg Expressions spec (format/expressions-spec.md). The existing UnaryExpression, LiteralExpression, and SetExpression schemas (which use 'term'/'value'/'values') are marked deprecated but retained in Expression.oneOf for backward compatibility with older clients per the spec's "Backward compatibility" section. A ValueExpression placeholder schema is added and referenced by the new predicates; concrete LITERAL/REFERENCE/APPLY forms will be added in a follow-up change. * OpenAPI: Support bare boolean predicate form in REST expressions Extends Expression.oneOf with a bare `boolean` alternative, matching the spec-conformant PREDICATE production in the Iceberg Expressions spec (format/expressions-spec.md, Appendix B) which defines `PREDICATE: true | false | { ... }`. The pre-existing TrueExpression and FalseExpression object-form schemas (`{"type": "true"}` / `{"type": "false"}`) are marked deprecated and retained in Expression.oneOf for backward compatibility with older clients that send the object form. * OpenAPI: Add ValueExpression schemas per Iceberg Expressions spec Appendix B Replaces the ValueExpression placeholder with concrete oneOf schemas for LITERAL, REFERENCE, and APPLY forms, matching Appendix B of the Iceberg Expressions spec. Also adds the deprecated reference form for backward compatibility with older REST predicates. Schemas added: - Literal (bare scalar, typed literal object with/without data-type) - Literals (bare array or typed object; used by SetPredicate.values) - FieldReference (bound by id or unbound by name) - Apply (function application over ValueExpression or Expression args) - FunctionReference (four forms: name, name list, identifier object, catalog + identifier object) - DeprecatedReference (backward-compat form wired into Term's oneOf) SetPredicate.values now references Literals rather than an array of ValueExpression to match Appendix B's LITERALS grammar. * OpenAPI: Regenerate rest-catalog-open-api.py * OpenAPI: Tighten ValueExpression schemas for Appendix B correctness Apply spec-mandated fixes surfaced by adversarial review against format/expressions-spec.md Appendix B: - Literal.oneOf: remove the type:integer branch — every integer already matches type:number, so the duplicate broke strict oneOf validators. - Apply.arguments.items: switch oneOf to anyOf. Spec line 265 defines FUNC_ARG as EXPR | PREDICATE (semantic "or"), and a bare boolean legitimately matches both branches; oneOf's "exactly one" rejects it. - FieldReference and DeprecatedReference: add additionalProperties:false so the shared type:"reference" discriminator resolves to exactly one branch. Without it, payloads mixing id/name/term validate ambiguously. - Literals typed-object form: mark data-type required per spec line 258 grammar (it appears with no optional marker). - FunctionReference: add additionalProperties:false to the {identifier} form so it is disjoint from the {catalog, identifier} form (spec lines 267-269 enumerate them as distinct productions). Add minItems:1 to every identifier array per spec line 82: "At least one part, the function name, is required." * OpenAPI: Regenerate rest-catalog-open-api.py * OpenAPI: Further tighten ValueExpression schemas per adversarial review Additional spec-fidelity fixes surfaced by iterated compliance/divergence review against format/expressions-spec.md Appendix B: - Literal typed-object, Literals typed-object, Apply, and FunctionReference {catalog, identifier} all get additionalProperties:false so their JSON shapes match the closed grammar productions in spec lines 255-269. - LiteralExpression.value now refs Literal (was PrimitiveTypeValue) — spec line 304 explicitly says the deprecated CMP_OP form takes LITERAL. - SetExpression.values now refs Literals (was inline PrimitiveTypeValue array) — spec line 305 says LITERALS, which includes bare-array and typed-object forms. - TransformTerm.term now refs Term (was Reference) — spec line 310 has term:TERM, which is recursive and admits nested TRANSFORM and DEPRECATED_REF, not only bare NAMEs. - Literal bare oneOf drops type:'null' — Iceberg VALUE grammar (spec 254, 274) is a single scalar value; the typed form's PrimitiveTypeValue has no null branch either, so the null form was asymmetric and dead-weight. - FunctionReference NAME strings (bare-string form and identifier array items across both object forms) get minLength:1 — spec line 82 requires the function name to be a real identifier, not empty. * OpenAPI: Regenerate rest-catalog-open-api.py * OpenAPI: Address reviewer feedback on ValueExpression schemas Adopt Nevin's review comments in narrower form: - Strip "Replaces the deprecated X" clauses from UnaryPredicate, ComparisonPredicate, and SetPredicate descriptions - Rewrite SetPredicate description in operand-neutral form: "A predicate that operates on a list of literals" - Rename Literals from "set of literal values" to "list of literal values" to match wire form (arrays are ordered, admit duplicates) - Dedupe "Iceberg Expressions spec, Appendix B" citations across Literal, Literals, FieldReference, Apply, and FunctionReference; keep the citation on ValueExpression as the parent anchor - Split FieldReference into named FieldReferenceById and FieldReferenceByName joined via oneOf, preserving the bound-vs-unbound context clause in the parent description * OpenAPI: Regenerate rest-catalog-open-api.py * OpenAPI: Tighten predicate/literal schemas per second-round review - Add additionalProperties: false to UnaryPredicate, ComparisonPredicate, and SetPredicate. Without this, a payload carrying both a new field (e.g. child) and a legacy field (e.g. term) matches both the new predicate and its deprecated counterpart under Expression.oneOf, which is a validation error. Matches the pattern already applied to Literal, FieldReference*, and Apply. - Replace the bare-value Literal branch (string | number | boolean) with a $ref to PrimitiveTypeValue. Per the expressions spec, VALUE is a single value JSON drawn from the table spec's primitive types, which includes dates, decimals, UUIDs, and other types the previous three- primitive union excluded. - Rewrite Literal, Literals, and FunctionReference descriptions using the "X is either A, B, C" convention for consistency across the expression schemas. * OpenAPI: Regenerate rest-catalog-open-api.py * OpenAPI: Consolidate predicate schemas per Ryan's feedback Merge deprecated UnaryExpression, LiteralExpression, and SetExpression into UnaryPredicate, ComparisonPredicate, and SetPredicate by adding optional deprecated 'term'/'value' fields. Rename Expression to Predicate, extract FunctionArgument and FunctionIdentifier types, rename FieldReferenceById/ByName to IdReference/NamedReference, and remove DeprecatedReference (never produced by implementations). * OpenAPI: Complete naming pass per Ryan's feedback Rename AndOrExpression/NotExpression to AndOrPredicate/NotPredicate for consistency with the Predicate top-level schema. Rename FieldReference to Reference (the new structured oneOf) and rename the old bare-string Reference to TermReference (deprecated), matching Ryan's stated preference for the prestigious name on the new type. * OpenAPI: Inline function reference and reuse CatalogObjectIdentifier Delete the FunctionReference and FunctionIdentifier wrapper schemas (each was used only once) and inline the function oneOf directly into Apply.function. Reuse the existing CatalogObjectIdentifier for identifier arrays instead of redefining them inline, per Ryan's narrow ask on the two identifier fields. * OpenAPI: Drop Appendix B references from deprecation notices Apply Ryan's principle ("this is its own spec definition") to the TrueExpression and FalseExpression deprecation descriptions, removing the Appendix B / JSON serialization references. * OpenAPI: Restrict TransformTerm.term to TermReference Per Ryan's feedback, TransformTerm.term should be a direct name reference rather than a recursive Term union, so a transform can no longer wrap another transform. * OpenAPI: Mark legacy term types as deprecated Per Ryan's feedback, deprecate the legacy term types consistently so they are not reused: Term and TransformTerm now carry deprecated: true alongside TermReference. * OpenAPI: Allow Apply.function identifier object without catalog The expressions spec permits a FUNC_REF of the form { "identifier": [ NAME* ] } where catalog is missing/null. Make catalog optional on the object variant of Apply.function so this form is representable. --------- Co-authored-by: Prashant Singh <prashant.singh@snowflake.com>
Bumps `bouncycastle` from 1.84 to 1.85. Updates `org.bouncycastle:bcpkix-jdk18on` from 1.84 to 1.85 - [Changelog](https://github.com/bcgit/bc-java/blob/main/docs/releasenotes.html) - [Commits](https://github.com/bcgit/bc-java/commits) Updates `org.bouncycastle:bcprov-jdk18on` from 1.84 to 1.85 - [Changelog](https://github.com/bcgit/bc-java/blob/main/docs/releasenotes.html) - [Commits](https://github.com/bcgit/bc-java/commits) Updates `org.bouncycastle:bcutil-jdk18on` from 1.84 to 1.85 - [Changelog](https://github.com/bcgit/bc-java/blob/main/docs/releasenotes.html) - [Commits](https://github.com/bcgit/bc-java/commits) --- updated-dependencies: - dependency-name: org.bouncycastle:bcpkix-jdk18on dependency-version: '1.85' dependency-type: direct:production update-type: version-update:semver-minor - dependency-name: org.bouncycastle:bcprov-jdk18on dependency-version: '1.85' dependency-type: direct:production update-type: version-update:semver-minor - dependency-name: org.bouncycastle:bcutil-jdk18on dependency-version: '1.85' dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Use Guava-compatible placeholders in commit timeout, prefix listing, and async planner errors. Verify that affected messages include their values across supported engine versions. Generated-by: GitHub Copilot (github-copilot/gpt-5.6-sol) Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
* Build: bump Jackson to fix CVE GHSA-r7wm-3cxj-wff9 * Build: bump Jackson to fix CVE GHSA-r7wm-3cxj-wff9 * change to 2.18 * remove CVE-2026-54512 and CVE-2026-54513 * remove jackson215 * Remove line about jackson in spark ignore
Co-authored-by: Isaac Co-authored-by: Uros Bojanic <uros.bojanic@databricks.com>
… Awaitility.await (apache#17405)
…apache#17403) Renames the streaming merge-append write config to `use-merge-append-for-streaming` in Spark 4.1, and ports the option to the Spark 3.5 and 4.0 modules. When enabled, streaming appends use a merge append instead of a fast append so manifests are merged on commit rather than accumulating as small manifests.
Co-authored-by: Shawn Chang <yxchang@amazon.com> --------- Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com> Co-authored-by: Shawn Chang <yxchang@amazon.com>
* Docs: Centralize Slack invitation URL Use the stable ASF redirect and share it through the MkDocs configuration to avoid repeated expiring invitation links. Generated-by: Codex * Apply suggestion from @kevinjqliu
* Core: Drop test prefix from v4 struct test methods JUnit 5 does not require the test prefix on test methods; it is a JUnit 3 holdover. Test methods are identified by their annotations. Drop the prefix from the v4 tracked-file struct tests, matching the convention used for newer tests. * DV prefix * spotless apply
Co-authored-by: Isaac Co-authored-by: Uros Bojanic <uros.bojanic@databricks.com>
apache@100d062 updated content stats to also include avg_value_size_in_bytes for Geo types. This makes sure to properly reflect this in the Spec
…pache#17419) * API: Fix null counts on a required column in evaluator test The single_value_nulls fixtures in testNotEqWithSingleValue and testNotInWithSingleValue reported 2 nulls for field 3, the "required" column. A required column cannot contain nulls, so no writer produces those metrics. Move both to the optional "some_empty" column, which keeps the case the tests cover: a file whose bounds hold a single value still has to be read when the column also contains nulls, because those nulls match != and NOT IN. The NOT IN fixture also reported a NaN count for a string column, which only float and double columns track. * API: Remove NaN counts for a string column in evaluator test The range_of_values and single_value fixtures reported a NaN count for field 3, a string column. Only float and double columns track NaN counts, which is why the single_value fixture in testNotEqWithSingleValue already passed null for them. Generated-by: Cursor
* Core: Add v4 manifest reader
…ring (apache#17358) (apache#17360) nullableSnapshotIdAsOfTime previously relied on snapshot-log entries being in chronological order, returning the last entry with timestamp <= requested. The spec does not guarantee ordering, and TableMetadata allows up to 1 minute of clock skew between entries. This could return the wrong snapshot when entries are out of order (e.g. due to WAP cherry-pick workflows). Fix by finding the entry with the maximum timestamp <= requested timestamp, which is correct regardless of iteration order. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
nastra
force-pushed
the
read-content-stats-from-manifest
branch
from
July 30, 2026 15:29
cfd0e3d to
0e72e44
Compare
github-actions
Bot
added
API
INFRA
BUILD
DOCS
SPARK
SPECIFICATION
FLINK
AWS
GCP
OPENAPI
KAFKACONNECT
labels
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on apache#16958, which landed the v4 manifest reader but left content stats reading out.
The table schema comes in through
projectStats, since stats field IDs and types are derived from the table's fields:No schema means no stats, so existing callers are unaffected. Stats for fields the row filter references are always read even when the caller's projection leaves them out.
Model: Claude Opus 5
Platform/Tool: Cursor
Human Oversight: fully reviewed
Prompt Summary: Add content stats reading to V4ManifestReader — schema derivation, projection rules, and custom type registration — then reconcile it against review feedback on apache#16958.