Core: Read content stats from v4 Manifest - #17433
Open
nastra wants to merge 10 commits into
Open
Conversation
nastra
marked this pull request as draft
July 30, 2026 15:45
nastra
force-pushed
the
read-content-stats-from-manifest
branch
from
July 30, 2026 15:53
0e72e44 to
44cadbc
Compare
nastra
force-pushed
the
read-content-stats-from-manifest
branch
from
July 30, 2026 18:21
44cadbc to
c62305e
Compare
nastra
force-pushed
the
read-content-stats-from-manifest
branch
from
July 30, 2026 18:25
c62305e to
6cf0fd0
Compare
anoopj
reviewed
Jul 30, 2026
| readBuilder.setCustomType(TrackedFile.CONTENT_STATS_ID, ContentStatsStruct.class); | ||
| // content_stats holds one stats struct per projected column | ||
| for (Types.NestedField fieldStats : statsField.type().asStructType().fields()) { | ||
| readBuilder.setCustomType(fieldStats.fieldId(), FieldStatsStruct.class); |
Member
There was a problem hiding this comment.
Does geo and variant need any special handling here? Can we add a test to make sure those types work?
Contributor
Author
There was a problem hiding this comment.
good point, I've added tests for geo + variant types with single/multiple files and this uncovered a bug around Geo types copying, which I've fixed in FieldStatsStruct
anoopj
reviewed
Jul 30, 2026
anoopj
left a comment
anoopj
left a comment
Member
There was a problem hiding this comment.
Is the file pruning coming in a followup?
nastra
force-pushed
the
read-content-stats-from-manifest
branch
from
July 31, 2026 07:46
5420c62 to
ad7a0fe
Compare
nastra
force-pushed
the
read-content-stats-from-manifest
branch
from
July 31, 2026 10:17
d6593b4 to
4975ce5
Compare
nastra
force-pushed
the
read-content-stats-from-manifest
branch
from
July 31, 2026 10:38
4975ce5 to
33d52b2
Compare
nastra
marked this pull request as ready for review
July 31, 2026 10:54
nastra
requested review from
amogh-jahagirdar,
danielcweeks,
rdblue and
stevenzwu
July 31, 2026 10:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
V4ManifestReadernow reads thecontent_statscolumn, so eachTrackedFilecomesback with per-column bounds and counts.
The stats schema is derived from the table schema, so
builder()takes it as a secondargument. Stats for every column are read by default, which is what copying entries into
a new manifest needs. Callers that want less can narrow:
projectStats(fieldIds)reads stats only for the given columns,projectStats()reads noneforScanPlanning()reads only what the filter needsselect/projectomits themReading stats surfaced two bugs in the copy path, fixed here:
ContentStatsStruct.copy()threw an NPE when a projected column had no stats in themanifest
FieldStatsStructcopiedStructLikebounds by reference. Geometry and geography bounda bounding-box struct, so under
reuseContainers()every entry reported the last row'sbounds. Bounds are now deep-copied through
StructLikeUtil.copy, andStructCopyisSerializableso a copied bound still survives serialization.Used Claude for the initial prototyping but reviewed and adjusted the code manually