Support incremental ingest for SortedTemporalChunks - #12812
Open
0n41rd4 wants to merge 2 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Hi! Thanks for opening this pull request.
Because this is your first time contributing to this repository, make sure you've read our Contributor Guide and Code of Conduct.
0n41rd4
marked this pull request as ready for review
June 4, 2026 02:21
0n41rd4
force-pushed
the
incremental-update/no-test-bench
branch
5 times, most recently
from
June 11, 2026 21:27
84fdb53 to
37ecf56
Compare
Author
|
hi, is this algorithm something you were interested in reviewing? |
0n41rd4
force-pushed
the
incremental-update/no-test-bench
branch
from
July 17, 2026 14:08
37ecf56 to
c0e1564
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related
// TODO(emilk): handle incremental ingestionfrom 9a67723 ("Support progressive ingest of the rrd manifest")What
RrdManifestIndex::append()is callingSortedTemporalChunks::new()with thefull concatenated manifest on every delta, rebuilding the sorted cache from scratch.
In this PR:
RrdManifestIndex::append()callsself.sorted_chunks.update()with justthe delta manifest, not the full accumulation.
SortedTemporalChunks::update()tracks delta-introduced chunks in a localBTreeMapscratch map. Per-component merges extend existing sorted vecswith only the new entries. The bottom-up tree walk reads from the scratch map
instead of rebuilding
per_entityfrom all chunks.A separate branch
incremental-update/test-benchon this fork has a test and a timing benchmark I used during development:Benchmark (2000 entities, 40 deltas, 4 components, release mode):
new()+ extend/sort/dedup)