Skip to content

Support incremental ingest for SortedTemporalChunks - #12812

Open
0n41rd4 wants to merge 2 commits into
rerun-io:mainfrom
0n41rd4:incremental-update/no-test-bench
Open

Support incremental ingest for SortedTemporalChunks#12812
0n41rd4 wants to merge 2 commits into
rerun-io:mainfrom
0n41rd4:incremental-update/no-test-bench

Conversation

@0n41rd4

@0n41rd4 0n41rd4 commented Jun 4, 2026

Copy link
Copy Markdown

Related

  • Implements // TODO(emilk): handle incremental ingestion from 9a67723 ("Support progressive ingest of the rrd manifest")

What

RrdManifestIndex::append() is calling SortedTemporalChunks::new() with the
full concatenated manifest on every delta, rebuilding the sorted cache from scratch.

In this PR:

  1. RrdManifestIndex::append() calls self.sorted_chunks.update() with just
    the delta manifest, not the full accumulation.

  2. SortedTemporalChunks::update() tracks delta-introduced chunks in a local
    BTreeMap scratch map. Per-component merges extend existing sorted vecs
    with only the new entries. The bottom-up tree walk reads from the scratch map
    instead of rebuilding per_entity from all chunks.


A separate branch incremental-update/test-bench on this fork has a test and a timing benchmark I used during development:

Benchmark (2000 entities, 40 deltas, 4 components, release mode):

Approach Time
Full rebuild (new() + extend/sort/dedup) ~2000ms
Incremental (this PR) ~490ms

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi! Thanks for opening this pull request.

Because this is your first time contributing to this repository, make sure you've read our Contributor Guide and Code of Conduct.

@0n41rd4
0n41rd4 marked this pull request as ready for review June 4, 2026 02:21
@0n41rd4
0n41rd4 force-pushed the incremental-update/no-test-bench branch 5 times, most recently from 84fdb53 to 37ecf56 Compare June 11, 2026 21:27
@0n41rd4

0n41rd4 commented Jun 11, 2026

Copy link
Copy Markdown
Author

hi, is this algorithm something you were interested in reviewing?

@0n41rd4
0n41rd4 force-pushed the incremental-update/no-test-bench branch from 37ecf56 to c0e1564 Compare July 17, 2026 14:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant