Skip to content

Cache RootLists with KVCachedFile#435

Merged
ihtefyw merged 8 commits into
mainfrom
luigi/cache-root-list
Jul 24, 2026
Merged

Cache RootLists with KVCachedFile#435
ihtefyw merged 8 commits into
mainfrom
luigi/cache-root-list

Conversation

@ihtefyw

@ihtefyw ihtefyw commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

The RootList in the MPT archive implementation keeps all the root hashes in memory and can grow indefinitely.
Similarly to #434, this PR adds a caching layer to RootList by using the KVCachedFile.
A new KVFile called OrderedFile is added, an array-like storage with a key-file offset mapping.

Closes https://github.com/0xsoniclabs/sonic-admin/issues/845

@ihtefyw
ihtefyw force-pushed the luigi/cache-root-list branch 2 times, most recently from 8d83aad to 348153b Compare July 14, 2026 12:18
@ihtefyw
ihtefyw requested a review from Copilot July 14, 2026 12:40

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a disk-backed caching layer for the MPT archive RootList using common.KVCachedFile to prevent unbounded in-memory growth, and adds migration logic to convert the legacy roots.dat format into the new key+value format.

Changes:

  • Add a new root_list.go implementing rootList on top of KVCachedFile with a versioned header + legacy migration.
  • Update ArchiveTrie to use the new rootList API (error-returning Get/Append) and adjust verification to read roots via GetAll.
  • Add comprehensive tests for root list persistence, migration, and checkpoint/restore behavior; update archive trie tests accordingly.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 10 comments.

File Description
go/database/mpt/root_list.go New KVCachedFile-backed root list, including on-disk format/versioning and migration + restore/truncate helpers.
go/database/mpt/root_list_test.go New/ported tests covering header/version, read/write roundtrips, migration, checkpoint integration, and truncate/restore flows.
go/database/mpt/archive_trie.go Switch archive trie root handling to the new rootList methods and update verification/root access paths.
go/database/mpt/archive_trie_test.go Update tests to construct a real rootList and remove tests tied to the old in-memory slice behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread go/database/mpt/archive_trie.go
Comment thread go/database/mpt/archive_trie.go
Comment thread go/database/mpt/archive_trie.go
Comment thread go/database/mpt/root_list.go Outdated
Comment thread go/database/mpt/root_list.go Outdated
Comment thread go/database/mpt/root_list.go Outdated
Comment thread go/database/mpt/root_list_test.go Outdated
Comment thread go/database/mpt/archive_trie_test.go Outdated
Comment thread go/database/mpt/archive_trie_test.go Outdated
Comment thread go/database/mpt/root_list.go Outdated
@ihtefyw
ihtefyw force-pushed the luigi/cache-root-list branch 2 times, most recently from 31e8f03 to b8660df Compare July 20, 2026 12:15
@ihtefyw
ihtefyw force-pushed the luigi/kv-cached-file branch 4 times, most recently from 398e739 to 5e52d3b Compare July 20, 2026 13:40
@ihtefyw
ihtefyw force-pushed the luigi/cache-root-list branch 2 times, most recently from a3b5937 to 5fe7d90 Compare July 20, 2026 14:28
@ihtefyw
ihtefyw force-pushed the luigi/kv-cached-file branch from 5e52d3b to 229b751 Compare July 21, 2026 07:54
@ihtefyw
ihtefyw force-pushed the luigi/cache-root-list branch 2 times, most recently from 715ebc4 to d1cbdc2 Compare July 21, 2026 08:03
@ihtefyw
ihtefyw force-pushed the luigi/kv-cached-file branch 2 times, most recently from b3bcaa4 to a916eb7 Compare July 21, 2026 08:30
@ihtefyw
ihtefyw force-pushed the luigi/cache-root-list branch from d1cbdc2 to 2d08ed1 Compare July 21, 2026 08:31
@ihtefyw
ihtefyw requested a review from Copilot July 21, 2026 08:31

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

Comment thread go/common/ordered_file.go
Comment thread go/database/mpt/archive_trie.go Outdated
Comment thread go/database/mpt/archive_trie.go Outdated
Comment thread go/common/ordered_file_test.go Outdated
Comment thread go/database/mpt/archive_trie.go
@ihtefyw
ihtefyw force-pushed the luigi/cache-root-list branch from 2d08ed1 to 302db22 Compare July 21, 2026 08:58
@ihtefyw
ihtefyw marked this pull request as ready for review July 21, 2026 09:01
@ihtefyw
ihtefyw requested a review from HerbertJordan July 21, 2026 09:07
@codecov

codecov Bot commented Jul 21, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 60.13986% with 57 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
go/database/mpt/archive_trie.go 56.81% 36 Missing and 21 partials ⚠️

❌ Your patch check has failed because the patch coverage (60.13%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Files with missing lines Coverage Δ
go/database/mpt/live_trie.go 98.40% <100.00%> (-1.60%) ⬇️
go/database/mpt/verification.go 83.33% <100.00%> (+0.16%) ⬆️
go/database/mpt/archive_trie.go 87.88% <56.81%> (-10.35%) ⬇️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ihtefyw
ihtefyw force-pushed the luigi/cache-root-list branch 2 times, most recently from f562035 to 762ea25 Compare July 21, 2026 13:39
@ihtefyw
ihtefyw changed the base branch from luigi/kv-cached-file to luigi/ordered_file July 21, 2026 13:42
@ihtefyw
ihtefyw force-pushed the luigi/ordered_file branch from a7bcb7e to c216905 Compare July 22, 2026 07:56
@ihtefyw
ihtefyw force-pushed the luigi/cache-root-list branch from 762ea25 to 6a284ef Compare July 22, 2026 08:02
@ihtefyw
ihtefyw force-pushed the luigi/ordered_file branch from c216905 to e408190 Compare July 22, 2026 08:50
@ihtefyw
ihtefyw force-pushed the luigi/cache-root-list branch from 3d0da97 to 0b3d2ea Compare July 22, 2026 08:50
Comment thread go/database/mpt/archive_trie.go Outdated
Comment thread go/database/mpt/archive_trie.go
Comment thread go/database/mpt/archive_trie.go Outdated
Comment thread go/database/mpt/archive_trie.go
Comment thread go/database/mpt/archive_trie.go Outdated
@ihtefyw
ihtefyw force-pushed the luigi/ordered_file branch from e408190 to 004bb79 Compare July 22, 2026 11:31
@ihtefyw
ihtefyw force-pushed the luigi/cache-root-list branch from a5fe4c8 to 64dbc96 Compare July 22, 2026 11:43
@ihtefyw
ihtefyw force-pushed the luigi/ordered_file branch from eb0362f to 5605fe5 Compare July 22, 2026 11:46
@ihtefyw
ihtefyw force-pushed the luigi/cache-root-list branch 4 times, most recently from f2d0294 to d2cf3ea Compare July 22, 2026 14:08
@ihtefyw
ihtefyw force-pushed the luigi/ordered_file branch 2 times, most recently from 780309c to beb39b7 Compare July 23, 2026 13:55
Base automatically changed from luigi/ordered_file to main July 24, 2026 11:42
@ihtefyw
ihtefyw force-pushed the luigi/cache-root-list branch from 1fe2fa3 to 3dca1e1 Compare July 24, 2026 12:33
@ihtefyw
ihtefyw merged commit 02aea59 into main Jul 24, 2026
3 of 4 checks passed
@ihtefyw
ihtefyw deleted the luigi/cache-root-list branch July 24, 2026 13:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants