feat: persist and backfill historical epoch data - #39
Merged
Conversation
Merge upstream MiningEntry data with a permanent KV store on every
request. The historical store accumulates entries across invocations,
preventing data loss when epochs drop off the upstream 100-entry window.
- Add HISTORICAL_CACHE_KEY ('epochs-historical') — no TTL, permanent
- Add mergeHistoricalEntries() — deduplicate by epoch, fresh data wins
- Pass merged entries to buildPriceMap and normaliseEntries
- This also triggers ZKC price backfill for all historical epochs
since buildPriceMap only fetches missing dates
Preview deployment
The branch alias URL always points to the latest commit on this PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Prevent historical epoch data loss by persisting raw
MiningEntrydata in a permanent KV key (epochs-historical), ensuring all historical ZKC prices are backfilled over time.Problem
The Boundless upstream API only returns the most recent 100 epoch entries. As new epochs complete, older ones silently drop off the list. With the current implementation:
zkc_price_usd: 0because prices were never fetched for those datesChanges
HISTORICAL_CACHE_KEY(epochs-historical) — permanent KV store with no TTLmergeHistoricalEntries()— deduplicates fresh upstream data with historical entries by epoch number (fresh data wins for conflicting epochs)onRequestGet— on every cache miss:epochs-historical(no TTL)buildPriceMapandnormaliseEntriesbuildPriceMaponly fetches missing dates, feeding it the full historical dataset triggers price lookups for all epochs that were previously skippedArchitecture
Closes #37
Closes #22 (100 epochs target superseded by "all available epochs")