Skip to content

feat: persist and backfill historical epoch data - #39

Merged
mintybasil merged 1 commit into
mainfrom
zeroklaw/persist-historical-epochs
May 7, 2026
Merged

feat: persist and backfill historical epoch data#39
mintybasil merged 1 commit into
mainfrom
zeroklaw/persist-historical-epochs

Conversation

@zeroklaw

@zeroklaw zeroklaw commented May 7, 2026

Copy link
Copy Markdown
Member

Summary

Prevent historical epoch data loss by persisting raw MiningEntry data in a permanent KV key (epochs-historical), ensuring all historical ZKC prices are backfilled over time.

Problem

The Boundless upstream API only returns the most recent 100 epoch entries. As new epochs complete, older ones silently drop off the list. With the current implementation:

  • Only 100 entries are ever visible
  • Epochs that drop off the window are permanently lost
  • 51 epochs (20–70) have zkc_price_usd: 0 because prices were never fetched for those dates

Changes

  • Add HISTORICAL_CACHE_KEY (epochs-historical) — permanent KV store with no TTL
  • Add mergeHistoricalEntries() — deduplicates fresh upstream data with historical entries by epoch number (fresh data wins for conflicting epochs)
  • Modify onRequestGet — on every cache miss:
    1. Merge fresh upstream entries with the historical store
    2. Persist the merged dataset back to epochs-historical (no TTL)
    3. Pass the full merged dataset through buildPriceMap and normaliseEntries
  • ZKC price backfill happens automatically — since buildPriceMap only fetches missing dates, feeding it the full historical dataset triggers price lookups for all epochs that were previously skipped

Architecture

Upstream (mining) → fresh 100 entries
                      ↓
              merge with KV "epochs-historical" (permanent)
                      ↓
              buildPriceMap (only fetches missing dates)
                      ↓
              normaliseEntries → KV "epochs" (2h TTL, hot cache)
                      ↓
              applyLimit per request → response

Closes #37
Closes #22 (100 epochs target superseded by "all available epochs")

Merge upstream MiningEntry data with a permanent KV store on every
request. The historical store accumulates entries across invocations,
preventing data loss when epochs drop off the upstream 100-entry window.

- Add HISTORICAL_CACHE_KEY ('epochs-historical') — no TTL, permanent
- Add mergeHistoricalEntries() — deduplicate by epoch, fresh data wins
- Pass merged entries to buildPriceMap and normaliseEntries
- This also triggers ZKC price backfill for all historical epochs
  since buildPriceMap only fetches missing dates
@github-actions

github-actions Bot commented May 7, 2026

Copy link
Copy Markdown

Preview deployment

URL
Latest commit https://fe261f3a.boundless-profit-explorer.pages.dev
Branch alias https://head.boundless-profit-explorer.pages.dev

The branch alias URL always points to the latest commit on this PR.

@mintybasil
mintybasil merged commit ed48e76 into main May 7, 2026
1 check passed
@mintybasil
mintybasil deleted the zeroklaw/persist-historical-epochs branch May 7, 2026 18:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: persist and backfill historical epoch data to prevent data loss Increase epoch data range

2 participants