Summary
ResilientDB currently persists all data in node-local LevelDB or MemoryDB storage indefinitely. Since ResilientDB is an append-only ledger by design, storage grows unboundedly in long-running production environments. There is no mechanism to offload historical or archival data, even though such data is accessed far less frequently than recent state. This leads to SSD bloat and makes the system impractical for large-scale deployments.
This issue proposes introducing ResTier, a tiered storage engine that transparently migrates cold data (calculated based on timestamp and configured threshold) from hot storage (MemoryDB/LevelDB) to IPFS as a decentralized, content-addressed cold tier - without any overhead on the write path or modifications to the PBFT consensus process.
Motivation
The current single-tier storage model restrictions:
Unbounded local SSD growth leading to vertical scaling of nodes
Cost-effective archival of historical ledger data - historical data is less frequently queried
Separation of hot (frequently accessed) and cold (archival) data
Applications running ResilientDB at scale must manage storage externally. ResTier solves this natively within the storage layer.
Proposed Changes - PR link: #260
IPFS Client - Introduce an HTTP-based IPFS client against the Kubo RPC API
Support Add, Cat, Exists, Unpin operations with retry logic
Add IPFSConfig proto for endpoint, timeout, and retry configuration
Tiered Storage - Introduce TieredStorage as a three-tier orchestrator (HOT → WARM → COLD)
Background migration thread moves cold-eligible keys to IPFS based on configurable checkpoint thresholds
Transparent read fallback across tiers with no changes to existing KV interfaces
Manifest persistence to warm LevelDB for crash recovery with zero data loss
Secondary Index Manifest - Add SecondaryIndex virtual interface for key-to-CID lookups
Implement InMemoryHashIndex for O(1) cold read resolution
Persist index manifest to warm LevelDB across restarts
Storage Backend Changes - Add DeletableStorage interface with DeleteKey to support hot tier eviction
Extend LevelDB and MemoryDB with DeleteKey, mutex-protected batch writes, and timer instrumentation
KV Service & Configuration - Refactor backend selection via StorageConfig enum supporting 4 storage modes: MEMORYDB, LEVELDB, TIERED (LevelDB->IPFS), TIERED (MemoryDB->IPFS)
Add server_tiered.config, server_tiered_leveldb.config, server_tiered_memorydb.config
Migration Daemon - Add standalone sidecar process for orchestrating cold data migration independently of the main service
Compatibility
Fully optional module - existing KV applications remain unaffected
No modifications to consensus or ledger protocols
Tiered mode enabled via runtime configuration only
Each PBFT replica migrates independently with no cross-node coordination
Validation
Unit tests with stub IPFS client covering migration logic and read fallback
IPFS integration test against a live Kubo node
Crash recovery correctness verified across all four migration windows
Expected Benefits
Handles unbounded SSD disk growth for long-running ResilientDB deployments
Enables cost-effective archival of historical ledger data via IPFS
Provides O(1) cold read performance via in-memory secondary index
Zero write-path overhead - PBFT consensus latency is fully preserved
Built-in deduplication and trustless content verification via IPFS CIDs
Summary
ResilientDB currently persists all data in node-local LevelDB or MemoryDB storage indefinitely. Since ResilientDB is an append-only ledger by design, storage grows unboundedly in long-running production environments. There is no mechanism to offload historical or archival data, even though such data is accessed far less frequently than recent state. This leads to SSD bloat and makes the system impractical for large-scale deployments.
This issue proposes introducing ResTier, a tiered storage engine that transparently migrates cold data (calculated based on timestamp and configured threshold) from hot storage (MemoryDB/LevelDB) to IPFS as a decentralized, content-addressed cold tier - without any overhead on the write path or modifications to the PBFT consensus process.
Motivation
The current single-tier storage model restrictions:
Unbounded local SSD growth leading to vertical scaling of nodes
Cost-effective archival of historical ledger data - historical data is less frequently queried
Separation of hot (frequently accessed) and cold (archival) data
Applications running ResilientDB at scale must manage storage externally. ResTier solves this natively within the storage layer.
Proposed Changes - PR link: #260
IPFS Client - Introduce an HTTP-based IPFS client against the Kubo RPC API
Support Add, Cat, Exists, Unpin operations with retry logic
Add IPFSConfig proto for endpoint, timeout, and retry configuration
Tiered Storage - Introduce TieredStorage as a three-tier orchestrator (HOT → WARM → COLD)
Background migration thread moves cold-eligible keys to IPFS based on configurable checkpoint thresholds
Transparent read fallback across tiers with no changes to existing KV interfaces
Manifest persistence to warm LevelDB for crash recovery with zero data loss
Secondary Index Manifest - Add SecondaryIndex virtual interface for key-to-CID lookups
Implement InMemoryHashIndex for O(1) cold read resolution
Persist index manifest to warm LevelDB across restarts
Storage Backend Changes - Add DeletableStorage interface with DeleteKey to support hot tier eviction
Extend LevelDB and MemoryDB with DeleteKey, mutex-protected batch writes, and timer instrumentation
KV Service & Configuration - Refactor backend selection via StorageConfig enum supporting 4 storage modes: MEMORYDB, LEVELDB, TIERED (LevelDB->IPFS), TIERED (MemoryDB->IPFS)
Add server_tiered.config, server_tiered_leveldb.config, server_tiered_memorydb.config
Migration Daemon - Add standalone sidecar process for orchestrating cold data migration independently of the main service
Compatibility
Fully optional module - existing KV applications remain unaffected
No modifications to consensus or ledger protocols
Tiered mode enabled via runtime configuration only
Each PBFT replica migrates independently with no cross-node coordination
Validation
Unit tests with stub IPFS client covering migration logic and read fallback
IPFS integration test against a live Kubo node
Crash recovery correctness verified across all four migration windows
Expected Benefits
Handles unbounded SSD disk growth for long-running ResilientDB deployments
Enables cost-effective archival of historical ledger data via IPFS
Provides O(1) cold read performance via in-memory secondary index
Zero write-path overhead - PBFT consensus latency is fully preserved
Built-in deduplication and trustless content verification via IPFS CIDs