system.mdb is explicitly skipped by compaction, so operators have no path to reclaim space once internal tables (notably hdb_analytics) have grown the file.
Current behavior
In resources/databases.ts:666, the compaction loop skips any path ending in system.mdb:
if (store.needsDeletion && !path.endsWith('system.mdb')) {
// ... compact ...
}Combined with long retention windows on hdb_analytics, customers can end up with system.mdb in the 40–60GB range with no supported way to shrink it (see customer nodes in CORE-3074, 48–58GB on all four nodes).
Asks
- Investigate the reason
system.mdb was excluded — assume an intentional concurrency/correctness concern rather than oversight, but confirm.
- Either: enable compaction for
system.mdb with appropriate safeguards (e.g. quiesce internal writers), or provide an alternative reclamation mechanism (offline-compaction CLI, copy-and-replace utility based on copyDb.ts, etc.).
- Document operator guidance for shrinking an over-grown
system.mdb.
Acceptance criteria
- Operators have a documented, supported path to reduce a bloated
system.mdb without manual database surgery.
- The chosen approach is safe under typical operating conditions (or clearly requires Harper to be stopped).
Priority context
Lower priority than the retention configurability change — current strategic direction is RocksDB, which does not require compaction. This is a workaround for customers still on LMDB.
Tracked in Jira: CORE-3074 (a customer escalation)
Related: CORE-3045 (Create RocksDB migration/copy script based on existing copyDb.ts)
🤖 Filed by Claude on behalf of Kris.
system.mdbis explicitly skipped by compaction, so operators have no path to reclaim space once internal tables (notablyhdb_analytics) have grown the file.Current behavior
In resources/databases.ts:666, the compaction loop skips any path ending in
system.mdb:Combined with long retention windows on
hdb_analytics, customers can end up withsystem.mdbin the 40–60GB range with no supported way to shrink it (see customer nodes in CORE-3074, 48–58GB on all four nodes).Asks
system.mdbwas excluded — assume an intentional concurrency/correctness concern rather than oversight, but confirm.system.mdbwith appropriate safeguards (e.g. quiesce internal writers), or provide an alternative reclamation mechanism (offline-compaction CLI, copy-and-replace utility based oncopyDb.ts, etc.).system.mdb.Acceptance criteria
system.mdbwithout manual database surgery.Priority context
Lower priority than the retention configurability change — current strategic direction is RocksDB, which does not require compaction. This is a workaround for customers still on LMDB.
Tracked in Jira: CORE-3074 (a customer escalation)
Related: CORE-3045 (Create RocksDB migration/copy script based on existing copyDb.ts)
🤖 Filed by Claude on behalf of Kris.