Description:
lndinit migrate-db writes tombstone markers inside the source bbolt database files upon successful migration. These tombstones are permanent and cause LND to refuse to open the bbolt files with:
refusing to use db, it was marked with a tombstone after successful data migration
The problem: lndinit does not verify that the target LND binary actually supports the destination backend before writing these tombstones. If the LND binary was not compiled with the required build tag (e.g. kvdb_sqlite), the migration completes successfully but LND fails on startup with:
error opening sqlite graph DB: unknown database type
At this point the user is stuck. The sqlite files exist but LND can't use them. The bbolt files are tombstoned and LND refuses to open them. The only recovery path is restoring from a pre-migration backup. If no backup exists, the node's channel state is irrecoverably lost.
What happened
- Ran
lndinit migrate-db from bbolt to sqlite on a BTCPay Server LND node (v0.19.3-beta)
- All five databases migrated and verified successfully (channeldb, macaroondb, decayedlogdb, walletdb, neutrinodb)
lndinit wrote tombstones into each bbolt file
- LND failed on startup because the BTCPay LND image isn't compiled with
kvdb_sqlite
- LND also refused to fall back to bbolt because of the tombstones
- Had to restore all bbolt files from a manual backup made before migration
What should happen instead
Any of the following would prevent data loss:
- Pre-flight check: Before migrating, verify the target LND binary supports the destination backend. Fail early with a clear error if it doesn't.
- Revert command: Provide
lndinit migrate-db --remove-tombstone or lndinit migrate-db --revert to strip tombstones from bbolt files and restore them to a usable state.
- Deferred tombstone: Don't write tombstones immediately. Instead, provide a separate
lndinit migrate-db --finalize command that writes tombstones only after the user confirms LND started successfully on the new backend.
- At minimum: Print a loud warning before writing tombstones that this operation is irreversible and the user should have a backup.
Environment
- lndinit: v0.1.33-beta (linux-arm64)
- LND: v0.19.3-beta (BTCPay Server Docker image, missing
kvdb_sqlite build tag)
- Platform: Raspberry Pi 5
- Active Lightning channel with real funds at risk
Description:
lndinit migrate-dbwrites tombstone markers inside the source bbolt database files upon successful migration. These tombstones are permanent and cause LND to refuse to open the bbolt files with:The problem:
lndinitdoes not verify that the target LND binary actually supports the destination backend before writing these tombstones. If the LND binary was not compiled with the required build tag (e.g.kvdb_sqlite), the migration completes successfully but LND fails on startup with:At this point the user is stuck. The sqlite files exist but LND can't use them. The bbolt files are tombstoned and LND refuses to open them. The only recovery path is restoring from a pre-migration backup. If no backup exists, the node's channel state is irrecoverably lost.
What happened
lndinit migrate-dbfrom bbolt to sqlite on a BTCPay Server LND node (v0.19.3-beta)lndinitwrote tombstones into each bbolt filekvdb_sqliteWhat should happen instead
Any of the following would prevent data loss:
lndinit migrate-db --remove-tombstoneorlndinit migrate-db --revertto strip tombstones from bbolt files and restore them to a usable state.lndinit migrate-db --finalizecommand that writes tombstones only after the user confirms LND started successfully on the new backend.Environment
kvdb_sqlitebuild tag)