Skip to content

feat(backup): hold shared backup directory lock during restore#699

Open
cb1kenobi wants to merge 2 commits into
mainfrom
feat/restore-shared-backup-lock
Open

feat(backup): hold shared backup directory lock during restore#699
cb1kenobi wants to merge 2 commits into
mainfrom
feat/restore-shared-backup-lock

Conversation

@cb1kenobi

Copy link
Copy Markdown
Member

Problem

backups.restore() took no lock on the backup directory, so a concurrent writer — db.backup(), backups.delete(), or backups.purge() — could delete the very backup a restore was copying from. RocksDB offers no protection here: it only serializes work within a single BackupEngine instance, and restore/purge each open their own.

The failure is asymmetric, which is what makes it worth fixing: the default purgeAllFiles restore mode wipes the destination directory before copying, so a restore that fails mid-copy leaves no usable database and no fallback. A rejected writer, by contrast, just retries later.

Change

tryFileLock / tryAcquireFileLock gain a shared (reader) modeflock(LOCK_SH) on POSIX, LockFileEx without LOCKFILE_EXCLUSIVE_LOCK on Windows (same byte range, so shared and exclusive locks conflict correctly across processes, containers sharing a kernel, and worker_threads).

backups.restore() now holds the directory's .backup.lock in shared mode for its duration (native restore + transaction-log snapshot copy):

  • Concurrent restores from the same directory coexist.
  • A writer racing a restore rejects with the existing Backup directory is locked error, and vice versa — contention still rejects, never queues, consistent with the existing writer-vs-writer contract.
  • list/verify remain unlocked, so cheap listings don't reject during a long backup.

The no-op degradation on filesystems without flock support (FUSE/9p mounts) applies to shared mode the same way it already did for exclusive.

Also in this PR

  • tryFileLock now creates missing parent directories — the point of the function is to create the lock — and coerces its shared argument by truthiness, matching normal JS semantics for the public utility API.
  • Because the lock file no longer throws on a missing parent, withBackupDirLock now rejects a missing backup directory explicitly, so delete/purge/restore on a typo'd path surface Backup directory does not exist: … instead of silently conjuring an empty directory and failing deeper in the engine.

Tests

  • Shared/exclusive conflict matrix in both test/file-lock.test.ts and the native GoogleTest suite (test/native/file_lock_test.cc).
  • test/backup.test.ts: writers (purge/delete/backup) reject while a restore's shared lock is held and a restore succeeds alongside another shared holder; a restore rejects while a writer holds the lock; the missing-directory error now also covers restore and asserts no directory is created.
  • Parent-directory tests flipped from expect-throw to expect-create in both suites.
  • All suites pass locally (macOS): backup + file-lock (38), backup-stream + backup-transaction-logs (19), native GTest (52). The Windows LockFileEx shared path follows documented semantics but is only exercised in CI.

Docs updated: README (backup locking note, backups.restore(), file-locking API), AGENTS.md backup-lock section, and the doc comments at each layer.

🤖 Generated with Claude Code

A backups.restore() previously took no lock on the backup directory, so
a concurrent db.backup(), backups.delete(), or backups.purge() could
delete the very backup a restore was copying from. The failure is
asymmetric: the default purgeAllFiles restore mode wipes the
destination before copying, so a restore failed mid-purge leaves no
usable database, while a rejected writer just retries.

tryFileLock/tryAcquireFileLock gain a shared (reader) mode — flock
LOCK_SH on POSIX, LockFileEx without LOCKFILE_EXCLUSIVE_LOCK on
Windows. Restore holds the .backup.lock in shared mode for its
duration: concurrent restores coexist, but writers and restores
mutually reject with the existing "locked" error. list/verify remain
unlocked so cheap listings don't reject during a long backup.

Also:
- tryFileLock now creates missing parent directories (the point of the
  function is to create the lock), and coerces its shared argument by
  truthiness. withBackupDirLock rejects a missing backup directory
  explicitly so delete/purge/restore on a typo'd path surface a clear
  error instead of conjuring an empty directory.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@cb1kenobi cb1kenobi requested a review from kriszyp as a code owner July 10, 2026 16:38

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for shared (reader) file locks alongside exclusive (writer) locks in the native file locking mechanism. This is utilized to allow concurrent database restores while preventing concurrent writers (such as backups, deletes, or purges) from modifying files during a restore. Additionally, the lock acquisition now automatically creates missing parent directories, and explicit checks are added to prevent creating empty directories for nonexistent paths. The review feedback highlights a critical issue in the Node-API binding code (src/binding/binding.cpp) where accessing argv[1] without checking argc can lead to undefined behavior or crashes if the second argument is omitted by the caller.

Comment thread src/binding/binding.cpp
@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

📊 Benchmark Results

get-sync.bench.ts

getSync() > random keys - small key size (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 24.57K ops/sec 40.71 39.20 600.889 0.116 122,830
🥈 rocksdb 2 11.50K ops/sec 86.98 83.93 4,490.481 0.180 57,483

getSync() > sequential keys - small key size (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 28.56K ops/sec 35.02 33.88 532.926 0.055 142,787
🥈 rocksdb 2 11.67K ops/sec 85.69 83.85 540.16 0.055 58,350

ranges.bench.ts

getRange() > small range (100 records, 50 range)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 24.08K ops/sec 41.53 36.37 3,741.756 0.324 120,389
🥈 rocksdb 2 16.32K ops/sec 61.27 54.82 1,112.645 0.122 81,613

realistic-load.bench.ts

Realistic write load with workers > write variable records with transaction log

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 rocksdb 1 437.83 ops/sec 2,284.012 172.437 55,706.568 13.11 876
🥈 lmdb 2 26.99 ops/sec 37,048.435 435.376 1,155,210.692 135.083 64.00

transaction-log.bench.ts

Transaction log > read 100 iterators while write log with 100 byte records

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 rocksdb 1 36.12K ops/sec 27.69 13.68 14,177.785 0.581 180,578
🥈 lmdb 2 449.09 ops/sec 2,226.737 73.57 13,650.096 1.20 2,246

Transaction log > read one entry from random position from log with 1000 100 byte records

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 rocksdb 1 720.77K ops/sec 1.39 1.17 508.573 0.060 3,603,862
🥈 lmdb 2 476.20K ops/sec 2.10 1.08 7,521.776 0.513 2,380,997

worker-put-sync.bench.ts

putSync() > random keys - small key size (100 records, 10 workers)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 rocksdb 1 840.15 ops/sec 1,190.262 1,037.311 2,811.193 0.330 1,681
🥈 lmdb 2 1.16 ops/sec 862,177.006 832,085.084 904,269.518 2.08 10.00

worker-transaction-log.bench.ts

Transaction log with workers > write log with 100 byte records

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 rocksdb 1 20.53K ops/sec 48.72 30.83 1,033.64 0.504 41,055
🥈 lmdb 2 820.76 ops/sec 1,218.384 282.269 12,674.775 5.06 1,643

Results from commit a0c1afc

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant