LMDB SIGSEGV on freelist corruption under high-throughput sharded writes

[kriszyp]

Problem

Under extreme sharding write load (90k req/s across a 5-node cluster, 3× k6 servers at 30k req/s each), LMDB crashes with a SIGSEGV originating from freelist corruption.

Log excerpt preceding the crash:

overlapping duplicate entry 37910899
freelist duplicates/overlaps in free list 1959654 0 1959654 1959664 1959654 1978075 1 1 2147483647 2147483647 11 1 1
conflicting freelist to save:
freelist: 30/254: _-5144654069346994375-8751527078568789555 ...

Full SIGSEGV stack in @lmdb/lmdb-linux-x64:

PID 1 received SIGSEGV for address: 0x7f84042b5000
@lmdb/lmdb-linux-x64/node.abi127.glibc.node(+0x9cb91)
@lmdb/lmdb-linux-x64/node.abi127.glibc.node(+0x890ea)
@lmdb/lmdb-linux-x64/node.abi127.glibc.node(+0x8a66e)
@lmdb/lmdb-linux-x64/node.abi127.glibc.node(+0x54d25)
...

Reproduction

Performance test repo: https://github.com/HarperDB/sharded-blob-test

Triggered by multiRightWrites.js (1 KB record size, 5-node sharded cluster, 4.6.0-alpha.3).

Impact

Server process terminates (SIGSEGV) under stress. Occurs at write rates that exceed the freelist's ability to maintain consistency — likely a race or overflow in LMDB's page recycling under concurrent write pressure.

Next steps

• Investigate whether upgrading lmdb (currently 3.5.4) resolves the freelist overlap detection
• Determine if our transaction/write batching strategy can reduce concurrent write pressure on the freelist
• Consider whether the sharding write path needs back-pressure to stay below LMDB's concurrency limits

🤖 Filed by Claude on behalf of Kris.