Concurrent PM bug

[Jonyleo]

While developing a concurrency PM bug detecting tool, we have detected 3 bugs in WIPE.

WIPE's get operations execute in a lock-free manner, and as far as we are aware, no other mechanism exists to guarantee data retrieved by a get operations is explicitly persisted (via flush fence).
Furthermore, a missing flush in node expansion results in potential data loss.

To aid the developers, we provide some of the stack traces for the PM accesses involved in this bug.

using UnSortBuncket = letree::UnSortBuncket<256ul, 8ul, 8ul, 64ul>;

First bug - Put/Delete key races with Get

PM stores:

UnSortBuncket::PutBufKV (/root/WIPE/test/../src/pointer_bentry.h:1771)
UnSortBuncket::Put (/root/WIPE/test/../src/pointer_bentry.h:1937)
PointerBEntry::Put (/root/WIPE/test/../src/pointer_bentry.h:2506)
group::Put (/root/WIPE/test/../src/letree.h:289)
Put (/root/WIPE/test/../src/letree.h:812)

UnSortBuncket::remove_key (/root/WIPE/test/../src/pointer_bentry.h:1799)
UnSortBuncket::Expand_ (/root/WIPE/test/../src/pointer_bentry.h:1894)
PointerBEntry::Put (/root/WIPE/test/../src/pointer_bentry.h:2522)
group::Put (/root/WIPE/test/../src/letree.h:289)
Put (/root/WIPE/test/../src/letree.h:812)

PM load

UnSortBuncket::Find (/root/WIPE/test/../src/pointer_bentry.h:1606)
UnSortBuncket::Get (/root/WIPE/test/../src/pointer_bentry.h:1973)
PointerBEntry::Get (/root/WIPE/test/../src/pointer_bentry.h:2441)
group::Get (/root/WIPE/test/../src/letree.h:313)
group::fast_fail (/root/WIPE/test/../src/letree.h:343)
find_fast (/root/WIPE/test/../src/letree.h:905)
Get (/root/WIPE/test/../src/letree.h:847)

Second bug - Put/Delete value races with Get

PM stores:

UnSortBuncket::SetValue (/root/WIPE/test/../src/pointer_bentry.h:1550)
UnSortBuncket::Update (/root/WIPE/test/../src/pointer_bentry.h:1963)
PointerBEntry::Update (/root/WIPE/test/../src/pointer_bentry.h:2430)
group::Update (/root/WIPE/test/../src/letree.h:356)
Update (/root/WIPE/test/../src/letree.h:835)

UnSortBuncket::PutBufKV (/root/WIPE/test/../src/pointer_bentry.h:1772)
UnSortBuncket::Put (/root/WIPE/test/../src/pointer_bentry.h:1937)
PointerBEntry::Put (/root/WIPE/test/../src/pointer_bentry.h:2506)
group::Put (/root/WIPE/test/../src/letree.h:289)
Put (/root/WIPE/test/../src/letree.h:812)

PM load:

UnSortBuncket::Get (/root/WIPE/test/../src/pointer_bentry.h:1601)
PointerBEntry::Get (/root/WIPE/test/../src/pointer_bentry.h:2441)
group::Get (/root/WIPE/test/../src/letree.h:313)
group::fast_fail (/root/WIPE/test/../src/letree.h:343)
find_fast (/root/WIPE/test/../src/letree.h:905)
Get (/root/WIPE/test/../src/letree.h:847)

Third bug - Expansion after Put races with other Put

When a node expands, it allocates a bigger a buffer of PM for its entries "new_entry_space", setting "entry_space" to the pointer.
This pointer is never persisted.
Even though Put operations on the same node are protected by a common mutex, subsequent Put/Update/Delete operations will perform modifications to the newly allocated entries.
However, after a crash, since the "entry_space" pointer was never explicitly persisted, the value may revert back to its old value, losing every operation since the node expansion.

PM store:

group::expand (/root/WIPE/test/../src/letree.h:393)
group::Put (/root/WIPE/test/../src/letree.h:303)
Put (/root/WIPE/test/../src/letree.h:812)

[Jonyleo]

Pinging this issue