There's currently no way for pebble to inform the application that a batch is enqueued, which would be useful to provide stronger semantics especially when writing with Sync: true. An example may best motivate the use case.
Consider an application that handles many tenants, where the desire is that an individual tenant's writes are sequenced. To improve performance, a tenant's writes may be buffered in a batch that is periodically applied to the DB. In this case, the data resides on a NAS where slow syncs may be on the order of tens of milliseconds (or worse). To maintain sequencing, the application must apply the tenant's first batch in full (i.e. the call must return) before it may attempt to apply a subsequent batch.
The reason it must wait for the write to finish is that there's an inherent race condition between finalizing a batch (at the application level) and applying it to the DB. In the example, if the application doesn't wait for the first batch apply to succeed, it wouldn't be impossible (although unlikely) for the second batch to be applied before the first batch without some form of synchronization. The obvious workaround would be to enclose the batch finalizing + apply to DB in a lock, but that must wait for the write to finish before unlocking.
If there was a way for the DB indicate that a batch is enqueued (but not yet necessarily applied), a func(err error) callback could unlock the mutex instead before writing and syncing to storage. Note that assumption would be that, once the callback is invoked, the write will eventually succeed (or the application will fail without subsequent batches being applied).
There's currently no way for pebble to inform the application that a batch is enqueued, which would be useful to provide stronger semantics especially when writing with
Sync: true. An example may best motivate the use case.Consider an application that handles many tenants, where the desire is that an individual tenant's writes are sequenced. To improve performance, a tenant's writes may be buffered in a batch that is periodically applied to the DB. In this case, the data resides on a NAS where slow syncs may be on the order of tens of milliseconds (or worse). To maintain sequencing, the application must apply the tenant's first batch in full (i.e. the call must return) before it may attempt to apply a subsequent batch.
The reason it must wait for the write to finish is that there's an inherent race condition between finalizing a batch (at the application level) and applying it to the DB. In the example, if the application doesn't wait for the first batch apply to succeed, it wouldn't be impossible (although unlikely) for the second batch to be applied before the first batch without some form of synchronization. The obvious workaround would be to enclose the batch finalizing + apply to DB in a lock, but that must wait for the write to finish before unlocking.
If there was a way for the DB indicate that a batch is enqueued (but not yet necessarily applied), a
func(err error)callback could unlock the mutex instead before writing and syncing to storage. Note that assumption would be that, once the callback is invoked, the write will eventually succeed (or the application will fail without subsequent batches being applied).