Is RocksDict compatible with RocksDB-Python?

[zjuzhfbloodz]

Here is my code, I am trying to save some sst files and read with rocksdb in python:

from rocksdict import Rdict, Options, SstFileWriter
import random
import rocksdb

# generate some rand bytes
rand_bytes1 = [random.randbytes(200) for _ in range(100000)]
rand_bytes1.sort()

# write to file1.sst
writer = SstFileWriter(options=Options(raw_mode=True))
writer.open("file1.sst")
for k, v in zip(rand_bytes1, rand_bytes1):
    writer[k] = v

writer.finish()

# bulid Rdict
d = Rdict("tmp", options=Options(raw_mode=True))
d.ingest_external_file(["file1.sst"])
d.close()

# rocksdb read
db = rocksdb.DB("tmp", read_only=True, opts=rocksdb.Options())

And I am getting the error as below:

---------------------------------------------------------------------------
Corruption                                Traceback (most recent call last)
remote%2B10.97.150.93/opt/huawei/data1/z00811450/code/test.ipynb#Y211sdnNjb2RlLXJlbW90ZQ%3D%3D?line=22) db = rocksdb.DB("tmp", read_only=True, opts=rocksdb.Options())

File /opt/huawei/wisebrain/anaconda3/lib/python3.9/site-packages/rocksdb/_rocksdb.pyx:1636, in rocksdb._rocksdb.DB.__cinit__()

File /opt/huawei/wisebrain/anaconda3/lib/python3.9/site-packages/rocksdb/_rocksdb.pyx:76, in rocksdb._rocksdb.check_status()

Corruption: b'Corruption: unknown checksum type 4 from footer of tmp/000019.sst, while checking block at offset 22672231 size 35'

Does this mean that rocksdict is not compatible with rocksdb in python? Looking forward to anyone's help!

[Congyuwang]

rocksdict uses rocksdb 8.0 for the moment. So it maybe incompatible to rocksdb 6 which is used by python-rocksdb

[Congyuwang]

But it should be compatible with other rocksdb clients, e.g., java, c++, rust

[zjuzhfbloodz]

Thank you congyu and for this package. May I ask if you have ever compared Rocksdict and Rocksdb on Python? like CPU usage, memory usage and latency. Our service has been used RocksDB-Python for a while and want to check if we can switch to Rocksdict.

[Congyuwang]

Hi. Since I can't really build rocksdb-python on my MacBook Air I haven't compared their performances. But I would be glad to see how these two compares, and if necessary based on the benchmark to improve it.

Notice that you will want to use raw_mode=True for good performance.