Is there a fast way to get all keys available on the DB

[orena1]

The only method that I say is:
list(Rdict(f'...DBS/rocksdict/{date}',options=opt).keys())
but it takes a lot of time, from what I understand it iterate over all the data.
Thanks,

[Congyuwang]

I think this already is the fastest way. Use raw mode too.

[Congyuwang]

All data have to be retrieved from the disk, since this is by design what rocksdb does. It assumes that there is difficulty storing everything inside main memory.

However, from what I can see, in your use case, there is no difficulty keeping all keys within main memory. So, what you can do is perhaps just keep all your keys in memory, and also perhaps separately deserialize all your keys in another format for fast batch retrieval.

[Congyuwang]

BTW, I don't understand the specific need about your use case yet. Why reading all keys into a list?

[Congyuwang]

You may also want to try BLOB DB which is supported in pip install rocksdict==0.3.24b3 if you have large values and smaller keys, via Options.set_enable_blob_files.

It is possible that this can improve the efficiency of your use case.

[orena1]

Thank you very much for all the information, I'll try your suggestions.
The use case is the following - I have many samples that are created from a different department and I download then and inset to rockdb, I want to filter the samples that already exists and only retrieve and add the samples that don't already exists in the DB.

BTW, why do you suggest to use raw mode, it that better ? thanks

[Congyuwang]

Raw mode generally performs better, and you can have your own way of turning objects into bytes. So you have more control. Non-raw-mode uses pickle by default, which isn’t always the most optimal.

[orena1]

Thanks @Congyuwang
I've tried using set_enable_blob_files=True but still doing:

opt = Options(raw_mode=True)
opt.set_max_open_files(450_000)
opt.set_max_background_jobs(2)
opt.set_enable_blob_files(True)
db_read = Rdict(f'/mnt/NVM/oren/DBS/rocksdict_one/full_dataset/',options=opt,access_type=AccessType.read_only())
all_tarids = set(list(db_read.keys()))

took a long time. Maybe I missing something?

[Congyuwang]

How many keys do you have? And how long does it take? Also on average how many bytes are the keys?