I get a "Too many open files" error

[orena1]

Hi,
I get this error when tryin to write to a file, I actually have many DB files open, one for each date, and I get:
OSError: Too many open files (os error 24)
I use WriteBatch and write it every 25k samples to accelerate the writetime

[Congyuwang]

I don't think it is a good idea to have MANY DBs. You just create one DB. Encode your date information into your key. Say b'20240625-somekey'. Then you may want to use column families for different types of data, which also eliminates the need for multiple instances of DBs. Try this, it is more optimal.

[Congyuwang]

Even like this you can choose to iterate over just a single date, by seeking to, say, b'20240625'.

[orena1]

How can I seek by date? what do you suggest?

[Congyuwang]

Say you have keys b“20140101-key1”, b“20140101-key2”. Just use the seek method of the iterator to b”20140101”, then the cursor is at the first key with prefix 20140101.

[Congyuwang]

BTW, how many 'dates' do you have.

[orena1]

about 40 currently

[orena1]

I have only 40 dates, I am afraid of corruption and the need to recreate the full dataset if my write stops in the middle, so it will be easier to recreate a single date than to recreate the full dataset. What do you think?

[orena1]

Hi @Congyuwang thanks a lot!
I've tried only with uuid as you suggested using binary and I setll got the same error:

Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0

Uptime(secs): 2446.7 total, 600.0 interval
Flush(GB): cumulative 84.699, interval 31.899
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 897.08 GB write, 375.44 MB/s write, 829.38 GB read, 347.11 MB/s read, 1097.2 seconds
Interval compaction: 264.74 GB write, 451.81 MB/s write, 243.78 GB read, 416.05 MB/s read, 300.0 seconds
Estimated pending compaction bytes: 179605288705
Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 0, pending-compaction-bytes-delays: 1460, pending-compaction-bytes-stops: 1979, total-delays: 1460, total-stops: 1979
interval: 1035 total count
Block cache LRUCache@0x15500d0#510448 capacity: 8.00 MB seed: 63466949 usage: 0.08 KB table_size: 256 occupancy: 1 collections: 5 last_copies: 0 last_secs: 5.3e-05 secs_since: 0
Block cache entry stats(count,size,portion): Misc(1,0.00 KB,0%)

** File Read Latency Histogram By Level [default] **
2024/07/03-15:15:27.189969 140573527545664 [db/db_impl/db_impl_compaction_flush.cc:1977] [default] Manual flush start.
2024/07/03-15:15:27.189988 140573527545664 [db/db_impl/db_impl_compaction_flush.cc:1988] [default] Manual flush finished, status: IO error: While open a file for random read: .../rocksdict_one/full_dataset_boldFalse_addDateFalse/277234.sst: Too many open files
2024/07/03-15:15:27.189991 140573527545664 [db/db_impl/db_impl.cc:488] Shutdown: canceling all background work
2024/07/03-15:15:27.190186 140573527545664 [db/db_impl/db_impl.cc:488] Shutdown: canceling all background work
2024/07/03-15:15:28.084892 140573527545664 [db/db_impl/db_impl.cc:681] Shutdown complete

Any idea? thanks

This is how I write the files:

from rocksdict import Rdict
from tqdm.auto import tqdm
from rocksdict import WriteBatch
from rocksdict import Options
import io
import pandas as pd
from rocksdict import AccessType
import numpy as np


bolb = False
add_date = False
opt = Options(raw_mode=True)
opt.set_max_open_files(450_000)
if bolb:
    opt.set_enable_blob_files(True)

db_path = f'full_dataset_bold{bolb}_addDate{add_date}'
print(db_path)
db = Rdict(db_path,options=opt)
all_tarids = set(list(db.keys()))


spl_written = 0
write_every = 25000
wb = WriteBatch(raw_mode=True)
for recording_date, run_path, uuid in tqdm(df_to_save_sample[['recording_date','uuid']].values):
    if add_date:
        key = f'{recording_date}-{uuid}'.encode()
    else:
        key = str.encode(str(uuid))
        
    if key not in all_tarids:
        with open(f'/mnt/npy_files/{uuid}.npy', 'rb') as fid:
            wb.put(key, fid.read())
            spl_written+=1

    if spl_written==write_every:
        print("Start Writting",flush=True)
        db.write(wb)
        wb = WriteBatch(raw_mode=True)
        spl_written = 0
        print("End Writting",flush=True)
    
db.write(wb)   

[Congyuwang]

First try to use

ulimit -n

To check your per process open file limit.

[Congyuwang]

Say you have ulimit as 1024. Then try opt.set_max_open_files(1024) to make sure that rocksdb does not open more than 1024 files which triggers "Too many open files" error.

[Congyuwang]

If this still does not work, try to increase user limits:

You can increase the limits temporarily using ulimit, or you can make permanent changes by editing configuration files.

• Temporarily (current session):

  ulimit -n 4096  # Example: set limit to 4096

before running your program.

• Permanently:
  Edit the /etc/security/limits.conf file and add or modify the following lines:

  username soft nofile 4096
  username hard nofile 8192

  Replace username with the actual username.

  You can also apply changes system-wide:

  * soft nofile 4096
  * hard nofile 8192