Containerd 2.0 removed LimitNOFILE=infinity from its service file, resulting in allowing only 1024 open files in the docker container. See containerd/containerd#8924
This leeds to msg="Error writing snapshot from compactor" engine=tsm1 op_name=tsm1_cache_snapshot error="compaction in progress: open /var/lib/influxdb/data/db/retention/333/002671920-000000001.tsm.tmp: too many open files". This then triggers influxdata/influxdb#26073, when using a not-uptodate version, resulting in filling up the disk.
The non-docker versions set LimitNOFILE=65536 in its service file (respectively ulimit -n 65536 in init.sh). I think this is also needed in Dockerfile.
The issue has been seen with influxdb:1.8.10-alpine, but probably applies to all versions.
Containerd 2.0 removed
LimitNOFILE=infinityfrom its service file, resulting in allowing only 1024 open files in the docker container. See containerd/containerd#8924This leeds to
msg="Error writing snapshot from compactor" engine=tsm1 op_name=tsm1_cache_snapshot error="compaction in progress: open /var/lib/influxdb/data/db/retention/333/002671920-000000001.tsm.tmp: too many open files". This then triggers influxdata/influxdb#26073, when using a not-uptodate version, resulting in filling up the disk.The non-docker versions set
LimitNOFILE=65536in its service file (respectivelyulimit -n 65536in init.sh). I think this is also needed inDockerfile.The issue has been seen with influxdb:1.8.10-alpine, but probably applies to all versions.