Hello,
I have three Mimir backend pods, running the compactor and store-gateway roles, running with replication factor=3. Each of the pods has a persistent volume of size 1Gi. The problem I'm having is that I see logs like:
2026-05-25T20:14:27.088Z [mimir] ts=2026-05-25T20:14:27.019411838Z caller=lazy_binary_reader.go:182 level=info user=anonymous msg="could not download sparse index-header from bucket; will reconstruct when the block is queried" err="getting sparse index-header from bucket: The specified key does not exist."
2026-05-25T20:14:27.088Z [mimir] ts=2026-05-25T20:14:27.065057295Z caller=bucket.go:450 level=error user=anonymous msg="loading block failed" elapsed=90.539934ms id=01KS4NCRN82HNRMXMV8ZGR0QW3 err="create index header reader: write index header: new binary index header writer: mkdir /data/tsdb-sync/anonymous/01KS4NCRN82HNRMXMV8ZGR0QW3: no space left on device"
I probably could resize the persistent volume, but I wonder if there is a better course of action here or whether this is a symptom that the compactor is failing to offload the sparse index-headers to S3 storage. Could lazy loading help here? Is there another way of reducing the number of index-headers being stored on disk?
Hello,
I have three Mimir backend pods, running the compactor and store-gateway roles, running with replication factor=3. Each of the pods has a persistent volume of size 1Gi. The problem I'm having is that I see logs like:
I probably could resize the persistent volume, but I wonder if there is a better course of action here or whether this is a symptom that the compactor is failing to offload the sparse index-headers to S3 storage. Could lazy loading help here? Is there another way of reducing the number of index-headers being stored on disk?