Failure creating btrfs filesystem on nbd device using Hetzner

[tve]

I created a bucket in Hetzner's object storage and started zerofs with:

export SLATEDB_CACHE_DIR='/var/cache/zerofs'
export SLATEDB_CACHE_SIZE_GB='10'
export ZEROFS_ENCRYPTION_PASSWORD='...'
export ZEROFS_MEMORY_CACHE_SIZE_GB='1'
export ZEROFS_NBD_PORTS='10809'
export ZEROFS_NBD_DEVICE_SIZES_GB='2000'
export RUST_LOG='zerofs=debug'

# Hetzner Nurenberg S3 API
export AWS_ENDPOINT='https://nbg1.your-objectstorage.com'
export AWS_ACCESS_KEY_ID='...'
export AWS_SECRET_ACCESS_KEY='...'

Launching zerofs went fine. I ran nbd-client and then mkfs.btrfs. It hung for ~4 minutes before spitting out an error:

> sudo mkfs.btrfs --checksum xxhash /dev/nbd0
btrfs-progs v6.15
See https://btrfs.readthedocs.io for more information.

Performing full device TRIM /dev/nbd0 (1.95TiB) ...
NOTE: several default settings have changed in version 5.15, please make sure
      this does not affect your deployments:
      - DUP for metadata (-m dup)
      - enabled no-holes (-O no-holes)
      - enabled free-space-tree (-R free-space-tree)

ERROR: error during mkfs: Operation not permitted
Took 4.2min

zerofs did spit out 4 errors around that time (this is from one of them):

2025-08-21T08:45:41.712640Z DEBUG zerofs::nbd::server: NBD command: Flush, offset=0, length=0
2025-08-21T08:46:51.364103Z ERROR zerofs::nbd::server: Error handling NBD client 127.0.0.1:43070: IO error: early eof

I unmounted the nbd device and restarted zerofs, this time it seems it has created the filesystem OK. I mounted the FS but at some point it failed:

2025-08-21T09:04:30.232714Z ERROR zerofs: Garbage collection failed: IoError

Maybe I need to try all this with AWS S3 first...

Update:

• I restarted zerofs
• I re-ran mkfs.btrfs
• I re-mounted the filesystem
• Now I can create a file...
• But a bit later (sitting idle) I got:

2025-08-21T09:22:29.764160Z DEBUG zerofs::nbd::server: NBD command: Flush, offset=0, length
=0                                                                                         
                                                                                           
thread 'tokio-runtime-worker' panicked at /home/runner/.cargo/git/checkouts/slatedb-a6e7398
2df30678a/6e13868/slatedb/src/compactor.rs:297:18:                                         
fatal error loading manifest: ObjectStoreError(Generic { store: "S3", source: ListResponseB
ody { source: HttpError { kind: Interrupted, source: reqwest::Error { kind: Body, source: r
eqwest::Error { kind: Decode, source: reqwest::Error { kind: Body, source: hyper::Error(Bod
y, Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }) } } } } } 
})                                                                                         
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace              
2025-08-21T09:22:34.248958Z ERROR zerofs::fs::flush_coordinator: Automatic flush failed: Er
r(IoError)                                                                                 

[Barre]

What nbd-client command did you use to connect the device? If the s3 provider is little flaky or that you have low throughput to it you probably want something higher as a timeout than what the docs recommend (say 600 or no timeout at all).

Can you share dmesg output? That'd help confirm if it's timeout related. Most filesystems will hate any disconnection of the device.

[Barre]

Your second issue with connection reset by peer is related to slatedb/slatedb#766 which should get fixed soon. It seems that hetzner specifically throws a lot of these.

[tve]

I was using the nbd-client command from the docs:

nbd-client 127.0.0.1 10809 /dev/nbd0 -N device_10809 -persist -timeout 60 -block-size 4096 -connections 4

I can easily increase the timeout...
I can't pick out the dmesg errors that pertain to the above test, but will post the next time.
Thanks for all the replies!!!

[Barre]

Indeed 60 secs seems low, zerofs retries internally and with back off and a bit of backpressure you'll definitely go past that. I'll amend the docs to recommend something higher.

It might be worthwhile to try to contact hetzner's support about this, they have an almost strong product but this kind of stuff is a bummer, indeed.

[tve]

[maybe this should be moved to discussions?]

Continuing my btrfs woes... :-)

I switched to AWS (but it's still high latency). When creating the filesystem the initial flush after all the trimming and zeroing takes a very long time and nbd times out, but it does retry and eventually succeed. I was then able to mount the filesystem and write & read data. Eventually things failed while the filesystem was idle. Here's a successful autoflush followed by the first error. After that it's all errors.

2025-08-21T17:33:25.014306Z DEBUG zerofs::fs::flush_coordinator: Auto-flush triggered      
2025-08-21T17:33:25.356657Z DEBUG zerofs::fs::flush_coordinator: Automatic flush completed 
successfully                                                                               
2025-08-21T17:34:25.014098Z DEBUG zerofs::fs::flush_coordinator: Auto-flush triggered
2025-08-21T17:34:34.033783Z ERROR zerofs::fs::flush_coordinator: Automatic flush failed: Er
r(IoError)
2025-08-21T17:34:45.399238Z ERROR zerofs: Garbage collection failed: IoError

and sometime later:

thread 'tokio-runtime-worker' panicked at /home/runner/.cargo/git/checkouts/slatedb-a6e7398
2df30678a/6e13868/slatedb/src/compactor.rs:297:18:                                         
fatal error loading manifest: ObjectStoreError(Generic { store: "S3", source: ListResponseB
ody { source: HttpError { kind: Interrupted, source: reqwest::Error { kind: Body, source: r
eqwest::Error { kind: Decode, source: reqwest::Error { kind: Body, source: hyper::Error(Bod
y, Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }) } } } } } 
})

I'm not seeing anything in dmesg around that time (and I wouldn't expect since there is no filesystem/nbd activity).

Do I gather correctly that after this error I have to tear the mount, nbd-client, and zerofs down and restart from scratch?

[Barre]

@tve Are you using ZeroFS from aws itself? If not, what is your local network throughput to AWS like? Ultimately ZeroFS still has to push data there, "taking a long time" seems to mean you just don't have the network capacity to handle whatever you are requesting ZeroFS to do.

[tve]

Yes, it is a lowish bandwidth long distance link, and yes, I understand it will be slow. That seemed like a good way to stress test ZeroFS... The question really is whether the backpressure from the network works all the way through and whether the (inevitable) network errors are fully retried.

The test I'm doing is cloning a 20GB btrfs subvolume (btrfs send | btrfs receive) and the link seems to run at ~30Mbps. So it's going to take a while, which is totally fine. So far btrfs is only mildly annoyed:

[  +0.000001] INFO: task btrfs:1392600 blocked for more than 122 seconds.                  

[Barre]

The question really is whether the backpressure from the network works all the way through and whether the (inevitable) network errors are fully retried.

Aside from the "reset by peer" stuff, this should work well (albeit slow) once this is fixed. If your use case allows for it, you'll probably have better luck with 9p as there's going to be less overhead.

[tve]

Something I'm observing is that the read bandwidth is very high, visually more than half the write bandwidth. So this is not going to work from a cost point of view (S3 egress costs). Oh well...

[Barre]

Something I'm observing is that the read bandwidth is very high, visually more than half the write bandwidth. So this is not going to work from a cost point of view (S3 egress costs). Oh well...

I'd go Cloudflare r2 (which has free egress) if you want to keep your setup local, or have a very small vm at AWS that hosts ZeroFS if you want to keep using AWS S3. You can reduce S3 reads by increasing your local disk cache too.