Excessive disk usage

[adisbladis]

Chunks are written as 1 file per chunk, in a single flat .xit/chunks/.

This has two problems.

• No chunk file will ever take less than the file system block size, amplifying the repo size.

It's especially problematic for large repos with many small files, such as https://github.com/nixos/nixpkgs.
The solution here is probably a Git packfile like mechanism.

• Flat directory

The directory structure should be sharded.

[xeubie]

Yep you're right that there needs to be a way to bundle chunks together. It doesn't need to be as complicated as git packfiles though. Even just a simple mechanism to append chunks together, perhaps based on partially matching hashes, would work. I've been thinking about this for haxy.

Regarding your second point about sharding the directory, are you sure that matters for modern filesystems? I know git's approach of sharding based on the first two hex characters was necessary for older filesystems that had perf issues with giant directories but is that still necessary?

[adisbladis]

Even just a simple mechanism to append chunks together, perhaps based on partially matching hashes

I was thinking along the same lines:
You can reuse a packfile until a high water mark, let's say at a couple of megabytes, and then roll over to the next one.
The db would contain a reference like (hash, pack_id, offset, len) for each chunk.

Compaction/gc would need to rewrite the packfiles and any db references.

I've been thinking about this for haxy.

One thing to consider for a forge is that you probably want shared chunk storage across all hosted repos?
Actually that may be a feature that I would want locally too: A centralised chunk store that dedups not just the current repo, but between repos.

are you sure that matters for modern filesystems

Not as much as it used to, and would probably depend on the behaviour of any gc/compacting. If the chunk store needs to be enumerated with readdir things will get very slow, but if the compaction is purely driven by db contents you're probably right.

[xeubie]

This is now fixed in bba40ef

Now .xit/chunks is a single file instead of a directory. All chunks are stored there. I could eventually shard it across several files, but I am going for the simplest solution right now.

BTW this change already supports sharing chunks between repos. You can just symlink .xit/chunks for any repo to a centralized location and they will use it. The is a test showing this. The downside is that they need to open a write lock on the file so it will add contention. For haxy I will probably just make forks share the same chunks file, rather than all repos.