Use local metadata file generation to check HF repo

[hlapp]

My understanding (from Imageomics/Collaborative-distributed-science-guide#65) is that sum-buddy can generate a file metadata csv for a local (multi-file, multi-directory) dataset.

Maybe it would be useful to add the capability to use this local, or locally generated metadata file to check a HF upload for completeness.

[thompsonmj]

This is a great idea. This could be the ground floor of a broader verify subcommand that could work against multiple remote and local targets and would be an effective solution for that issue you linked to.

Starting with HF, we could compare the local manifest file contents by filename to the remote dataset contents through huggingface_hub by filename, and raise meaningful asymmetries.

Thinking through what meaningful vs expected asymmetries look like ...

Meaningful and worth raising:

• Missing on HF, present in manifest: almost certainly a failed upload.
• Extra on HF, missing from manifest: probably a .sbignore misconfiguration? Maybe something legitimate that HF added.

Output buckets:

• [OK] files match
• [MISSING_IN_TARGET] in manifest, not on HF (likely upload failed)
• [EXTRA_IN_TARGET] on HF, not in manifest (check .sbignore or HF defaults)
• [SKIPPED] virtual archive members (contents in manifest but not listed by API response).

We could use huggingface_hub.list_repo_tree() to get paths for the entire repo.

For future targets (Zenodo, S3, OSF, local-vs-local), the comparison logic would stay the same; each target just needs a small adapter implementing "list files".

Does the [EXTRA] bucket sound useful to you, @hlapp, or is just [MISSING] enough for the failed-upload case you had in mind?

Deferred:
There seems to be a path to getting cheap byte-perfect validation if we add a few features to sum-buddy. huggingface_hub.list_repo_tree() will return file size info:

So for another issue, add size detection to the manifest output, and integrate that check here.

HF also stores blob SHA-1 and SHA-256 (LFS) of repo contents. We could add a sum-buddy feature to save multiple hashes to its manifest. This could support multi-purpose uses of the same manifest. E.g. someone uses MD5 for quick local duplicate/change detection (current sum-buddy default), then ALSO has SHA-256 in the same manifest for byte-perfect verification against HF LFS-tracked files, AND a git-blob-SHA-1 column for byte-perfect verification against HF non-LFS files. One trip through the data by sum-buddy with multiple downstream uses. CLI might be --algorithm md5,sha256,git-blob-sha1 with one column per algorithm in the CSV.

So also for another issue, multiple hash algorithm support and integration here with what HF provides on git blobs and LFS objects.

[hlapp]

Do we have a good use-case for the EXTRA_IN_TARGET bucket? I suppose in theory we could be checking a local download of a HF repo's contents, but the hf CLI does this already, doesn't it?

[egrace479]

@thompsonmj, for reference, I have a suggested "integrity check" in the update that @hlapp mentioned. It's purely filename-based. Including an SHA-256 suggested check could be valuable; however, it likely wouldn't work for non-LFS (or Xet) files (example CSV).

[thompsonmj]

@egrace479 I think a verify subcommand in here would remove the need for a downstream custom snippet such as that. That is correct on the SHA-256 not working for non-LFS. I was playing with this and it would work like this:

• The HF API will give you SHA-256 on the file data for LFS objects
• The HF API will give you a SHA-1 on the file bytes plus a string header "blob {byte_size}\0" prepended to the bytes (source). So we could add git-blob-sha1 as one of the supported hash algorithms that prepends this and calculates the appropriate hash. It could be compared for free to what the HF API returns.

@hlapp slightly reframing the EXTRA_IN_TARGET angle ... sum-buddy gives visibility into local data for dedup, integrity, etc. A verify subcommand extends that to comparison to a remote (or target in general), showing overlap and differences. What a user would do with each bucket is up to them. But having each bucket of "what's in the metadata list only", "what's in the target only", and "what's in both" feels like a complete feature.

[egrace479]

@egrace479 I think a verify subcommand in here would remove the need for a downstream custom snippet such as that. That is correct on the SHA-256 not working for non-LFS. I was playing with this and it would work like this:

• The HF API will give you SHA-256 on the file data for LFS objects
• The HF API will give you a SHA-1 on the file bytes plus a string header "blob {byte_size}\0" prepended to the bytes (source). So we could add git-blob-sha1 as one of the supported hash algorithms that prepends this and calculates the appropriate hash. It could be compared for free to what the HF API returns.

@hlapp slightly reframing the EXTRA_IN_TARGET angle ... sum-buddy gives visibility into local data for dedup, integrity, etc. A verify subcommand extends that to comparison to a remote (or target in general), showing overlap and differences. What a user would do with each bucket is up to them. But having each bucket of "what's in the metadata list only", "what's in the target only", and "what's in both" feels like a complete feature.

This would also line up with some of the cautious-robot buddy_check functionality, which has sum-buddy as a dependency, so we may want to consider their interaction in this setup as well.

[hlapp]

I would just say that (at least so far) for our types of use-cases we receive data not by means of an HF dataset, which we therefore wouldn't have put through any QC yet, but instead we receive data, put it through various QC, consolidations, etc and then put them up on HF. So within that scope, there would be no such thing as needing to check for dups etc for a HF dataset of ours.

We could expand that scope, but given where we're at in the Institute's lifecycle, I'm reluctant to go there (i.e., I'd rather focus on polishing up what is within our existing scope) unless there is some kind of late-breaking use-case.

[thompsonmj]

Agreed, all of those use cases (dedup, integrity, etc.) are downstream capabilities enabled by sum-buddy.

Sum-buddy's only job thus far is creating a metadata/manifest file of a dataset.

But now,

Maybe it would be useful to add the capability to use this local, or locally generated metadata file to check a HF upload for completeness.

This is a late-breaking use case (comparing two lists, local and HF). I'm just trying to understand what shape it should take and what it should share with the user.

So when two lists are compared, we get three buckets: what's only in A, what's only in B, and what's in both. Membership from each list in these buckets defines various states relevant to the user.
States based on bucket contents:

• If everything in list A and list B is present in both, that definitely means OK.
• If something is only in either A or B, that's worth telling the user and letting them decide what to do with it (e.g. MISSING_FROM_TARGET or EXTRA_IN_TARGET, could use different names than these).
• If something in A can't exist in B (spilled contents of an archive file), SKIPPED.
  👉 I think handling each of the 3 buckets and communicating states to the user would go in a PR for this issue.

So then we just need to decide what info to include in the buckets:

• Just comparing file names between the two? (a reasonable start; sufficient for addressing this issue I'd say)
• Comparing file sizes and hashes? Data on HF provides hashes (SHA-256 for LFS, SHA-1 for git blobs), so that side comes for free. These bits of functionality are pretty natural for sum-buddy, perhaps as a quick series of separate Issues for expanding the capability of sum-buddy to 1) provide file size in the output; 2) enable multiple hashes at the same time, adding a git-blob-sha1; 3) extending the verify feature to use these; 4) extending the verify feature to multiple targets.
  👉I think just doing it by filename would go into a PR for this issue.

@egrace479 yes, it will definitely be good to coordinate between sum-buddy and cautious-robot's BuddyCheck.

[hlapp]

Maybe it would be useful to add the capability to use this local, or locally generated metadata file to check a HF upload for completeness.

This is a late-breaking use case (comparing two lists, local and HF).

I don't think it's late-breaking, and I don't think it's "comparing two lists, local and HF". The use-case that we upload datasets from local to HF is far from new, and the question has been for a while, how can we automatically, efficiently and with high confidence check that what's on local has all made it up to HF.

[thompsonmj]

A filename-only "is A fully contained in B?" check, scoped to HF, sits oddly in sum-buddy. Sum-buddy's job so far has been general-purpose manifest generation. This bolts a specific verification workflow onto it without the surrounding capability (size, hashes, target abstraction) that would make verification more than a shallow filename check. The narrow version answers the immediate use case, but it commits the tool to 'verify' functionality without the pieces that make 'verify' actually do much.

As @egrace479 has alluded, cautious-robot's BuddyCheck already implements the "compare two manifests, surface what's missing" logic as a df merge on filename and/or checksum. Since cautious-robot depends on sum-buddy, that capability is already in the ecosystem. This functionality could be moved upstream into sum-buddy and positioned to extend organically.

I'd rather make a verify subcommand structured as target-agnostic to begin with, even if HF is the only adapter and filenames are the only comparison. I believe this satisfies the essence of this Issue and leaves room for size, hashes, and additional targets to slot in later without reshaping things.

[hlapp]

Sounds good, though now I'm wondering whether you're suggesting this better belongs elsewhere. I did not mean to say it seems like a great fit here; I was merely going by the documentation in the Collab Guide, which gives a piece of python code for an integrity check and says sum-buddy can generate the file metadata input for it.

[thompsonmj]

To clarify, I think this direction is a great fit for sum-buddy, and whether it belongs depends on how it's shaped. This feature introduces a new axis of functionality into the package: verification rather than just manifest file generation.

When a feature opens a new orthogonal direction like this, I think its first implementation should both establish that direction cleanly and take a discrete step in that direction. For this:

• The direction is a 'verify' (or other name) interface with target abstraction and joins on certain keys for comparison.
• The discrete step is defined by this narrow use-case: HF as the first supported target and filenames as the first join key and a full outer join to share all information without the user potentially needing to do follow-up operations.