Skip to content

Include file size in dataset fingerprint - #8338

Open
yuxin00j wants to merge 3 commits into
huggingface:mainfrom
yuxin00j:include-file-size-1784174977
Open

Include file size in dataset fingerprint#8338
yuxin00j wants to merge 3 commits into
huggingface:mainfrom
yuxin00j:include-file-size-1784174977

Conversation

@yuxin00j

@yuxin00j yuxin00j commented Jul 16, 2026

Copy link
Copy Markdown

Motivation

Currently, the dataset cache fingerprints rely entirely on the ETag/etag/mtime metadata fetched during fs.info(). If a file's byte size changes on the remote end—and the remote backend fails to reliably update the ETag—the dataset fingerprint stays improperly static. This leads to insidious bugs where an old, structurally distinct dataset cache continues mapping to the new remote file.

Summary of Changes

  • Modified _get_single_origin_metadata in src/datasets/data_files.py:
    Checks if size is present in the underlying filesystem's info payload, and appends it directly to the ETag string ({ETag}-{size}). If the size doesn't exist, it elegantly falls back to the original ETag string structure.
  • Updated tests/test_data_files.py:
    Implemented test_get_single_origin_metadata_includes_size using a mocked url_to_fs patch to guarantee that the size byte integer correctly concatenates to the final string tuple emitted by _get_single_origin_metadata().

Testing

Tested unit test logic behavior locally by verifying that the mock tuple returns ("my_etag_123-1024",).

@yuxin00j
yuxin00j marked this pull request as draft July 16, 2026 04:13
@yuxin00j
yuxin00j marked this pull request as ready for review July 16, 2026 05:44
Comment thread tests/test_data_files.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants