Skip to content

feat: add return_file_name option to JSON loader - #8378

Open
RenhaoSong wants to merge 3 commits into
huggingface:mainfrom
RenhaoSong:feat/return-file-name
Open

feat: add return_file_name option to JSON loader#8378
RenhaoSong wants to merge 3 commits into
huggingface:mainfrom
RenhaoSong:feat/return-file-name

Conversation

@RenhaoSong

Copy link
Copy Markdown

What does this PR do?

Adds a return_file_name option to the JSON loader. When enabled, the resulting dataset includes a file_name column indicating the source file of each row.

Motivation

When training is interrupted and resumed, it's useful to know which file each row came from so already-trained shards can be skipped.

Tests

Added tests covering both enabled and default-disabled behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant