Skip to content

fix(tokenizer): honor max_size bytes and default files_per_process#303

Open
Chessing234 wants to merge 1 commit into
allenai:mainfrom
Chessing234:fix/272-files-per-process-default
Open

fix(tokenizer): honor max_size bytes and default files_per_process#303
Chessing234 wants to merge 1 commit into
allenai:mainfrom
Chessing234:fix/272-files-per-process-default

Commits

Commits on Jun 24, 2026