Add Japanese 3B training configs#35
Open
iamtatsuki05 wants to merge 34 commits into
Open
Conversation
Develop to Main
Sync Develop to Main
Sync: Develop to Main
Sync: Develop to Main
- Created JSON configuration files for Sentence-Llama-Bi-3B, Sentence-ModernBERT-3B, and Sentence-Sarashina-Bi-3B models for benchmarking with the wiki40b_ja dataset. - Added isotropic evaluation configurations for Llama-3B, Llama-Bi-3B, ModernBERT-3B, and Sarashina-3B models, specifying parameters for evaluation. - Introduced PT evaluation configurations for Llama-Bi-JP-3B and Llama-JP-3B models across JCoLA, JNLI, and JSTS tasks, including training and evaluation settings. - Ensured all configurations include necessary parameters such as model paths, output directories, and training hyperparameters.
…aining configurations
- Added support for JGLUE tasks in the configuration, including a new function to retrieve benchmark dataset specifications. - Introduced dataset revision parameter in DataTrainingArguments for better dataset version control. - Expanded CLIConfig in sentence model to include parameters for positive sentence pairs datasets. - Updated dataset preparation logic to handle positive pairs from specified datasets. - Added unit tests for JGLUE configuration to ensure correct dataset retrieval and error handling. - Updated dependencies in the project lock file to include new packages and versions.
…ModernBERT models
…ning configurations
…d Japanese - Introduced new JSON configuration files for Llama and ModernBERT models, including fine-tuning and pre-training setups for both English and Japanese. - Updated README files to reflect the addition of new configurations for Llama-Bi-EN, ModernBERT-EN, and their respective training stages. - Included detailed configurations for training and fine-tuning processes, specifying parameters such as learning rates, batch sizes, and dataset paths.
…ate validation_split_percentage for Llama and ModernBERT models - Added "ddp_timeout": 86400 to training configurations for Llama and ModernBERT models in both English and Japanese. - Updated validation_split_percentage to 10 in Llama-EN-3B-PT-stage1.json and ModernBERT-EN-3B-PT-stage1.json. - Increased num_train_epochs to 6 in Llama-JP-3B and ModernBERT-JP-3B training configurations. - Set streaming to false in training configuration files for Llama and ModernBERT models.
- Increased gradient accumulation steps for Llama-EN-3B-PT-stage1 and Llama-JP-3B-PT-stage1 from 2048 to 4096. - Increased gradient accumulation steps for Llama-EN-3B-PT-stage2 and Llama-JP-3B-PT-stage2 from 512 to 1024. - Increased gradient accumulation steps for Llama-EN-1B-PT-stage2 and Llama-JP-1B-PT-stage2 from 512 to 1024. - Added new configuration files for Llama-JP-1B and Llama-EN-1B for both stage 1 and stage 2 pre-training. - Added new configuration files for ModernBERT-JP-1B and ModernBERT-EN-1B for both stage 1 and stage 2 pre-training. - Updated README files to include new model configurations and examples. - Enhanced model initialization script to allow optional tokenizer path.
…s training configurations
…requency - Changed "logging_steps" from 10 to 1 in multiple training configuration files for various models, including Sentence-Llama, ModernBERT, and Sarashina, to enhance the granularity of logging during training.
… ModernBERT training configurations
…and ModernBERT training configurations
- Changed wandb version from >=0.19.10 to ==0.20.1 in pyproject.toml and uv.lock. - Added setproctitle package with version 1.3.7 to uv.lock.
…ng configurations across multiple models
…ss multiple models
…ng configurations
…e dependencies - Added "gather_across_devices": true to various model configuration JSON files to enable gathering embeddings across devices before computing cached loss. - Updated the sentence-transformers dependency version from 4.1.0 to 5.1.0 in pyproject.toml and uv.lock. - Modified run_st.py to pass the gather_across_devices argument to the CachedMultipleNegativesRankingLoss. - Introduced gather_across_devices field in ModelArguments class for better control over the gathering behavior.
…or various models
…gurations across multiple models
…dient accumulation steps for improved performance
…dient accumulation steps for multiple models
…dient accumulation steps for multiple models
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WHY
WHAT