I want to reproduce this work. Currently, I am in the first stage (monolingual training).
My script is as follows:
OUTPUT_DIR=${1:-"./saves/llama-2-7b-oscar-ft"}
random port between 30000 and 50000
port=$(( RANDOM % (50000 - 30000 + 1 ) + 30000 ))
accelerate launch --main_process_port ${port} --config_file configs/deepspeed_train_config.yaml
run_llmmt.py
--model_name_or_path /llms/Qwen2.5-3B-Instruct
--oscar_data_path oscar-corpus/OSCAR-2301
--oscar_data_lang en,ru,cs,zh,is,de
--interleave_probs "0.17,0.22,0.14,0.19,0.08,0.2"
--streaming
--max_steps 600000
--do_train
--low_cpu_mem_usage
--fp16
--learning_rate 2e-5
--weight_decay 0.01
--gradient_accumulation_steps 4
--lr_scheduler_type cosine
--warmup_ratio 0.01
--ignore_pad_token_for_loss
--ignore_prompt_token_for_loss
--per_device_train_batch_size 4
--per_device_eval_batch_size 4
--save_strategy steps
--save_steps 2000
--save_total_limit 1
--logging_strategy steps
--logging_steps 1
--output_dir ${OUTPUT_DIR}
--max_new_tokens 1024
--max_source_length 1024
--seed 42
--overwrite_output_dir
--use_flash_attention_2
--report_to wandb
--run_name ${OUTPUT_DIR}
There will be a key error problem under the above script.
So I went and modified the utils.py file, changing column_names_oscar = ["id", "meta", "text", "raw_text"] to column_names_oscar = [] (I checked the load data and there is only one field： raw_text). However, a new problem will arise:
so，what is the reason for this? I went to check and the data update of oscar was a year ago. The code should support the existing oscar data. Looking forward to your reply.
I want to reproduce this work. Currently, I am in the first stage (monolingual training).
My script is as follows:
OUTPUT_DIR=${1:-"./saves/llama-2-7b-oscar-ft"}
random port between 30000 and 50000
port=$(( RANDOM % (50000 - 30000 + 1 ) + 30000 ))
accelerate launch --main_process_port ${port} --config_file configs/deepspeed_train_config.yaml
run_llmmt.py
--model_name_or_path /llms/Qwen2.5-3B-Instruct
--oscar_data_path oscar-corpus/OSCAR-2301
--oscar_data_lang en,ru,cs,zh,is,de
--interleave_probs "0.17,0.22,0.14,0.19,0.08,0.2"
--streaming
--max_steps 600000
--do_train
--low_cpu_mem_usage
--fp16
--learning_rate 2e-5
--weight_decay 0.01
--gradient_accumulation_steps 4
--lr_scheduler_type cosine
--warmup_ratio 0.01
--ignore_pad_token_for_loss
--ignore_prompt_token_for_loss
--per_device_train_batch_size 4
--per_device_eval_batch_size 4
--save_strategy steps
--save_steps 2000
--save_total_limit 1
--logging_strategy steps
--logging_steps 1
--output_dir ${OUTPUT_DIR}
--max_new_tokens 1024
--max_source_length 1024
--seed 42
--overwrite_output_dir
--use_flash_attention_2
--report_to wandb
--run_name ${OUTPUT_DIR}
There will be a key error problem under the above script.
So I went and modified the utils.py file, changing column_names_oscar = ["id", "meta", "text", "raw_text"] to column_names_oscar = [] (I checked the load data and there is only one field： raw_text). However, a new problem will arise:
so，what is the reason for this? I went to check and the data update of oscar was a year ago. The code should support the existing oscar data. Looking forward to your reply.