Parameter Settings

[yyao123-svg]

Hi，I want to know how to set these two parameters during the LoRA fine-tuning stage: --hi_lora_retrain /patch_to_output/output_v101/unc/best_checkpoint.pth --hi_lora_clip /patch_to_output/output_v101/unc/clip_lora_stage_with_bridge.pth. Especially for the first stage, should hi_lora_clip be set to clip_b_ml_cascade_maskrcnn_model_224, and hi_lora_retrain be set to the model saved before using LoRA?

[linhuixiao]

@yyao123-svg Hi, thank you for your question. I'm sorry that this part of the code is indeed somewhat confusing. Due to multiple LoRa trainings involved, the position of CLIP in LoRA varies with the increase of stages. Therefore, the entire model from the previous stage cannot be directly loaded.

The parameters configured in the train_and_eval_script in the code repository are correct.

First, you can refer to the code from line 220 to line 280 in hivg_train.py. Specifically, --resume is for resuming training from a breakpoint, while --rerain is for retraining after having an initial model, which is used for training warm-up. When --hi_lora_stage and --hi_lora_clip are used together, multi-stage training of Hi LoRA can be achieved; meanwhile, --hi_lora_retrain is the path to load the entire model checkpoint of the last stage. --clip_model is only loaded and used in the initial state (Stage 0).

Thanks.