how to load clip_b_ml_cascade_maskrcnn_model_224

[zouxiangyu66]

Thank you for your work! I have a question about how to load clip_b_ml_cascade_maskrcnn_model_224 when evaluation? I replaced the original clip-base model with the clip_b_ml_cascade_maskrcnn_model_224. However, due to the differences in the parameter list, the model cannot be loaded.

[linhuixiao]

@zouxiangyu66 Hi, the readme has been detailed on configuring the inference and training process of the code in the repository. You can double-check the flow in the readme. Also, you can post your error message here and I can help you find where the bug is.

[sansan1456]

@linhuixiao
Thank you for your work on this project! I'm encountering an issue when trying to load the clip_b_ml_cascade_maskrcnn_model_224. Specifically, I'm seeing the following messages:

Missing keys when loading LoRA fine-tuned CLIP model:
['base_model.model.text_model.encoder.layers.0.self_attn.k_proj.base_layer.weight', 'base_model.model.text_model.encoder.layers.0.self_attn.k_proj.base_layer.bias', 'base_model.model.text_model.encoder.layers.0.self_attn.v_proj.base_layer.weight', 'base_model.model.text_model.encoder.layers.0.self_attn.v_proj.base_layer.bias',
...
Unexpected additional keys when loading LoRA fine-tuned CLIP model:
['base_model.model.text_model.encoder.layers.0.self_attn.k_proj.weight', 'base_model.model.text_model.encoder.layers.0.self_attn.k_proj.bias', 'base_model.model.text_model.encoder.layers.0.self_attn.v_proj.weight', 'base_model.model.text_model.encoder.layers.0.self_attn.v_proj.bias',
...
Could you provide some guidance on how to resolve these issues? Any advice or steps to ensure the model loads correctly would be greatly appreciated.

[zouxiangyu66]

@sansan1456 @linhuixiao
I guess that‘s because the model only load the lora weights of clip. i'm not sure if I'm right

[linhuixiao]

@sansan1456 @zouxiangyu66 Hi, which command are you running? Did you download clip_b_ml_cascade_maskrcnn_model_224 and clip-vit-base-patch16 as required in my readme? /path_to_clip in models/HiVG.py needs to fill in the directory address of clip-vit-base-patch16.

[linhuixiao]

@zouxiangyu66 @sansan1456 Generally, when training the LoRA stage of the current stage, we need to save the CLIP lora model weights of the previous stage and reload them separately in the LoRA part of the current stage. Specific see https://github.com/linhuixiao/HiVG/blob/master/hivg_train.py#L238

[zouxiangyu66]

@linhuixiao I'm running the training command. Change --clip_model to clip_b_ml_cascade_maskrcnn_model_224 /path_to_clip in HiVG.py to clip-vit-base-patch16.

[zouxiangyu66]

@linhuixiao Thank you for your reply! When evaulation . I change /path_to_clip in HiVG.py to clip-vit-base-patch16. I download the finetuned-base model on refcoco,however I only get 84.8 on refcoco-val

[linhuixiao]

@zouxiangyu66 Hi, you may not have the HiLoRA configuration paired correctly. Generally speaking, because of the compatibility of the A and B matrices, the correct result will run even if the HiLoRA stage is wrong, but the performance will drop by about 2%. The results of my RefCOCO split experiments on my server are similarly equal to the results of the paper. Maybe you can check your code configuration again. Alternatively, you can try training yourself. The results in the paper can be obtained in general.

[sansan1456]

Hello, @linhuixiao

HiVG/hivg_train.py

Line 225 in 2b4c0dd

print(missing_keys)

I am encountering a missing key error during the warm-up training phase. The missing key has one more ".base_layer" than the additional key:
"base_model.model.text_model.encoder.layers.0.self_attn.k_proj.base_layer.weight"
'base_model.model.text_model.encoder.layers.0.self_attn.k_proj.weight'
. I have changed the weight path as follows: --clip_model: clip_b_ml_cascade_maskrcnn_model_224 /path_to_clip in HiVG.py: clip-vit-base-patch16.

[linhuixiao]

Hello, @linhuixiao

HiVG/hivg_train.py

Line 225 in 2b4c0dd

print(missing_keys)

I am encountering a missing key error during the warm-up training phase. The missing key has one more ".base_layer" than the additional key:
"base_model.model.text_model.encoder.layers.0.self_attn.k_proj.base_layer.weight"
'base_model.model.text_model.encoder.layers.0.self_attn.k_proj.weight'
. I have changed the weight path as follows: --clip_model: clip_b_ml_cascade_maskrcnn_model_224 /path_to_clip in HiVG.py: clip-vit-base-patch16.

@sansan1456 Hi, which command are you running? Is there successful reproduction of reasoning? Please provide more details about your environment and process so that I can help you debug it. Otherwise it will take a lot of time, after all, I am busy and also need to work.

[sansan1456]

@linhuixiao I used the command bash train_and_eval_script/train_single_dataset_finetuning_base.sh and did not perform inference. My environment dependency versions are consistent with those provided in your requirements.txt. When I set the --clip_model path to empty, I was able to successfully train on RefCOCOG, but the training precision achieved was only 71%, which is 7% lower than the results reported in your paper. I believe the main issue lies in loading the clip_b_ml_cascade_maskrcnn_model_224.

[sansan1456]

@zouxiangyu66 I was wondering if you’ve managed to solve the problem of loading clip_b_ml_cascade_maskrcnn_model_224. If so, could you kindly share the solution with me?

@sansan1456 @linhuixiao I guess that‘s because the model only load the lora weights of clip. i'm not sure if I'm right

[linhuixiao]

@linhuixiao I used the command bash train_and_eval_script/train_single_dataset_finetuning_base.sh and did not perform inference. My environment dependency versions are consistent with those provided in your requirements.txt. When I set the --clip_model path to empty, I was able to successfully train on RefCOCOG, but the training precision achieved was only 71%, which is 7% lower than the results reported in your paper. I believe the main issue lies in loading the clip_b_ml_cascade_maskrcnn_model_224.

@sansan1456 In warmup stage, --clip_model should be used the clip_b_ml_cascade_maskrcnn_model_224 model path, and should not leave empty. In the following HiLoRA stage, --clip_model should be used the last stage's saved HiLoRA CLIP model.

[sansan1456]

@sansan1456 In warmup stage, --clip_model should be used the clip_b_ml_cascade_maskrcnn_model_224 model path, and should not leave empty. In the following HiLoRA stage, --clip_model should be used the last stage's saved HiLoRA CLIP model.

@linhuixiao During the warmup stage, I encountered the "missing keys" error when using the clip_b_ml_cascade_maskrcnn_model_224 path that I mentioned earlier. To work around this, I tried setting the --clip_model path to empty, which allowed the training to proceed but resulted in a precision of only 71%. Could you kindly advise on how to resolve the "missing keys" issue when loading clip_b_ml_cascade_maskrcnn_model_224? Your guidance would be greatly appreciated.

[linhuixiao]

@sansan1456 In warmup stage, --clip_model should be used the clip_b_ml_cascade_maskrcnn_model_224 model path, and should not leave empty. In the following HiLoRA stage, --clip_model should be used the last stage's saved HiLoRA CLIP model.

@linhuixiao During the warmup stage, I encountered the "missing keys" error when using the clip_b_ml_cascade_maskrcnn_model_224 path that I mentioned earlier. To work around this, I tried setting the --clip_model path to empty, which allowed the training to proceed but resulted in a precision of only 71%. Could you kindly advise on how to resolve the "missing keys" issue when loading clip_b_ml_cascade_maskrcnn_model_224? Your guidance would be greatly appreciated.

@sansan1456 I read your reply, and I guess that you did not load the clip model correctly. I guess that you may have downloaded the wrong checkpoint and misnamed clip-vit-base-patch16 to clip_b_ml_cascade_maskrcnn_model_224. I suggest you download it again. Also, I recommend that you run inference successfully first, it is inexperienced to train directly without running inference successfully.