关于Visual Symptom Generator (VSG)的疑问

[newfish-lab]

作者你好！非常感谢你优秀的工作，我有几点关于Visual Symptom Generator (VSG)的疑问想咨询你：
文章里你提到“the refined set is obtained by intersecting the initial coarse set with the response", 这里的intersecting是什么操作呢，看不太懂，是怎么通过第二次gpt response的结果refine第一次gpt reponse的结果的？

[xiaofang007]

Hi, our idea is that by querying gpt, we could get a set of visual symptoms but we cannot guarantee the correctness of those symptoms because gpt responds based on its learned knowledge but not data. Then we could provide a few samples to let the vqa model to provide another set of visual symptoms based on the visual features of samples. However, the obtained set is not accurate as well because there might be irrelevant visual features that are specific to the samples. Therefore, we just intersect the two sets and hoping to obtain a more accurate set of visual descriptions of symptoms.

[newfish-lab]

Thank you very much for your response! However, I'm still a bit confused about the "intersection" operation. Based on the illustrations in the paper, it seems like that you are essentially selecting values from the first set of visual symptoms, but the elements chosen must be related to those in the second set of visual symptoms. Is my understanding correct?

[xiaofang007]

Yes, you are correct. Sorry for the confusion. Since we are intersecting the two sets, I think it is reasonable to illustrate the process via selecting in the first set as the final symptoms should be present in the first set. The selected symptoms should also be related to the second set, and the red tick beside the symptom in the figure illustrates this.

[newfish-lab]

Thank you very much for your response! I have another question, why are you averaging the results of three vision backbones, shouldn't there be one result for each vision backbone individually?

[xiaofang007]

Thanks for your question. Since there are different vision backbones to choose from in clip model, we average the result of three vision backbones to eliminate the result unfairness caused by different vision backbones.

[newfish-lab]

Thank you once again for your patient responses~! I still have a question regarding your experimental setup. I noticed that you used 3 different seeds for each vision backbone. I'm curious about how you reported the final results in the paper. Did you just select the best performance among the three runs, or did you report the average results across all three seeds?

[xiaofang007]

Hi, sorry for the confusion. I average the results across all three seeds and all three backbones, which means I report the average of nine results on one method.

[YsYusaito]

Yes, you are correct. Sorry for the confusion. Since we are intersecting the two sets, I think it is reasonable to illustrate the process via selecting in the first set as the final symptoms should be present in the first set. The selected symptoms should also be related to the second set, and the red tick beside the symptom in the figure illustrates this.

Hi, I have also question about "intersection" operation.
Is the intersection operation performed by vectorizing the words in the two sets and comparing their similarity to extract the common elements? Or are you instructing GPT-4 to extract the common elements?

[xiaofang007]

Yes, you are correct. Sorry for the confusion. Since we are intersecting the two sets, I think it is reasonable to illustrate the process via selecting in the first set as the final symptoms should be present in the first set. The selected symptoms should also be related to the second set, and the red tick beside the symptom in the figure illustrates this.

Hi, I have also question about "intersection" operation. Is the intersection operation performed by vectorizing the words in the two sets and comparing their similarity to extract the common elements? Or are you instructing GPT-4 to extract the common elements?

Hi, I think both ways are reasonable. We instruct GPT-4 to find the common visual symptoms of the two sets.

[YsYusaito]

Hi, I think both ways are reasonable. We instruct GPT-4 to find the common visual symptoms of two sets.

Thank you for quick reply.
I understood.