CIDEr Score Mismatch

[dina-adel]

Hello,

Thanks for sharing your work!

I am trying to replicate your results using the shared checkpoints. However, I am not sure if I am using the correct metric. I followed this repo for calculating the CIDEr score. My results were 0.618 on COCO-Val and 1.42 on IU-XRay which does not make sense compared to the results in the paper.

Could you please guide me here or share your code for evaluation?

[ckzbullbullet]

We used the this repo https://github.com/EvolvingLMMs-Lab/lmms-eval for evaluation.
Basically, we implement the dragonfly class under their framework. Then you can choose different tasks for evaluation.

Regarding biomedical eval, we are also using pycocoevalcap. Few things to keep in mind. You are using the med version of the model for biomedical evaluations. Also, please make sure to use the correct prompt format (llama3). We have examples on readme. Perhaps, also try on other biomedical tasks and see if you get similarly low score.

Please feel free to reply back if you still see the issues.

[dina-adel]

Thanks for getting back to me.

I am still running the llm-eval now on the coco dataset.

I ran the CIDEr on the iu-xray dataset again and I got the same score. Am I doing it wrong?

from pycocoevalcap.cider.cider import Cider
from pycocoevalcap.tokenizer.ptbtokenizer import PTBTokenizer

p_tokenizer = PTBTokenizer()
reference_captions_tokenized = p_tokenizer.tokenize(reference_captions)
generated_captions_tokenized = p_tokenizer.tokenize(generated_captions)

cider_scorer = Cider()
score, scores = cider_scorer.compute_score(generated_captions_tokenized, reference_captions_tokenized)
print(f'CIDEr Score: {score}')