Skip to content

Which command to train on HotPotQA and test on Musique? #9

Description

@changy12

Hello, I plan to reproduce "Musique (OOD)" in Table 2 of your Q-rag paper.

(1) Should I do "E5 HotpotQA only" training and then "E5 MuSiQue only" evaluation in Section 1 of ReadMe file?
When I do this,
FileNotFoundError: [Errno 2] No such file or directory: '../datasets/musique/hotpot_dev_distractor_v1.json'

(2) Also, the "LLM evaluation" in Section 1 is only for hotpotQA. Should I change "qrag-ft-e5-on-hotpotqa" to "qrag-ft-e5-on-musique" as follows to have LLM evaluation on musique?

python eval_llm_openqa.py \
     --file_path your/path/to/qrag-ft-e5-on-musique/eval_seed42.jsonl \
     --model_name Qwen/QwQ-32B \
     --output_file_path your/path/to/qrag-ft-e5-on-hotpotqa/llm-answering_eval.json

Thanks.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions