Hi! Thank you for your great work! I apologize for bothering you, I just wanted to flag a potential issue with either IFBench_test.jsonl or sample_output.jsonl. It occurred to me that these files have different length: IFBench_test.jsonl has 300 while sample_output.jsonl contains 294. On top of that, running the following command from readme: python3 -m run_eval --input_data=IFBench_test.jsonl --input_response_data=sample_output.jsonl --output_dir=eval for me results in key errors. I believe it happens because of the fact that sample_output.jsonl contains some prompts with extra trailing whitespaces. For me those are 0,1,14 and 20 keys, but I might have missed some. After fixing those, the command works just fine. Thank you in advance for reading this issue.
Hi! Thank you for your great work! I apologize for bothering you, I just wanted to flag a potential issue with either IFBench_test.jsonl or sample_output.jsonl. It occurred to me that these files have different length: IFBench_test.jsonl has 300 while sample_output.jsonl contains 294. On top of that, running the following command from readme: python3 -m run_eval --input_data=IFBench_test.jsonl --input_response_data=sample_output.jsonl --output_dir=eval for me results in key errors. I believe it happens because of the fact that
sample_output.jsonlcontains some prompts with extra trailing whitespaces. For me those are 0,1,14 and 20 keys, but I might have missed some. After fixing those, the command works just fine. Thank you in advance for reading this issue.