Issues in Reproducing Paper Accuracy​

[hhhhhhy5]

Thank you for your excellent open-source work!
I encountered difficulties when trying to reproduce the accuracy reported in the paper - my reproduced accuracy is significantly lower than what was reported in the paper. I used the following command:

`export HF_ENDPOINT=https://hf-mirror.com
export HF_DATASETS_OFFLINE=0
export CUDA_VISIBLE_DEVICES=0,1

--limit 256

reproduce prophet

results_dir="./results/gsm8k_cot_zeroshot_2"
mkdir -p "$results_dir"
accelerate launch --num_processes=2 --num_machines=1 eval_llada.py
--tasks gsm8k_cot_zeroshot
--model llada_dist
--model_args model_path='/code/HHY/Prophet/models/LLaDA-8B-Instruct',enable_early_exit=true,constraints_text="200:The|201:answer|202:is",gen_length=256,steps=256,block_length=32,answer_length=5,early_threshold=8.0,mid_threshold=5.0,late_threshold=3.5,gap_data_path="$results_dir/gap_data.jsonl"
2>&1 | tee "$results_dir/eval.log"

reproduce original version of llada

results_dir="./results_reproduce/gsm8k_cot_zeroshot_2"
mkdir -p "$results_dir"
accelerate launch --num_processes=2 --num_machines=1 eval_llada.py
--tasks gsm8k_cot_zeroshot
--model llada_dist
--model_args model_path='/code/HHY/Prophet/models/LLaDA-8B-Instruct',enable_early_exit=false,gen_length=256,steps=256,block_length=32,answer_length=5,gap_data_path="$results_dir/gap_data.jsonl"
2>&1 | tee "$results_dir/eval.log"`

However, the result of reproducing Prophet is as follows:
`llada_dist (model_path=/code/HHY/Prophet/models/LLaDA-8B-Instruct,enable_early_exit=true,constraints_text=200:The|201:answer|202:is,gen_length=256,steps=256,block_length=32,answer_length=5,early_threshold=8.0,mid_threshold=5.0,late_threshold=3.5,gap_data_path=./results/gsm8k_cot_zeroshot/gap_data.jsonl), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 1

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k_cot_zeroshot	3	flexible-extract	0	exact_match	↑	0.671	±	0.0129
		strict-match	0	exact_match	↑	0.345	±	0.0131

The result of evaluating original version of LLaDA is as follows:
`llada_dist (model_path=/code/HHY/Prophet/models/LLaDA-8B-Instruct,enable_early_exit=false,gen_length=256,steps=256,block_length=32,answer_length=5,gap_data_path=./results_reproduce/gsm8k_cot_zeroshot/gap_data.jsonl), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 1

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k_cot_zeroshot	3	flexible-extract	0	exact_match	↑	0.7263	±	0.0123
		strict-match	0	exact_match	↑	0.0023	±	0.0013

What could be the reasons for such a discrepancy?

[pixeli99]

Hi, @hhhhhhy5

Thanks for your interest in our work.

Can u paste your gsm8k_cot_zeroshot.yaml here? Maybe we are using a different prompt.

I have for you here my prompt.

doc_to_text: "Solve the following math problem step by step. The last line of your response should be of the form The answer is $ANSWER (without quotes) where $ANSWER is the answer to the problem.\n{{question}}\n Remember to put your answer on its own line after 'The answer is', and you do not need to use a \\boxed command."
# doc_to_text: "Q: {{question}}\nA: Let's think step by step."
doc_to_target: "{{answer}}" #" {{answer.split('### ')[-1].rstrip()}}"

Thanks :)

[JaydencoolCC]

Hi, @pixeli99
Could you provide more details about the prompts used in other benchmarks?
By the way, could you also explain whether you made any other changes?
Thanks!