Only dev_scores.txt available in exp_out folder

[tanayshah23]

Hi,

I am trying to train a model on the heart dataset. The training works, but I am unable to find the following files in the exp_out folder:

train_pred.txt
dev_pred.txt
test_pred.txt
test_score.txt

Here's the screenshot for your reference:

My main concern is now that we have the trained model checkpoint, how can we use it to get results on the dev/test set and get the results?

Thanks!

[stefanhgm]

Hello @tanayshah23,

thanks for using the TabLLM project and please excuse the delayed response.

We do not use a separate validation set, since we fix the numbers epochs and no additional parameter tuning is done (following the original t-few setup). Instead, we treated our test set as the validation (=dev) set in the code because this gave us the test performance (named dev_scores) immediately during training without another test inference run.

This is definitely a bit confusing, so thanks for pointing this out. We also added this to our revised readme. As a result, you should find the test performance of each run in dev_scores.json. We also added the script for evaluation to the repository and the readme instructions now.

To obtain the prediction results themselves you probably have to change the t-few code to output the data. I don't think there exists a routine for this yet.

I hope that helps!