Reproducibility is key in mlperf but with the large number of submissions per submitter/benchmark, it is almost impossible to figure out exactly which config file or code was used to generate which result.
- We need checkers that can ensure that the config file name disclosed can be found in the code
- We need to work through https://github.com/mlcommons/policies/blob/master/submission_rules.adoc#training-1 and fix ambiguity
Reproducibility is key in mlperf but with the large number of submissions per submitter/benchmark, it is almost impossible to figure out exactly which config file or code was used to generate which result.