In certain situations, one might want to ensure that two or more configurations behave the same wrt some measurement.
For example, if one wants to evaluate clingo builds created with different compilers or different optimization settings, it is important that the explored search-space (e.g. expressed in the number of choices) is the same.
It might be nice to have a script or a separate btool subcommand that given a runscript, a list of configurations C1, ..., Cn, and a measurement M (or a list of measurements),
- compares the configurations on
M, and
- produces a warning/error for each instance
I, where configuration Ci differs from Cj on M.
To make the results more useful, the script/subcommand should probably ignore/skip instances with UNKNOWN status (timeout/error).
On the other hand, an inconsistent status (e.g. "SATISFIABLE" vs "UNSATISFIABLE") should probably always result in an error.
In certain situations, one might want to ensure that two or more configurations behave the same wrt some measurement.
For example, if one wants to evaluate
clingobuilds created with different compilers or different optimization settings, it is important that the explored search-space (e.g. expressed in the number of choices) is the same.It might be nice to have a script or a separate
btoolsubcommand that given a runscript, a list of configurationsC1, ..., Cn, and a measurementM(or a list of measurements),M, andI, where configurationCidiffers fromCjonM.To make the results more useful, the script/subcommand should probably ignore/skip instances with
UNKNOWNstatus(timeout/error).On the other hand, an inconsistent status (e.g. "SATISFIABLE" vs "UNSATISFIABLE") should probably always result in an error.