Use folder structure and relative paths to add encodings

[jorgefandinno]

Currently, there is no nice way to exploit the file structure of the benchmark to recognize instances and encodings, and allow different encodings for different systems.

For example, let us assume our benchmark folder has the following structure:

- benchmarks/
  | - problem1/
  | - problem2/
  |- ...

and each problem directly contains the programs to be run. Then, we can add

<benchmark name="benchmark">
    <folder path="benchmarks"/>
</benchmark>

and all problems will be run and grouped in classes appropriately.

If on the other hand, each problem is separated into instances and encoding, we have to manually state which encoding to use with each instance. In this case, we may have the following structure:

- benchmarks/
  | - problem1/
      | - instances/
      | - encoding.sys1
      | - encoding.sys2
  | - problem2/
      | - instances/
      | - encoding.sys1
      | - encoding.sys2
  |- ...

It would be nice to be able to set this automatically as follows:

<benchmark name="benchmark">
    <folder path="benchmarks", instances="instances">
      <encoding relative="encoding" /encoding>
   </folder>
</benchmark>

<system name="sistem1" version="0"
      measures="clingo"
      config="seq">
      <setting name="baseline" cmdline="--stats 1" encoding_suffix=".sys1"/>
</system>

<system name="sistem2" version="0"
      measures="clingo"
      config="seq">
      <setting name="baseline" cmdline="--stats 1" encoding_suffix=".sys2"/>
</system>

For each folder in benchmarks, this expects a folder called instances. For each file benchmarks/problem1/instances/file, the solver will be called with the command lines

system1 {sys_cmd} {setting_cmd} benchmarks/problem1/instances/file benchmarks/problem1/encoding.sys1
system2 {sys_cmd} {setting_cmd} benchmarks/problem1/instances/file benchmarks/problem1/encoding.sys2

[tzschmidt]

@jorgefandinno Thanks for your feedback!
Using the file structure to automatically recognize instances and encodings is nice but prerequisites all users using the same file structure. Nonetheless, I will have a look at it when I have some time.

Regarding system depended encodings. This is currently possible (although cumbersome) via setting depended encodings and absolute paths (see example below). System depended encodings as an extra attribute make sense.

<runscript output="test">

	<machine name="zuse" cpu="24x8xE5520@2.27GHz" memory="24GB"/>
	
  	<config name="seq-generic" template="templates/seq-generic.sh"/>
  
	<system name="sys1" version="0" measures="clasp" config="seq-generic">
		<setting name="setting-1" tag="basic">
			<encoding file="benchmarks/problem1/encoding.sys1" encoding_tag="problem1"/>
		</setting>
    </system>

	<system name="sys2" version="0" measures="clasp" config="seq-generic">
		<setting name="setting-1" tag="basic">
			<encoding file="benchmarks/problem1/encoding.sys2" encoding_tag="problem1"/>
		</setting>
    </system>

	<seqjob name="seq-gen" timeout="900s" runs="1" parallel="1"/>

	<benchmark name="test">
		<folder path="benchmarks/problem1/instances" encoding_tag="problem1">
			<encoding file="benchmarks/problem1/encoding.lp"/>
		</folder>
	</benchmark>

	<project name="clingo-seq-job" job="seq-gen">
		<runtag machine="zuse" benchmark="test" tag="*all*"/>
	</project>

</runscript>

[jorgefandinno]

Thanks,

Regarding system depended encodings. This is currently possible (although cumbersome) via setting depended encodings and absolute paths (see example below). System depended encodings as an extra attribute make sense.

I understand. It just becomes really cumbersome when you when you have many "problems" in the benchmarks. I also believe that it is not necessary to have the same structure if you allow relative paths. You can have any structure as long as you can set your path relative to the instance file.