--set flag ; return table

[bkj]

One new feature:

• --set flag in magnet.evaluation, to be able to set a field in the model card from the command line

One minor thing:

• return human_table from magnet.instance_predictor (so that our card can access results)

[dmjoy]

This is a good feature to have / add, any concerns @b-fenelon. Though I know the evaluate.py script is likely undergoing (or soon to undergo) some heavy refactoring to support the notion of pipelines

[Erotemic]

Porting a comment I made on slack:

Taking a look at this. Mechanically, you can probably simplify this a lot by just accepting "set" as string and parsing it with YAML. This is an extremely common pattern I use all the time. It would look something like this:

class EvaluationConfig(scfg.DataConfig):
    path = scfg.Value(
        None, required=True, position=1, help='Path to evaluation card YAML'
    )
    overrides = scfg.Value(
        None, type=str, help='Override symbol values (e.g. {dataset: legalbench, num_replicates: 5})'
    )

...

import kwutil
overrides = kwutil.Yaml.coerce(args.overrides)
if overrides is not None:
    for key, value in overrides.items():
        if key not in card.symbols:
            raise ValueError(f"Unknown symbol '{key}' -- available: {list(card.symbols.keys())}")
        card.symbols[key]['value'] = value

And then could call the script like:

python -m magnet.evaluation --path=some/path --overrides="
    dataset: legalbench
    num_replicates: 5
    "

YAML handles the types, and you avoid having to write a custom mini language and parser.

The kwutil.Yaml.coerce is also nice because if you specify a path to a YAML file with a big set of overrides in it, it will read the file and use that, but if the string looks like literal YAML text it parses it right off of the command line. You can get more strict behaviors with kwutil.Yaml.load and kwutil.Yaml.loads