Add override flag#50
Conversation
|
LGTM |
| self.symbols[key]['value'] = value | ||
| elif 'sweep' in self.symbols[key]: | ||
| # accept n entries | ||
| if isinstance(value, list): |
There was a problem hiding this comment.
This is the only thing that I see that raises a red flag. The flag is doing an "override", but this is extending an existing list and there is no way to tell it to use a new list instead. If this is needed there is some trickery that can be done. You could set the value to be a dictionary with one key like __extend__: [1, 2, 3] and that very particularly special case could be interpreted as wanting to extend the sweep list, and it would be explicit in the CLI (at least if you had knowledge of the behavior, but otherwise it looks weird enough to at least cue someone into some sort of special behavior happening)
There was a problem hiding this comment.
I'm in favor of dropping the append. This flag is mostly for debugging and only works for python symbols (i.e. not kwdagger components). I originally didn't want to override lists to dissuade users from using '--override' in place of creating a new card. But, I agree that it's a weird behavior. If users want more flexible sweep handling, we can revisit the solution you give here
There was a problem hiding this comment.
@Erotemic it looks like Ben addressed the issue here, are we good to merge?
Not an urgent feature
Following Jon's recommendation to #45, created an '--override' flag that accepts a YAML-like input. This adds a
replacemethod toEvaluationCardthat enables programmatic changes to multiple symbol values, which is useful for testing. This only allowsvalue(must preservetype) andsweepreplacements, limiting the changes possible without updating the card itself.I am currently omitting a README section because I don't think we should recommended this approach over explicitly defining a new card. Once we have a standard output format, the results should capture the necessary details anyways to recreate the card with overrides.
Example usage:
magnet evaluate magnet/cards/llama.yaml --override "{helm_runs_path: ./other-data-path/benchmark_output, threshold: 0.01}"