Jons/prod env sidecar configs#63
Open
Erotemic wants to merge 3 commits into
Open
Conversation
Sibling of materialize_helm_run that replays a fully-resolved run_spec.json directly (helm.common.codec.from_json -> RunSpec -> in-process run_benchmarking) instead of reconstructing a run-entry string and re-parsing it through helm-run. The reconstruct-and-reparse hop re-derives the recipe under the *currently installed* crfm-helm defaults; replaying the resolved spec removes that silent drift surface. - Dual input: explicit --run-spec-json (wins) or discovery via find_best_precomputed_run (the run-entry string then only LOCATES the official dir; precomputed_root is the recipe source). - Mirrors the helm_run registration preamble verbatim (incl. load_entry_point_plugins) BEFORE a class-resolution preflight that recurses into nested ObjectSpec/dict args (judge specs). The preflight catches broadly and reports every unresolvable class: ImportError/AttributeError = version drift, OptionalDependencyNotInstalled = a recipe/environment filter reason — neither should crash the scan mid-way. - Substitution is by-name only: no adapter_spec.model rewrite (the local model_deployments.yaml rebinds the official deployment name), so the produced run dir keeps run_spec.name and downstream identity is unchanged. Only max_eval_instances is substituted. - In-process: a Python exception is captured to cmd_stderr.txt for the failure classifier; DONE is written last. Tests (importorskip helm): nested-ObjectSpec collection, preflight pass/fail incl. a non-ImportError path, and a schema-drift round-trip over real public run_spec.json asserting no raw key is silently dropped on from_json. End-to-end replay is opt-in behind MATERIALIZE_FROM_SPEC_INTEGRATION. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…oint)
A faithful run_spec.json replay with by-name deployment substitution makes the
local run record the OFFICIAL deployment (e.g. together/phi-2), so the audit's
comparison reads same_deployment=yes and the engine substitution (local HF/vLLM
vs the hosted API) is invisible — the key difference the audit exists to flag.
Add an optional --model-deployment <local-name>: after from_json, rewrite
adapter_spec.model_deployment to that local name (the endpoint that actually
served the run). adapter_spec.model is never touched, and HELM run names encode
model=… not model_deployment=…, so the produced run dir stays name-identical to
the official and pairing/logical_run_key are inert. Default (unset) is pure
by-name — the general replay path is unchanged.
The substitution is extracted into a pure, unit-testable
apply_adapter_substitutions(run_spec, *, max_eval_instances, model_deployment) ->
(run_spec, replay_record); it records replay.deployment_substitution={from,to}
(null when unset) as auditable provenance (the comparability signal comes for
free from the rewritten spec). Real-helm-dataclass unit tests cover
set/unset/deployment-only and the flag parse.
See docs/planning/from-spec-deployment-rewrite-plan.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…configs) prepare_local_helm_config now materializes optional model_metadata.yaml and tokenizer_configs.yaml sidecars into <local_path>/ alongside the existing model_deployments.yaml, and both materialize CLIs (run-entry + from-spec) expose --model-metadata-fpath / --tokenizer-configs-fpath and record them in the adapter manifest. HELM's register_configs_from_directory already reads all three from --local-path, so net-new model/tokenizer ids can be registered per-experiment without editing HELM itself. Both keys are identity-bearing algo_params on MaterializeHelmRunNode. Tests: tests/test_prepare_local_helm_config.py (canonical-name copy, optionality, missing-file error names the offending param). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Work on top of edwards work.