Skip to content

feat(eval): support Dynamo backend in run_eval#4

Draft
jthomson04 wants to merge 3 commits into
dynamo-k8s-integrationfrom
jt/eval-dynamo-run-eval
Draft

feat(eval): support Dynamo backend in run_eval#4
jthomson04 wants to merge 3 commits into
dynamo-k8s-integrationfrom
jt/eval-dynamo-run-eval

Conversation

@jthomson04

@jthomson04 jthomson04 commented Jun 22, 2026

Copy link
Copy Markdown
Owner

Summary

  • add eval_dynamo.yaml for running eval against an external DynamoGraphDeployment
  • keep the existing vLLM eval text-generation path intact
  • add a small backend setup factory and a generic text-only token-ID generation path for Dynamo and future GenerationInterface.generate() backends
  • fix run_eval.py CLI override handling so --config is not folded into OmegaConf overrides
  • cover basic backend setup, CLI override parsing, Dynamo multimodal rejection, and token-ID generation/decoding in unit tests

Validation

  • NRL_IGNORE_VERSION_MISMATCH=1 uv run pytest tests/unit/evals/test_eval.py -q
  • NRL_IGNORE_VERSION_MISMATCH=1 uv run pytest tests/unit/test_config_validation.py -q
  • uv run ruff check nemo_rl/evals/eval.py tests/unit/evals/test_eval.py examples/run_eval.py
  • real Dynamo smoke: examples/run_eval.py against jothomson-dyn-nano-v2-mx, score=1.0000 (1.0/1)

Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant