Skip to content

Align aggregate metrics with persisted rollouts#1799

Open
fallintoplace wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
fallintoplace:fix/aggregate-only-persisted-rollouts
Open

Align aggregate metrics with persisted rollouts#1799
fallintoplace wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
fallintoplace:fix/aggregate-only-persisted-rollouts

Conversation

@fallintoplace

Copy link
Copy Markdown

Summary

  • aggregate rollout metrics from the rows written to the main rollouts JSONL only
  • keep sidecar failures and _ng_no_persist attempts out of runtime /aggregate_metrics
  • add a regression test covering success, sidecar failure, and no-persist lanes

Root cause

run_from_config() appended every result to the in-memory results list before the persistence gate, then passed that full list into _call_aggregate_metrics(). That let runtime aggregation include rows that never landed in output.jsonl, so gym eval run could disagree with gym eval aggregate.

Validation

  • uv run --extra dev python -m pytest tests/unit_tests/test_rollout_collection.py

@copy-pr-bot

copy-pr-bot Bot commented Jun 27, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@nemo-automation-bot nemo-automation-bot Bot added the community-request Issue reported or requested by someone from the community label Jun 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-request Issue reported or requested by someone from the community

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant