fix(train): show resumed position in epoch progress bar#850
Conversation
Signed-off-by: Igor Margulis <igor.margulis@intel.com>
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughChangesMid-epoch progress reporting
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/speculators/train/trainer.py`:
- Around line 432-436: In src/speculators/train/trainer.py lines 432-436, keep
the requested resume offset separate from the number of batches actually skipped
by the fast-skip path, and pass the actual skipped count to tqdm’s initial value
so fallback replay starts at zero. In tests/unit/train/test_mid_epoch_resume.py
lines 408-448, add coverage using a sampler without fast-skip that verifies
progress initialization remains zero during replay.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 4a35a412-bab7-4a33-966d-7d4f8c3a62c6
📒 Files selected for processing (2)
src/speculators/train/trainer.pytests/unit/train/test_mid_epoch_resume.py
Signed-off-by: Igor Margulis <igor.margulis@intel.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Merge Protections🔴 1 of 1 protections blocking · waiting on 👀 reviews
🔴 Require approval from approved reviewers listWaiting for any of
This rule is failing.All pull requests must have at least one approving review from a member of the approved reviewers list before merging.
|
Summary
Follow-up to #603, which added mid-epoch checkpoint save and resume.
The fast-skip resume path removes already completed batches from the training
loader. Because tqdm inferred the reduced iterable length, the progress bar starts
from zero even though training had resumed mid-epoch.
Initialize the rank-zero progress bar with the full epoch total and the resumed
local step so it displays the correct percentage and remaining work.
Also adds a regression test covering the progress-bar arguments for a fully
skipped resumed epoch.
Testing
tests/unit/train/test_mid_epoch_resume.py