Add integration test suite: end-to-end job lifecycle + failure/recovery

[RehanTaneja]

Summary

Adds the missing integration test suite called out in docs/chaos_testing.md: 11 tests under tests/integration/ (pytest marker integration, ~1 min) that spawn a real uvicorn coordinator and real worker subprocesses per test, with heartbeat/recovery timings shortened via DISTROML_* env vars.

Test coverage

test_job_lifecycle.py

• Submit → QUEUED + experiment record (API + persisted metadata files)
• Invalid submissions (422) and unknown job/experiment lookups (404)
• Full 2-worker lifecycle over real Gloo: auto-start on registration, heartbeat metrics flowing, clean COMPLETED, empty registry, populated run summary
• Runtime checkpoints committed with manifests; shard sha256/file sizes verified against disk
• Experiment metadata reuse for a resubmitted run_id; world-size mismatch rejected
• Cancel flow

test_failure_recovery.py (workers run --disable-dist; rationale in module docstring)

• Kill one of two workers mid-training → LOST via heartbeat timeout → RECOVERING → rank relaunched → resumes from last committed checkpoint ("resuming from step 51") → COMPLETED, with a time-to-recovery bound
• All workers lost → FAILED with reason
• POST /api/workers/deregister fast failure path (recovery < 5s, no heartbeat wait)
• Recovery attempts exhausted → FAILED ("max recovery attempts")
• Silent heartbeat drop (process alive, --drop-heartbeat-rate 1.0) detected as LOST

Product changes required to make these flows reachable

• Worker CLI flags for the existing failure-injection hooks + --checkpoint-interval, --heartbeat-interval, --disable-dist
• Workers report checkpoint shards (POST /api/workers/checkpoint-complete) so manifest.json is actually generated at runtime (previously unreachable — recovery had nothing to restore from)
• Heartbeats carry step/loss/throughput so metrics aggregation works in real runs
• submit_job respects submitted total_steps/checkpoint_dir (were hardcoded to 100/./checkpoints); recovery relaunch forwards the new flags
• Jobs auto-start QUEUED → RUNNING once all expected workers register (nothing called start_job() before)
• LOST workers no longer block job completion or manifest generation
• New routes: GET /api/jobs/{id} (decorator was missing), GET /api/jobs/{id}/metrics, POST /api/jobs/{id}/cancel, POST /api/workers/checkpoint-complete

Bug fixes uncovered by the suite

1. JobStateMachine.can_transition used a private python-statemachine API removed by 2.5.0 → jobs could never reach COMPLETED; now uses the public allowed_events
2. restore_checkpoint crashed on PyTorch ≥ 2.6 (weights_only=True default rejects saved RNG state) → recovery always failed; loads with weights_only=False
3. Resubmitting an existing run_id crashed on job_spec.deterministic (no such field on JobSpec)

Test plan

pip install -r requirements-dev.txt
PYTHONPATH=. python -m pytest tests/integration -m integration -v   # 11 passed, ~1 min
PYTHONPATH=. python -m pytest tests/ -q -m "not integration"        # unit tests

• Integration suite run 3× consecutively: green each time, no leaked worker/coordinator processes
• Note: 6 unit tests in tests/test_checkpoint_restore.py fail on main already (test worker never initializes its model before restoring) — pre-existing, untouched by this PR

🤖 Generated with Claude Code