perf(fsdp2): reduce per-step host/comm overhead

[kcz358]

Summary

Two small follow-ups discussed in #179. Each is one commit, independently revertable.

1. non_blocking=True for host-to-device

`send_to_device` defaulted to `non_blocking=False`, so the H2D transfer was synchronous on every step even though the repo already defaults `dataloader_pin_memory=True`. Flipping the flag lets the copy overlap the first kernels of the following `training_step`.

Adds a one-time warning at trainer init when `dataloader_pin_memory=False`, since the flag becomes a no-op on pageable memory (mirrors the existing check in `train/hf/trainer.py`).

2. Five `all_reduce` collectives → one `all_gather`

`calculate_training_metrics` issued five separate `all_reduce` calls (`AVG` mfu, `SUM` total seq len, `AVG`/`MIN`/`MAX` seq len stats) on scalar tensors. Each one paid full collective latency and forced a `.item()` GPU→CPU sync.

Replaced with a single `all_gather_into_tensor` over a 2-element per-rank tensor `[mfu_local, seq_len_sum_local]`, then mean/sum/min/max are computed locally on the gathered tensor with one batched `.tolist()` to pull all five scalars at once.

Also dropped the `torch.tensor(flops, device=...)` wrapper at the caller — `calculate_training_metrics` now accepts the Python float from `estimate_flops` directly, removing one host→device roundtrip.

Notes

Per-step wins here are small in absolute terms — `training_metrics` was already only a few ms — but the changes are pure cleanup with no behavior change. Output metrics are bit-equivalent to before (modulo float32 reduction order, which already varied across ranks).

Test

Verified via `cicd/run_traincicd.sh --model-name qwen3_vl --gpu-count 4` locally; loss curve and reported metric values match main.

Out of scope

Discussed but deferred:

• Skipping the metrics computation entirely on non-`logging_steps` steps (largest potential win, but changes user-visible logging behavior).
• Removing the `seq_len.detach().cpu().tolist()` sync inside the metrics block. This would require changing `flops_counter.estimate_flops` to accept a tensor; cross-cutting and outside this PR.
• Async-pipelined metrics (compute on step N, log on step N+1).