maxP measured#8
Open
m-wojnar wants to merge 14 commits into
Open
Conversation
- maxp: add measure_only mode (passive alignment logging, LRs untouched) - scales: width-only ladder (fixed depth 12, head_dim 64, GQA 4), s1-s5 = 256..4096 dim, 4.00x non-embed per width doubling - methods: drop mup-full; add maxP-meas (c solved once from a measured alignment table via --alignment-table); wire --seed (was a no-op) - launch_sweep: 7-pt LR grid incl. 1.0, chained jobs for s4/s5 (48h cap), per-LR runtime gating via --require sentinels, job-id capture - pipeline.sh: login-node submit-only DAG (sweeps s1+s2 -> table export -> re-tune s1+s2 -> s3 brackets -> transfer proof + E1 gate -> s4/s5) - export_alignment / pipeline_analyze: table export and coordinator logic
Failed sbatch no longer leaves the next chain link dependency-less (two concurrent writers on one checkpoint dir) or the coordinator afterany list incomplete. Chained --measure-only/maxP runs are rejected at submit time: resume re-snapshots z0/w0 and corrupts alignment measurement. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Raise NCCL collective timeouts (init 600s, train 1800s) via CommConfig and bump HF hub download/etag timeouts to 120s. Both target the streaming-data stalls that tripped the default 300s NCCL watchdog on s4/s5 chain links. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adapt the LM width-ladder pipeline to a ViT second modality. Two arms only (mup-no, maxP-meas); dynamic maxP and mup-full dropped. - maxp_timm.py: width-only s1-s5 ladder (embed_dim varies, head_dim=64 and depth=12 held constant); stock timm ViT biases kept. - train.py: maxP-meas + --measure-only + --alignment-table; step-driven loop; checkpoint/resume (latest-checkpoint load, RNG restore) for chained runs. - utils.py: atomic checkpoint save/prune + load/latest helpers. - hf_vision_data.py: fix missing train shuffle (shuffle=is_train + per-pass reseeded generator). - launch_sweep.py: two-arm ladder, fixed-epoch budget, completion guard, resume chaining (afterany), runtime sentinel gating, job-id capture. - export_alignment.py / pipeline_analyze.py / pipeline.sh (new): submit-only DAG (stages A-E + coordinators B/C/D) mirroring experiments/lm/pipeline.sh. - coord_check_vit.py: width-only widths, alignment="no"; confirmed width-stable. - docs/vision_experiment_plan.md: acceptance-gate plan. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…sults - train.py / launch_sweep.py: default --num-workers 16, --prefetch-factor 2. Same RAM envelope as the old 8x4 (~9 GB prefetch buffers) but double the decode parallelism to hide random-read latency under shuffle. - docs/width_ladder_results.md: Llama3 s1-s5 maxP-meas vs muP results report. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
torch.load(map_location=device) moves the saved RNG ByteTensors onto the GPU, but torch.cuda.set_rng_state_all requires CPU ByteTensors (set_rng_state raises "RNG state must be a torch.ByteTensor"). Move each state back to CPU before applying, mirroring the existing CPU-rng .cpu() guard. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
One line at end of run: peak allocated vs device total. Useful for memory headroom checks (e.g. confirming s5 fits one GH200) and for spotting OOM risk on any scale. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Log perf/mfu = 6 * N * tokens_per_sample * samples_per_sec / device_peak_flops alongside the existing throughput metrics. N = trainable params; ViT tokens/sample = (img/patch)^2 + 1 (CLS). Device peak via DEVICE_PEAK_FLOPS env (default GH200 bf16 989 TFLOP/s). Plain 6ND (model FLOPs; excludes attention s^2 term and grad-checkpoint recompute) — fine for a headline utilization number. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ated grad ckpt GPU was data-starved (~40% util, MFU 5-7%): 16 decode workers of 72 cores. - num_workers 16->64, OMP_NUM_THREADS 32->1 (avoid worker thread oversubscription) - ship bf16 input tensors (halves pinned prefetch buffers; first layer autocasts anyway) - grad checkpointing only s4/s5 (smaller scales fit, drop ~30% recompute) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…forward) capture_initial/_capture_activations run the model without autocast, so a bf16 align_sample hit float32 weights -> dtype mismatch. Cast the 32-sample probe to float32 (its pre-bf16 dtype); data buffers stay bf16. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Run names are date-prefixed, so a resubmit on a later day forks new dirs and neither skips completed runs nor resumes chains. --run-date pins the prefix; pipelines accept RUN_DATE env and forward it to every stage. Also: login-node python module-load in both pipeline.sh (server fix). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Delete obsolete docs (DESIGN.md: fictional Config API; lr_scheduler_compatibility.md: dead MaxPScheduler).
- CLAUDE.md: trim to high-signal; add experiments/{lm,vision} + module roles; fix stale layout.
- README: lead with maxP-meas (validated static-table recipe); simplify Quick Start to the
static path; correct LM/vision sections (two arms, non-streaming, total-param scale tables).
- research_context.md: drop superseded paramR/scheduler comparison + roadmap.
- walkthrough.md: fix test-suite table to the 5 real test files.
- verify.py: _resolve_chain -> _resolve (was crashing section 8); all checks pass.
- Rename example mup_vs_maxp.py -> mup_vs_conservative.py (it compares muP vs conservative, not maxP).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
re-tune s1+s2 -> s3 brackets -> transfer proof + E1 gate -> s4/s5)