v0.3 training implementation (Phases A–E): regularize, weight-averaging, deep-codebook, multitask by cataluna84 · Pull Request #10 · tiny-aya-simultaneous-translation/model

cataluna84 · 2026-06-28T15:40:32Z

v0.3 training — implementation blueprint. Operationalizes the Phase A–E
strategy from #9 (docs/next-v0.3-training-plan.md)
into concrete, file:function-level code work. Same principles (phased,
table-driven, EXA-grounded) — this is the engineering plan + execution order.
Full detail: docs/v0.3-implementation-plan.md.

Goal: ship tr-hi-s2st-v0.3 where validation improves over training, deep
codebooks learn, and both streams stay healthy.

⏱️ Revised execution order (owner)

Phase B (weight averaging) is deferred to the END, gated on evals — not a
do-it-now step. Averaging only helps if you can measure the improvement (needs
evals), and the few un-evaluated v0.2 checkpoints aren't worth blind-averaging.
Sequence:

Eval v0.2 — basic evals (ASR-BLEU min) → baseline score.
Implement + train v0.3 = Phase A (now) → C → D → E.
Eval v0.3.
Phase B last — average the final 2–3 v0.3 checkpoints (and/or v0.2), re-eval, keep only if it beats the best single on the eval metric.

average_checkpoints.py is not built now. Immediate work = Phase A.

A lot already exists — we build, not rebuild

Cosine scheduler (scheduler.py), the per-codebook CE loop + text/audio returns
(translation_loss.py — the exact Phase C hook), best_by_val, per-codebook val acc,
and unlimited checkpoint retention (keep_last_n<=0, landed in #9 → enables Phase B
averaging). Missing pieces this PR adds: early-stop, checkpoint-averaging util,
per-codebook weighting/unmasking, depth-unfreeze lever, SpecAugment.

Implementation plan (Phases A–E)

Phase	Build	Code hook (file:function)	Risk	DoD
A — Regularization	`lora_dropout`, `label_smoothing`, early-stop, `min_lr_ratio=0`, lr↓/wd↑	`lora_setup.apply_lora` (+dropout); `translation_loss` (+`label_smoothing`); `train_hierarchical` early-stop; `configs/tpu/stage2_tpu_v6e_v3.yaml`	Low	val flat-or-down; early-stop fires
B — Weight averaging (LAST, eval-gated)	`scripts/average_checkpoints.py` (SWA / topk-by-eval / EMA, CPU) — built after v0.3 trains + evals, not now	new script; opt. `src/training/ema.py`	util Low · EMA High (FSDPv2 sharded)	averaged model beats best-single on the eval metric
C — Deep codebooks	per-codebook multipliers + progressive coarse→fine unmask + staged depth unfreeze	`translation_loss` (cb-weight vector replaces `.mean()`); schedule helper; `lora_setup` depth lever	Med	cb0 acc ↑; cb1–7 non-trivial post-unmask
D — Multitask balance	composite val metric + `text_weight` curriculum/gating	`run_validation` (+`0.4·text+0.6·audio`); train-loop schedule	Low	both streams ↑; selection uses composite
E — Sweep + data + run	regularization sweep + SpecAugment-on-codes + run	`sweeps/sweep_stage2_v3.yaml`; `promote_sweep_winner` knobs; `src/data` aug	Med	sweep picks generalizing recipe; v0.3 val ↑

Run / experiment sequence (owner-revised)

Eval v0.2 (ASR-BLEU min) → baseline score.
Phase A (now) → C → D → E + smoke on the v3 config (dropout, label-smoothing, cb-unmask, early-stop all fire).
Regularization sweep → promote_sweep_winner → v6e_v3.yaml.
v0.3 run (early-stopped on composite) → eval v0.3.
Phase B last: average final 2–3 v0.3 ckpts → re-eval → release tr-hi-s2st-v0.3 only if averaging beats best-single on the eval metric.

Open decisions (owner)

In-training EMA now, or post-hoc averaging only (recommend post-hoc first)?
Depth-decoder unfreeze for v0.3, or stay frozen + rely on unmask/A/B?
Keep ~1.18M pairs vs. build synthetic/augmented set first?
Short early-stop (~2–4k) vs. longer-with-averaging horizon?

Rationale + citations: see #9's plan doc. This PR is where the v0.3 code lands.

🤖 Generated with Claude Code

…level Operationalizes the #9 Phase A-E strategy into concrete file:function work: A regularization (lora_dropout, label_smoothing, early-stop, lr/wd, min_lr_ratio=0), B weight averaging (scripts/average_checkpoints.py SWA/soup/EMA + step-0 v0.2 soup probe), C deep codebooks (per-codebook cb_weights = multipliers x progressive unmask in translation_loss; staged depth-unfreeze lever), D multitask balance (composite val metric + text_weight curriculum), E sweep redesign + SpecAugment-on-codes + run sequence. Grounded in existing hooks (cosine scheduler, per-codebook loss loop, best_by_val, keep_last_n=0 retention). Master table + run order + open decisions in the doc. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Per owner: averaging only helps if measurable via evals, and the few v0.2 checkpoints aren't worth blind-averaging. Reorder: (1) eval v0.2 baseline, (2) implement+train v0.3 = Phase A (now) -> C -> D -> E, (3) eval v0.3, (4) Phase B averaging LAST -> re-eval -> keep only if it beats best-single. average_checkpoints.py is NOT built now; immediate work is Phase A. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…arly stopping Implements Phase A of the v0.3 plan (anti-overfit; v0.2 overfit from step 1000): - lora_setup.apply_lora: + lora_dropout -> LoraConfig; wired from cfg[lora][dropout]. - translation_loss: + label_smoothing -> F.cross_entropy (train only; val stays clean CE for a true generalization metric). - early stopping: new pure src/training/early_stop.py (early_stop_step) + train-loop wiring. Patience in VAL CYCLES; best_by_val saved on every improvement so stopping never degrades the release. min_delta filters noise. Breaks -> final save. - configs/tpu/stage2_tpu_v6e_v3.yaml: r 64->16, alpha 128->32, lora_dropout 0.08, lr_lora 4.6e-4->1e-4, weight_decay 0.01->0.05, min_lr_ratio 0.1->0 (cosine to 0), label_smoothing 0.1, warmup 8%, val_every 1000->250, early_stop_patience 8 + min_delta 1e-3 (2000-step grace), fresh save_dir/run name. - tests/test_early_stop.py: 7 torch-free tests (CI runs tests/). Phase C/D/E values stay at the v2 baseline pending their PRs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…x label-smoothing site) Phase C deep-codebook learning (v0.2 cb1-7 collapsed <4%): - translation_loss: + cb_weights[CB] param -> weighted mean over active codebooks (0-weight = masked, no gradient); None = uniform mean (v2 behaviour, default). - src/training/codebook_schedule.py: pure codebook_weights/active_codebooks (multipliers x progressive coarse->fine unmask: supervise first k0 codebooks, ramp to all over progressive_unmask_fraction). 9 torch-free tests. - train loop: cb_weights cached BY VALUE (identical vectors reuse one device tensor -> only the few distinct vectors during the ramp recompile on XLA). Train-only (val stays uniform = stable generalization metric). - v3 config: gentle multipliers [1,1,1.5,1.5,2,2,2,2], unmask_fraction 0.10, k0=1. Also fixes a Phase A bug: label_smoothing was on the run_validation call; moved to run_macro_step (TRAIN only) so val/loss stays a clean CE metric. Phase C3 (staged depth-decoder unfreeze) deferred: mid-run unfreeze + param-group changes under FSDPv2/SPMD is complex + TPU-unvalidatable now; plan marks it opt-in. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…R group) Opt-in lever to adapt the frozen Moshi depth decoder to TR/HI acoustics: - freeze_depth_internals: + unfreeze_last_n_blocks -> unfreezes the last N depth transformer blocks at SETUP (before FSDP wrap). Design pivot from the deferred mid-run unfreeze (risky param-group surgery under FSDPv2/SPMD): from-start unfreeze + the Phase C2 unmask (which gates deep-codebook gradients) gives the same 'staged' effect -- the blocks only learn once their codebooks unmask -- without any mid-run mutation. - get_param_groups: + depth_blocks group at lr_depth_blocks (default lr_depth x0.1) via the pure src/training/param_classify.is_depth_block; empty group -> dropped, so v2/other configs are unaffected. - v3 config: depth_unfreeze_blocks 0 (off, opt-in) + lr_depth_blocks 1e-5. - tests/test_param_classify.py (3 torch-free tests). Phase C complete (C1+C2+C3). The unfreeze grad/FSDP behavior is TPU-validated on the smoke (needs a real model); the name-classification logic is unit-tested. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…-unfreeze grad flow Short (8-step) v3 smoke: depth_unfreeze_blocks=2 + unmask k0=6/frac=0.5 -> K=6,7,8 (3 distinct cb_weights graphs, with repeats to confirm cache hits). No val/ckpt/wandb. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…yers) — TPU smoke The TPU smoke caught the original C3 unfreeze as a silent no-op (0M, depth_blocks group absent). Two root causes, both from xla_grad_checkpoint wrapping .layers in a _ScannedLayerStack: 1. module-walk (model.depth_decoder.layers[-n:].parameters()) saw 0M -- the stack hides real layers in a tuple to avoid double-registration. 2. real param names become depth_decoder.layers.layers_list.<i>.layer.* -- the original regex missed the layers_list segment. Fix: param_classify.depth_block_layer_index parses the index after layers[_list] (regex ), and the unfreeze enumerates model.depth_decoder.named_parameters() (the same source the freeze loop uses) + asserts uf>0 so a 0-param unfreeze can never pass silently again. VERIFIED on the v6e-8 (scan-wrapped decoder): depth-unfreeze 206M, depth_blocks group 205.5M @ lr 1e-5. + scan-form unit tests. is_depth_block still excludes the scan-wrapped BACKBONE (no depth_decoder segment). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…ified Smoke config: depth_unfreeze_blocks 2->0. depth_unfreeze=2 OOM'd on v6e-8 (28.96G > 27.87G free) -- +205M trainable + AdamW + grad-ckpt backward exceeds HBM. Run #1 validates the unmask recompile-caching path that ships (depth-unfreeze is opt-in/off in production); C3 grad flow validated separately at reduced batch. Pre-flight gate (i) for the fresh 26K v0.3 run: verified eval CB1-7 delay indexing matches training. Eval reuses the training forward; audio_targets and eval gt are the same delayed model_audio_codes tensor with the same next-token shift. CB1-7~0 is a real signal (frozen depth + undertrained projection), not an eval read artifact -> a fresh run reports it honestly. Doc: Phase C3 depth-lever section + the scan-wrap (layers_list) naming finding. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Verified from training logs which dataset each run actually used and fixed the cards (also pushed live to the HF repos): - v0.2 (tr-hi-s2st-v0.2): trained on `fleurs-tr-hi-mimi-encoded` (FLEURS real+TTS, ~8.3k) — NOT the intended synthetic corpus. The launcher's HF_DATASET default pointed at the `fleurs-` sibling repo. Added an honest "Dataset disclosure" callout. - v0.1 (tr-hi-s2st-v0.1): actually trained on the synthetic `tr-hi-mimi-encoded` (FLORES/OPUS/conversational, 1,178,302 train pairs per its train log). Its dataset links were right; the prose mislabeled it "FLEURS" (FLORES != FLEURS). Corrected the data description + license-scope; added a clarification note. - v0.3 (new draft, NOT pushed): training-in-progress card documenting the corrected synthetic data source and the v0.2 honest mistake. Held until the v0.3 run produces weights, then publish card + checkpoints together. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…eight curriculum New torch-free `src/training/multitask.py`: - composite_val_loss(text, audio, 0.4/0.6): audio-heavier weighted mean of the RAW per-stream val losses. - text_weight_at(step,...): quantised linear anneal start->end over the first `frac` of the run (piecewise-constant so XLA recompiles a bounded number of times, not every step). Wiring in train_hierarchical.py: - run_validation now returns `val/composite` (weights from loss_cfg). - best_by_val + early stopping select on `val/composite` (fallback val/loss), so neither stream can be sacrificed to game the other. - train loss uses a per-step `_text_weight_for(step)` curriculum; val keeps a fixed text_weight so val stays a clean comparable metric. Config (stage2_tpu_v6e_v3.yaml): text_weight_start/end/curriculum_frac/quantum (0.5->0.3 over 25%, q=0.05) + composite_text_w/audio_w (0.4/0.6). Tests: tests/test_multitask.py (7 cases) green; existing phase tests + seam check still pass; train script py_compile OK. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

cataluna84 · 2026-06-29T04:31:32Z

✅ Phase D — Multitask balance (commit `91d4bf2`)

Composite validation metric + text_weight curriculum, so checkpoint selection and early stopping stop letting one stream game the other.

New src/training/multitask.py (torch-free, CI-testable):

composite_val_loss(text, audio, 0.4/0.6) — audio-heavier weighted mean of the raw per-stream val losses (audio = the CB1–7 bottleneck).
text_weight_at(step, …) — quantised linear anneal start→end over the first frac of the run. Piecewise-constant on purpose: a smooth per-step ramp would recompile the loss HLO every step; quantising to a 0.05 grid bounds it to ≤5 distinct values (same cache-by-value discipline as the Phase-C unmask).

Wiring (train_hierarchical.py):

run_validation now returns val/composite; best_by_val + early stopping select on it (fallback val/loss for older configs).
Train loss uses a per-step _text_weight_for(step) curriculum; val keeps a fixed text_weight so it stays a clean, comparable metric.
Val print + wandb now surface val/composite.

Config (stage2_tpu_v6e_v3.yaml): text_weight_start/end/curriculum_frac/quantum = 0.5→0.3 over 25%, q=0.05; composite_text_w/audio_w = 0.4/0.6.

Checks: tests/test_multitask.py (7) green · existing phase tests green · backend-seam check green · py_compile OK.

Also in this push — docs(model-cards) (6198a56): corrected dataset attribution after verifying the actual training logs. v0.1 trained on the synthetic tr-hi-mimi-encoded (1,178,302 pairs) — its prose mislabeled it "FLEURS"; v0.2 regressed to the fleurs- sibling repo (the launcher's HF_DATASET default). Both live HF cards updated; v0.3 card drafted and held until weights exist.

Next: Phase E — sweep_stage2_v3.yaml + promote-mapping, and flag-gated SpecAugment-on-codes.

Smoke config now exercises all three phases in one short run and was verified live on v6e-8: A: lora.dropout=0.08 + label_smoothing=0.1 (train-only: step-1 train CE 14.77 vs clean val 9.32) + early-stop armed (patience=2). C: per-codebook weighting + unmask (k0=6 -> K 6/7/8); loss 14.77->9.84 monotone, nonfinite=0, all param-group grads nonzero; recompiles only on first-occurrence. D: text_weight curriculum (0.5->0.3, quantised) fires; inline TPU val works and val/composite = 0.4*text + 0.6*audio (8.1542 verified), improves across val cycles so best_by_val/early-stop select on it. Changes: enable inline val (val_every=4 -> steps 4 & 8), arm early-stop (patience=2, min_delta=1e-3), add Phase-D loss knobs (curriculum + composite). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

cataluna84 · 2026-06-29T05:26:00Z

🔬 Smoke: Phase A + C + D all fire on v6e-8 (commit `13b0a2f`)

Extended the v3 smoke to exercise every lever in one short run; verified live.

Phase	Evidence
A	`[lora] r=16 exclude_top=2` (dropout plumbed); `label_smoothing=0.1` train-only → step-1 train CE 14.77 vs clean val 9.32; early-stop armed (`patience=2`), runs on composite
C	loss 14.77→9.84 monotone, `nonfinite=0` every step; depth/model_audio_embed/projection/text_embed/lora grads all nonzero; flat ~4 s/step except first-occurrence compiles
D	`[text-curriculum] 0.5->0.3` fired; inline TPU val works → `val/composite=8.1542 (text=9.148 audio=7.492)` = 0.4·t+0.6·a exactly, improves to 7.87 → best_by_val/early-stop select on it

The only non-trivial timing was step 5 (~1012 s) = the one-time val-graph compile bundled into that step's wall-time; step 8's val reused the cached graph (4.06 s). Recompiles are first-occurrence only — unmask + val caching both confirmed.

➡️ Proceeding to Phase E (sweep_stage2_v3 + promote-mapping + flag-gated SpecAugment-on-codes).

- sweeps/sweep_stage2_v3.yaml: Bayesian + hyperband search over the regularization knobs (lr_lora, lora_r in {8,16,32}, lora_dropout, weight_decay) with all v0.3 levers (A/C/D) ON. Metric = val/composite (minimize). - configs/tpu/stage2_tpu_v6e_v3_proxy.yaml: short-horizon proxy (2.5k steps, no checkpoints) the sweep points at, so the grid carries only RECIPE knobs and promote never drags the proxy horizon into production. - train_hierarchical.py: add `--lora_dropout` sweep arg (-> lora.dropout) and `define_metric("val/composite", summary="min")` so each trial is scored at its BEST early-stopped composite, not its last value. - promote_sweep_winner.py: PARAM_MAP gains lora_dropout -> (lora, dropout); default --metric is now val/composite (v2's val/audio_loss still selectable). - tests/test_sweep_promote.py: v3 coverage (every grid param promotable, points at the v3 proxy, optimises composite, lora.dropout key exists). 15 tests green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

New src/data/spec_augment.py (torch-free planning, importable in CI): - plan_time_masks: random (start,len) frame spans, clipped to T. - plan_codebook_drops: distinct codebooks to blank, never all of them. - apply_spec_augment: in-place mask of a [B,CB,T] stream; short-circuits before touching the tensor when disabled. Adaptation of SpecAugment to discrete Mimi codes: time-masking (blank frame spans -> SILENCE) + codebook dropout (blank whole RVQ levels). Applied to the INPUT (user) audio stream ONLY -- never targets, never val. Wiring: - InterleavedCollator gains `spec_augment` (off unless enabled); masks user_audio_codes capped to each sample's real frame count. - train_hierarchical builds a SEPARATE train collator with SpecAugment; the shared collator stays clean for validation. - configs/tpu/stage2_tpu_v6e_v3.yaml: `spec_augment` block, DEFAULT OFF (ablation / sweep axis for later). Tests: tests/test_spec_augment.py (9) green; seam check green; full torch-free suite green; py_compile OK. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

cataluna84 · 2026-06-29T05:38:43Z

✅ Phase E — Sweep redesign + SpecAugment (commits `4c58892`, `af2eb6e`)

E.1 — regularization sweep on the composite metric

sweeps/sweep_stage2_v3.yaml: Bayesian + hyperband over lr_lora, lora_r ∈ {8,16,32}, lora_dropout, weight_decay, with all v0.3 levers (A/C/D) ON. Metric = val/composite (minimize), logged summary="min" so each trial scores at its best early-stopped composite, not its last value.
configs/tpu/stage2_tpu_v6e_v3_proxy.yaml: short-horizon proxy (2.5k steps, no checkpoints) the sweep points at — so the grid holds only recipe knobs and the proxy horizon never leaks into production via promote.
train_hierarchical.py: --lora_dropout sweep arg (→ lora.dropout) + the composite define_metric.
promote_sweep_winner.py: PARAM_MAP gains lora_dropout → (lora, dropout); default --metric now val/composite.

E.2 — SpecAugment-on-codes (flag-gated, default OFF)

src/data/spec_augment.py (torch-free planning): time-masking (blank frame spans → SILENCE) + codebook dropout (blank whole RVQ levels), adapted from spectrogram SpecAugment to discrete Mimi codes.
Applied to the input (user) stream only — never targets, never val. A separate train collator carries it; the val collator stays clean.

Checks: test_sweep_promote (15), test_spec_augment (9), full torch-free suite, seam check, py_compile — all green.

This completes the code for Phases A–E. Remaining before the run: data repoint to tr-hi-mimi-encoded (+ build the training subset), then sweep → v0.3 run → eval → Phase B (post-hoc checkpoint averaging, eval-gated).

Data reprep (fixes the v0.2 FLEURS mistake): - startup_script.sh HF_DATASET default -> tiny-aya-translate/tr-hi-mimi-encoded (the synthetic FLORES/OPUS/conversational corpus, 1.24M pairs; the non-`fleurs-` repo v0.1 actually used). Extraction rewritten for the repo's layout: loop over mimi_encoded_batch*.tar.gz (each has a top-level encoded/ dir) -> $DATA_DIR, so files land at /mnt/data/encoded/<name>.pt (dataset _resolve matches basename). ~1.24M samples => ~4M files; fits the v6e VM (~12.5M inodes free, 64G disk). v6e_v3.yaml data paths unchanged (same /mnt/data/{encoded,splits} layout). Parallel sweep (Phase E.1 width, not depth): - scripts/tpu/launch_sweep.sh: one `wandb agent` per TPU worker -> N independent single-host trials. On a v6e-64 that's 8 parallel trials (~8x throughput); documents the multi-host isolation caveat (validate global=8 not 64) and the robust N-x-v6e-8 fallback. Rationale: the sweep is embarrassingly parallel and LoRA generalises worse past batch ~128, so width beats a 64-chip SPMD mesh. - sweeps/README.md: v0.3 parallel-sweep runbook. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Stand up the embarrassingly-parallel regularization sweep across N independent v6e-8 spot slices (allocation verified: probe v6e-8 ACTIVE in ~300s; 64-chip grant fits 8x8). - scripts/build_sweep_subset.py (+test, 7 cases): carve a source-stratified, direction-balanced ~200K subset of the 1.24M corpus. The sweep only needs to RANK recipes, and a 200K subset cuts per-slice staging ~6x (the full run still uses all 1.24M on one slice). - scripts/tpu/stage_sweep_subset.sh: one-shot prep -> selective-extract only the subset's files from the HF batch tarballs, pack, upload to gs://.../data/sweep-subset-200000.tar.gz (~2GB). - startup_script.sh SWEEP mode: `sweep-id` + `sweep-data-gs-uri` metadata -> the slice rsyncs the small subset from GCS (no 8x HF rate-limit) and runs `wandb agent` instead of a single-config train. launch_qr.sh forwards both. - scripts/tpu/launch_sweep_fleet.sh: package repo -> GCS, then provision N v6e-8 spot slices (tinyaya-sweep-<i>) with the sweep metadata; the production v6e-8 is untouched. Rationale (see PR brainstorm): width beats a 64-chip mesh -- LoRA generalises worse past batch ~128, and a v6e-64 has the same 8-host staging cost but adds isolation risk + single-preemption blast radius + needs the v6e-8 deleted. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

cataluna84 · 2026-06-29T09:09:22Z

✅ 8× v6e-8 parallel sweep fleet (commit `bde08a9`)

Allocation verified (probe v6e-8 ACTIVE in ~300 s; 64-chip grant fits 8×8), then built the fleet enablement:

build_sweep_subset.py (+7 tests): source-stratified, direction-balanced ~200K subset of the 1.24M corpus. The sweep only ranks recipes, and 200K cuts per-slice staging ~6× (full 1.24M still used for the single final run).
stage_sweep_subset.sh: one-shot — selectively extracts only the subset's files from the HF batch tarballs → packs → gs://…/data/sweep-subset-200000.tar.gz (~2 GB).
startup_script.sh sweep-mode: sweep-id + sweep-data-gs-uri metadata → slice rsyncs the small GCS subset (no 8× HF rate-limit) and runs wandb agent (not a single-config train). launch_qr.sh forwards both.
launch_sweep_fleet.sh: packages repo → GCS, provisions N tinyaya-sweep-<i> v6e-8 spot slices with the sweep metadata. Production v6e-8 untouched.

All torch-free tests + bash syntax green.

Execution runbook (live, on your go-ahead):

wandb sweep sweeps/sweep_stage2_v3.yaml                 # -> SWEEP_ID
bash scripts/tpu/stage_sweep_subset.sh                  # -> SWEEP_DATA_GS_URI (~2GB)
SWEEP_ID=... SWEEP_DATA_GS_URI=... N_SLICES=7 bash scripts/tpu/launch_sweep_fleet.sh

Then promote the winner (--metric val/composite) into stage2_tpu_v6e_v3.yaml and run the single v0.3 run on one v6e-8.

~11% of split rows reference .pt files absent from the published batch tarballs; the dataset torch.load()s directly and would crash a sweep trial on a miss. The stager now keeps only rows whose .pt is in the staged encoded/ dir (200K -> ~178K consistent). Already applied + re-uploaded to the live GCS subset. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Confirmed the synthetic tr-hi-mimi-encoded corpus ships NO text alignments (full tarball scan: 270k .pt, 0 _src/_tgt.json), so the text/inner-monologue stream can't be supervised (text loss structurally 0 -- same as v0.1). Switch to AUDIO-ONLY: - v3 + v3_proxy configs: text_weight=0, curriculum off, composite = val_audio (composite_text_w=0, composite_audio_w=1). Also fix the sweep crash: stage_sweep_subset.sh now also builds a held-out VAL subset (NVAL, default 1000) and extracts its .pt, so trials don't FileNotFoundError at the first validation. Both splits filtered to .pt actually present. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Record the full public-release brainstorm (docs/v0.3-public-release-plan.md): models (v0.1/v0.2 corrected + v0.3 audio-only), ALL checkpoints, dataset cards (with the no-alignments note), code repos, licensing, eval, reproducibility, the honest narrative (v0.1 synthetic/text-gap, v0.2 FLEURS mistake, v0.3 audio-only + swept), and run durability (TPU-side tmux, --resume auto). Checkpoint retention confirmed correct for "save everything": keep_last_n=0 -> prune_checkpoints returns immediately (unlimited retention); every checkpoint kept to GCS. No change needed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Two arms to empirically settle the global-batch question on the real model/data before the full run, both using the regularization-sweep winner (r=8/alpha=16, dropout=0.15, wd=0.10), AUDIO-ONLY, reusing the 200K GCS subset: - b256: batch_size=4 grad_accum=1 -> global 256 (LoRA-safe), lr=2.8e-4 (as swept), 2500 steps = 640k samples. - b2048: batch_size=8 grad_accum=4 -> global 2048 (per-chip stays 8), lr=7.9e-4 (sqrt(8)x), 313 steps = 640k samples (EQUAL data budget for a fair comparison). Decision: if b2048 matches b256's final val/audio at equal budget -> use 2048 for the full run (8x fewer steps, much faster); if it regresses -> the LoRA large-batch penalty is real, use 256. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…n knobs Pivot from v0.3 *regularization* tuning (which fixed the 200K-subset overfit) to a *capacity* sweep for the scale regime: with the full corpus + multi-host + global batch 256, overfitting is no longer the binding constraint, so the regularization-tuned winner (r=8 / dropout .15 / wd .10 / targets q,v,embed) likely UNDER-fits. This sweep explores the capacity/optimization axes instead. Code (backend-agnostic, shared train path): - lora_setup: add use_rslora (alpha/sqrt(r) rank-stable scaling) to apply_lora and to the custom LoRAEmbedding, so high ranks (32/64) don't gradient-collapse (vanilla alpha/r suppresses high-rank adapters). Plumbed from cfg.lora.use_rslora. - train_hierarchical --sweep: map the new structural axes onto cfg.lora.* (--use_rslora bool, --target_modules as JSON/list/csv, --lora_exclude_top int); robust _parse_target_modules handles every way W&B serializes a list. Namespace save_dir by the (rendezvous-shared) W&B run id so EVERY sweep trial keeps ALL its checkpoints with no cross-trial step-filename collision. Sweep design (2-stage; research-backed for ~10 sequential trials on one slice - pure Bayes wastes trials on categorical structure): - configs/tpu/stage2_tpu_v6e16_scale_proxy.yaml: de-regularized proxy (dropout .05, wd .01), global batch 256 (4x4x16), grad-clip ON (1.0), 1500-step common horizon, checkpointing ON, keep ALL checkpoints. - sweeps/sweep_stage2_scale_grid.yaml - Stage 1: structural grid over target_modules (q,v -> +attn k,o -> +MLP), rsLoRA on, ~3 trials. - sweeps/sweep_stage2_scale_bayes.yaml - Stage 2: Bayesian lr x rank{8,16,32,64} on the Stage-1 winner; Hyperband min_iter=750 (conservative, won't cull slow-but-better high-rank arms too early). Both carry the MULTI-HOST EXECUTION CAVEAT: drive via ONE coordinator that broadcasts identical trial args to all 4 hosts of the v6e-16 SPMD mesh; per-host wandb agents would desync (each pulls a different trial). Tests: tests/test_sweep_capacity_knobs.py (target_modules parser across all W&B list serializations; rsLoRA scaling). Full suite green (74 passed, 1 skipped). Also track the previously-uncommitted v6e16/v6e32 b256 batch configs - the v6e16 one is the live baseline (= sweep arm 0) now running on the v6e-16 slice. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

cataluna84 · 2026-06-30T17:54:34Z

v0.3-scale: capacity sweep + long-horizon hardening (plan + status)

Posting the full plan for visibility. This builds on the v0.3 work in this PR but pivots the objective: v0.3 tuned regularization to fix the 200K-subset overfit; in the scale regime (full corpus, multi-host, global batch 256) overfitting is no longer the binding constraint, so the regularization-tuned winner (r=8 / dropout .15 / wd .10 / targets q,v,embed) very likely under-fits. We now sweep the capacity / optimization axes instead.

Backed by 2× exa-research-pro deep reports + web research (rsLoRA, lr×rank scaling, batch-size bias, W&B-sweep-with-few-trials, S2S proxy metrics, long-horizon spot-TPU failure modes).

🖥️ TPU spun up

v6e-16 (Trillium) spot — tinyaya-v6e16-batch-ew4, zone europe-west4-a, 4 hosts × 4 chips.
This was the fallback after multi-host v6e spot proved un-gettable above 16 chips: v6e-64 failed 10/10 and v6e-32 failed in both zones (all PROVISIONING → SUSPENDING → FAILED spot reclaim). v6e-16 is the smallest multi-host footprint and provisioned cleanly.
First-ever multi-host SPMD validation succeeded: global=16 local=4 strategy=fsdpv2_lora, model loaded, W&B run v6e16-batch-256 live, no NaN.
The currently-running b256 job is our baseline (= sweep arm 0), config configs/tpu/stage2_tpu_v6e16_b256.yaml (per-chip batch 4 × grad_accum 4 × 16 = global 256, lr 2.8e-4, audio-only).

🔬 Part A — the capacity sweep (2-stage W&B, on the v6e-16)

One wandb-driven job per trial on the 16-chip mesh; trials run sequentially. Swept on the frozen 200K GCS subset for comparability, then the winner is promoted to the full corpus.

Why 2-stage (not one all-axes Bayesian): with only ~6–15 sequential trials, pure method: bayes wastes most of them exploring categorical/structural choices. Research-recommended structure:

Stage 1 — structural grid (sweeps/sweep_stage2_scale_grid.yaml, ~3 trials): OFAT over target_modules — q,v → +attn (k,o) → +MLP (gate,up,down) — with rsLoRA on, rank fixed 16, lr fixed 2e-4. → pick the best structure.
Stage 2 — Bayesian lr×rank (sweeps/sweep_stage2_scale_bayes.yaml, ~6–7 trials): structure fixed to the Stage-1 winner; sweep rank ∈ {8,16,32,64} × lr_lora (log 1e-5…5e-4) jointly. rsLoRA on (α/√r) so high ranks don't gradient-collapse. Hyperband min_iter=750 (conservative — won't cull slow-but-better high-rank arms too early).

Axes (all 4 requested), de-regularized: rank↑ + rsLoRA · more target modules (attn + MLP) · wider layer coverage (lora_exclude_top 2→0 probe, memory-gated) · de-regularize (dropout .15→.05, wd .10→.01) + lr retune.

Proxy + promotion: objective val/composite + per-codebook token accuracy (correlates with the eventual ASR-BLEU target). Winner gets a medium-length full-corpus confirmation run before committing to the full long run (guards against subset→full mismatch and proxy gaming).

All checkpoints saved: keep_last_n: 0 + save_every: 500 + save_dir namespaced by W&B run id so every trial's checkpoints are isolated.

🧱 Part B — long-horizon hardening (gate before the full production run)

Failure modes a multi-hour run on spot multi-host TPU must survive:

Preemption/resume: full-state checkpoints (Adam moments + LR sched + global_step + dataloader cursor + W&B run id); auto-resume from latest GCS checkpoint on a fresh slice.
bf16 stability: grad clipping enabled (1.0) + loss/grad-spike skip; XLA_NO_SPECIAL_SCALARS=1 + nan_to_num already in place.
FSDP/SPMD topology pinning: sharded checkpoints only resume on the same chip count (16) — pin the full run to v6e-16; consolidate→reshard offline if topology changes.
RVQ/codebook: monitor per-codebook usage/accuracy for collapse (levers per_codebook_multipliers, progressive_unmask_fraction already present).
XLA: fixed-shape padding (max_frames 300), watch for recompiles/mark_step deadlocks.
Ops: silent-stall watchdog + ntfy alerts, GCS write jitter + lifecycle policy.

⚙️ Execution model — detached & workstation-independent

The sweep runs long; the workstation will be powered off, so it is only a launch/monitor client. Everything long-running runs in tmux on the TPU VM with the W&B sweep controller server-side. For self-healing across spot reclaims while the workstation is off, a tiny always-on supervisor VM (the GCE prober pattern) re-provisions a v6e-16 on FAILED and the startup script auto-resumes from the latest checkpoint. Progress/failure alerts go to the ntfy topic (phone).

⚠️ Multi-host execution caveat (open infra item): on the single v6e-16 (4 hosts → one 16-chip SPMD mesh), a plain per-host wandb agent would desync — each host pulls a different trial, corrupting the mesh. Each trial must run as one 16-chip job with identical args on all 4 hosts, driven by a single coordinator that broadcasts the trial's args. The sweep launcher for this is the next infra piece.

📦 This commit (`ec5ba86`)

src/model/lora_setup.py: use_rslora (α/√r) in apply_lora + custom LoRAEmbedding.
scripts/train_hierarchical.py: --sweep mapping for use_rslora / target_modules (robust list parser) / lora_exclude_top; run-id-namespaced save_dir.
configs/tpu/stage2_tpu_v6e16_scale_proxy.yaml + the two sweeps/sweep_stage2_scale_*.yaml.
tests/test_sweep_capacity_knobs.py (full suite green: 74 passed, 1 skipped).
Also tracks the previously-uncommitted v6e16/v6e32 b256 batch configs.

⏭️ Next

Build the multi-host sweep coordinator/launcher (broadcasts trial args to all 4 hosts; runs detached in tmux).
TPU smoke of one Stage-1 grid trial (mesh forms, rsLoRA active, HBM fits the +MLP/exclude_top=0 arms, namespaced checkpoint lands, resume works).
Run Stage 1 → Stage 2 → promote winner → medium full-corpus confirmation → full production run.
Land remaining Part-B hardening (checkpointing-resume audit, watchdog, GCS lifecycle).
Rotate the previously-exposed HF/W&B tokens before public release.

Audit of the resume path: optimizer Adam moments + LR scheduler ARE already restored (post-wrap block in train_hierarchical; the stale load_checkpoint docstring claimed otherwise). The real remaining gaps were W&B run discontinuity and broken checkpoint discovery. - checkpointing: get_checkpoint_dirs/find_latest_checkpoint now match the actual `step_NNNNNN` save layout (was filtering a `checkpoint_` prefix -> local `--resume auto` found nothing; GCS only worked via gcsfs). Excludes best_by_val, sorts by step number, GCS via gsutil (drops the gcsfs dep). Add read_checkpoint_metadata() (cheap `gsutil cat`) and fix the misleading load_checkpoint docstring to describe the real (working) resume flow. - train_hierarchical: on resume, recover the original W&B run id from checkpoint metadata and re-init wandb with id=+resume="allow", so a preempted run continues the SAME dashboard run instead of fragmenting into one run per restart. Save wandb_run_id into every checkpoint's metadata (periodic / best / final). run_id capture moved out of the sweep-only guard so the production run records it too. Tests: 6 new (step parsing, latest-dir discovery excludes best_by_val, metadata read/missing). Full suite green (80 passed, 1 skipped). Remaining for the full run (tracked, NOT needed by the short sweep trials): RNG persistence (minor), startup-script `--resume auto` wiring, and TPU validation of the optimizer-state reshard on a real preemption. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…(build only) The single v6e-16 is 4 hosts forming ONE 16-chip SPMD mesh, so a per-host `wandb agent` desyncs (each host pulls a different trial -> mismatched programs -> corrupt mesh). This adds a coordinator that runs each trial as ONE 16-chip job with IDENTICAL args on all hosts, broadcast via a shared GCS control file. Nothing is launched here -- build + tests only (the slice is idle post-baseline). - scripts/tpu/sweep_coordinator.py (host-0 driver): `--stage grid` enumerates a sweep YAML's parameters (deterministic, no W&B engine); `--stage bayes` pulls suggestions from `wandb.controller`. Publishes current_trial.json, waits for all hosts' done-markers, reads the trial's val/composite from the W&B run summary, writes a completion marker so a coordinator reboot RESUMES the grid instead of restarting. Pure core (enumerate_grid / build_trial_args / load_sweep_parameters) is dep-free and unit-tested; `--dry-run` prints the plan. - scripts/tpu/sweep_host_loop.sh (all hosts): polls the control file and runs `train_hierarchical.py --config <proxy> --sweep <identical args>` with the same PJRT/XLA/secret env as startup_script.sh, so the mesh forms per trial; writes a per-host done-marker (unique by hostname). - scripts/tpu/launch_sweep_coordinated.sh: packages code -> GCS, refreshes all hosts, starts host loops (tmux sweephost, --worker=all) + coordinator (tmux sweepcoord, --worker=0). Detached; workstation can be powered off. - docs/sweep-coordinated-runbook.md: Stage-1 grid -> Stage-2 bayes -> promote. - target_modules serialized as COMPACT json so the host loop word-splits it as one token; _parse_target_modules (train side) round-trips it. Verified: coordinator --dry-run enumerates the real grid YAML to exactly 3 structural trials with correct args; bash -n on both scripts; 4 new unit tests; full suite green (84 passed, 1 skipped); backend seam check holds. Next: launch Stage-1 grid on the idle slice when ready (user-gated). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…e-16 Live-launch surfaced three bugs in the (previously build-only) coordinator path: - launch_sweep_coordinated.sh assumed the login user, but /opt/tinyaya + venv + uv + data + secrets all belong to ROOT (startup runs as root). Run every remote action via `sudo -H` (HOME=/root so uv + LIBPYTHON resolve); fetch untars with sudo; verify each tmux session started. - sweep_host_loop.sh exported WANDB_SWEEP_ID="" for the grid stage (no server-side sweep) -> wandb aborts with "Sweep ID cannot be empty". Only export it when non-empty; export WANDB_RUN_ID separately. - train_hierarchical honored only an explicit id=None and ignored WANDB_RUN_ID, so each trial got a fresh run id != the coordinator's pre-generated id (breaking metric read-back + checkpoint namespacing). Fall back to WANDB_RUN_ID env. - sweep YAMLs: add entity/project so the (Stage-2) sweep lands in tinyaya-stage2-tpu. Verified live: Stage-1 grid trial 1 now inits wandb (run id == coordinator id) and builds the model on the 16-chip mesh; host loops + coordinator run detached (root) in tmux on the v6e-16. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…resilience The coordinated sweep runs in tmux on the TPU (survives workstation shutdown), but a spot preemption of the v6e-16 kills the on-TPU coordinator (and the QR reboot relaunches the single smoke, not the sweep). This adds an always-on GCE supervisor (e2-small) so the sweep is fully hands-off: - scripts/tpu/sweep_supervisor.sh: loop that watches the QR + coordinator. ACTIVE+coordinator-dead -> relaunch (kills stray smoke first); QR FAILED/missing -> re-provision the v6e-16 then relaunch (grid resumes -- completed trials skipped); "sweep complete" -> ntfy the user for the Stage-2 decision. Periodic ntfy heartbeat with the W&B URL. - scripts/tpu/launch_sweep_supervisor.sh: packages the repo, creates the VM (default compute SA + cloud-platform), runs the supervisor as a Restart=always systemd unit. Deployed: tinyaya-sweep-supervisor (us-central1-a) watching tinyaya-v6e16-sweep-ew4. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The first supervisor used single gcloud calls for QR state + coordinator liveness, so a transient SSH/API blip would falsely conclude the TPU/coordinator was gone and DESTROY the running sweep (relaunch, or re-provision). Harden: qr_state + coord_alive retry 3x; transient/unknown states are treated as 'wait' (never destructive); relaunch needs 2 consecutive coordinator-down observations; re-provision only on 2 consecutive confirmed FAILED (spot reclaim). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Anneal leg completed 2026-07-20: 65,250 -> 76,250 linear LR->0, best val composite 2.8199 @ 76,000 (vs plateau 2.9048) -- the R5-predicted WSD harvest at full scale. Card: banner + annealed val-table column + anneal-leg subsection + Checkpoints section rewritten for the FULL ~87-point suite (repo flipped public on user decision; private 50 GB cap no longer applies; VM-side backfill in flight). do.md/PLAN ticked. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The naive suite publisher commits ONE FILE AT A TIME (~10 commits per checkpoint), so an ~87-point suite blows the hub's 128-commits/hour cap in minutes; its 3x180s retry loop then gives up (hit live during the v0.3 public flip after 17 branches). New script: - upload_folder => ONE atomic commit per checkpoint per mode (~10x fewer commits: 87 x 2 modes = ~174, ~2 hourly windows instead of ~12) - CommitBudget paces against the cap and sleeps to the window reset; server-side rate-limit errors are caught and treated as window-exhausted, never as failure - resumable/idempotent: each pass asks the hub which labels already carry WEIGHTS (the payload-less-branch trap) and publishes only the gaps - deepest-step-first ordering (release candidates never wait on the tail) - GCS status publishing for VM-side tmux monitoring Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Several supporting docs were still framed at the plateau-complete milestone (65,250 / 2.9048) and predated the anneal, the public flip, the preemption, and the prune. Bring them all current: - anneal result everywhere: 65,250->76,250, best 2.8199 @ step 76,000 - repo PUBLIC + full ~89-checkpoint suite (drop 'interim 12 / 78-at-flip') - slice PREEMPTED post-run (QR SUSPENDED); teardown = delete QR - GCS pruned (attempt-1 + xla-cache gone; r2 411 GB kept; bucket removed) - eval targets -> annealed best(76,000)/final/LAWA; best_by_val = 76,000 Files: README banner, release-plan header, reval-report A.6, evals-runbook Post-run + targets, do.md GPU-eval entry, tpu-capacity-log, evals-plan, .claude/{memories,PROGRESS,PLAN}. Model card was already current (pushed). Historical plateau/2.9048 mentions kept where they are correct narrative. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

cataluna84 · 2026-07-22T05:30:41Z

v0.3 training complete — plateau + WSD anneal (final numbers)

Run: v0.3-long-horizon-mh-r2 / xzcb60bl · Project: https://wandb.ai/cataluna84/tinyaya-stage2-tpu · Model (public): https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3

TR↔HI speech-to-speech (Cohere2 LoRA backbone + frozen Moshi depth decoder over Mimi codes), multi-host TPU v6e-16, global batch 32, WSD schedule. The run had two phases on one contiguous W&B run:

Phase 1 — plateau (steps 0 → 65,250). Warmup 1,100 → stable plateau at lr 1.716e-4. Early stopping (patience 10 val cycles) fired at 65,250, before the scheduled decay. Best plateau val composite 2.9048 @ 62,750.

Phase 2 — WSD anneal (65,250 → 76,250). Resumed the 65,250 checkpoint (weights + Adam + RNG), extended max_steps to 76,250 so the whole remaining horizon = the pre-registered 11,000-step linear LR 1.716e-4 → 0 (the shape round-3 probe R5 validated vs cosine/plateau). Val improved on essentially every 250-step cycle. Zero preemptions during training.

Validation metrics (teacher-forced, fixed 3,200-sample gate; composite = 0.4·text + 0.6·audio):

metric	step 250	plateau best (62,750)	anneal best (76,000)
val composite	6.719	2.9048	2.8199
text loss / ppl	4.328 / 75.8	0.486 / 1.63	0.398 / 1.489
audio loss	8.313	4.518	4.435
text token acc	25.6%	94.4%	96.6%
cb0 (semantic) acc	10.4%	40.4%	41.5%
cb1–7 acc	10.5→0.1%	21.0/17.4/11.5/8.8/7.2/6.1/5.9%	22.0/18.2/12.1/9.2/7.5/6.4/6.1%

The anneal harvested −0.085 composite (2.9048 → 2.8199), −0.09 text loss, and lifted every deep codebook to its run-high — the WSD decay phase working as the literature predicts, confirmed at full scale. Deep-codebook collapse (v0.2: cb0 ~14%, cb1–7 <4%) stays resolved. Final = step 76,250 ≈ best.

Model card (the repo's README.md, full card) covers: the two-phase training run + this table; click-to-play audio demos at 13 milestones (source / GT-target / generated); the full ~89-checkpoint suite (log-spaced early + every 1,000 through 76,000, best, final) browsable in the file tree and as revision="…" git branches (Pythia convention); the infra / XLA-lowering tables; dataset + honest-disclosure notes; intended-use + limitations; CC-BY-NC-4.0 (derivative of the NC tiny-aya-base). Curriculum onsets, the plateau, and the anneal descent are all visible across the suite — built for training-dynamics / mech-interp study, not just the final weights.

Next: end-task release evals (ASR-chrF++/BLEU/WER vs GT-audio topline, MOS Δ, BLASER-2.0, GEMBA, RTF) via scripts/eval_release.py on the annealed best/final + LAWA — numbers land in the card model-index and here.

🤖 Generated with Claude Code

@best

One-page execution checklist for the GPU release-eval session distilled from v0.3-evals-plan + evals-runbook: bringup, hub-only data staging, mandatory smoke, Tier-2 passes on @best(76,000)/@step-76250, sanity gates, LAWA, Tier-3 semantic venv, FLEURS acoustic-shift, paired bootstrap, landing steps. Includes the Exa-validated optional strengthenings (BLASER headline placement; multi-pass GEMBA). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ng groups The Lambda A100 release-eval box needs a CUDA torch build; the repo pins CPU torch (torch-xla lockstep) for the WSL2/TPU dev + TPU-host envs. Rather than ad-hoc `pip install --index-url cu128` (which `uv run`'s pre-sync silently reverts to +cpu), capture both environments in one uv.lock: - Move the TPU torch stack (torch, torch-xla[tpu], jax, jaxlib, torchax, flax) out of base `dependencies` into a `tpu` dependency-group, set as `[tool.uv].default-groups = ["tpu"]` so bare `uv sync`, dev installs, and the TPU host's `uv sync --frozen --extra eval` are UNCHANGED (CPU torch + xla). - Add an opt-in `cuda` group (torch==2.9.0, torchaudio==2.9.0) routed to the pytorch-cu128 index; declare it mutually exclusive with `tpu` via `[tool.uv].conflicts` so `--group cuda` auto-drops the default and excludes torch-xla/libtpu on the GPU box. - torchaudio leaves the `eval` extra (it must track the torch wheel variant) and lives in the `cuda` group at +cu128. GPU box: `uv sync --no-default-groups --group cuda --extra eval` (same version 2.9.0 as TPU, only the wheel variant differs -> identical model/metric behavior). Verified: `uv lock` resolves; dry-run plans torch/torchaudio 2.9.0+cu128 with torch-xla removed; default tpu plan still CPU+xla. Doc: eval-session-plan.md bringup + an `evalpy` wrapper so `uv run` never reverts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@best

…ite mirror Each published revision of tr-hi-s2st-v0.3 carries, at its top level, only that checkpoint's own weights-only bundle (~3.77 GB). But every branch ALSO mirrors the full `checkpoints/` file-tree-discoverability folder (all ~89 depth_decoders = ~51 GB) plus demo `samples/` wavs and `logs/` -- none of which the loader reads. `_stage_checkpoint`'s bare snapshot_download was pulling all 51 GB per checkpoint. Add ignore_patterns=[checkpoints/*, samples/*, logs/*] -> 3.77 GB. Also: the smoke must target @best (or @step-76250), NOT @step-1000. Verified via repo_info that the early per-1000 branches (published during the 50 GB private-cap era) are missing peft_adapter/adapter_model.safetensors at top-level (2.55 GB, not loadable); best / step-76000 / step-76250 are complete (3.77 GB, adapter present). eval-session-plan.md updated (smoke @best, evalpy). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Referee (audio-transcription cross-check, asr_judge.transcribe_gemini) and GEMBA adequacy judge (llm_judge.judge_adequacy) both default model_name to gemini-3.6-flash for the v0.3 release evals (was gemini-2.5-flash). Single model across all judge modes per the run owner's choice. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

vasista22/whisper-hindi-large-v2 ships generation_config.suppress_tokens = [] (empty). transformers' Whisper _prepare_decoder_input_ids does suppress_tokens[-2] -> IndexError "index -2 out of bounds for dim 0 size 0" the moment a tr->hi row invokes the Hindi judge (the smoke missed it: its 8 rows were all hi->tr, so only the Turkish large-v3 judge ran). Null out a degenerate/too-short suppress_tokens in WhisperJudge._load (None = suppress nothing, matching intent, hits the safe branch); large-v3's real list is left intact. Unblocks the asr stage of the 500-row Tier-2 pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

average_checkpoints.py writes metadata.json with save_kind=lawa_average + averaged_steps but no scalar `step`; ckpt_step did int(meta["step"]) -> KeyError when evaluating a LAWA average as a local --checkpoint. Fall back to max(averaged_steps) (the window end) so the report/labeling anchors at the latest constituent; 0 if neither present. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… step) average_checkpoints.py intentionally writes save_kind=lawa_average with averaged_steps and no bare int `step` (per its own docstring). load_checkpoint did step = meta["step"] -> KeyError when loading a LAWA average as an eval --checkpoint. Fall back to max(averaged_steps) when `step` is absent; real training checkpoints always carry `step` so the resume path is unchanged. Pairs with the ckpt_step fix (5a6813b). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

blaser_scores passed raw wav paths to SONAR's SpeechToEmbeddingModelPipeline; Mimi-decoded wavs are 24 kHz and can exceed unit amplitude, so SONAR's audio validator raised "np.ndarray values must be between -1 and 1" and BLASER was skipped for every checkpoint. Add _prep_wavs: peak-normalize (only when clipping) + resample to 16 kHz into temp files before encoding src+gen wavs. Unblocks the ASR-free speech-semantic metric (the most informative Tier-3 signal on unintelligible generated audio). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

gemini-3.6-flash is a thinking model: thinking tokens count against max_output_tokens, so the old cap of 4 was exhausted before any rating was emitted -> resp.text=None -> judge_adequacy returned mean=null for every sample. thinking_config={"thinking_budget":0} is rejected (400) for this model, so the fix is a generous cap (1024); it's a ceiling, not a target (generation stops after the single digit), so cost only rises when thinking runs long. Verified live: max_output_tokens=4 -> None, 1024 -> "1". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…failure blaser_scores computed QE + cosine (speech encoders only) then, inside the same try, ran ref-mode which needs the SONAR text encoder. A corrupt text_sonar_basic_encoder fairseq2 asset raised and discarded the whole BLASER block -> all None. Wrap ref-mode in its own try/except: on text-encoder failure, record blaser_ref_skipped and keep the QE + cosine scores (the headline ASR-free src-speech<->gen-speech signal). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@best

End-task GPU release-eval complete, framed as the answer to the blog's open question ("how much training before translation quality emerges, not just language identity?"). Capability emerges in order: language identity -> text translation (strong) -> audio synthesis (next frontier). Free-run text chrF++ climbs 14->25 over training while generated-audio ASR-chrF++ stays ~6; BLASER-QE 2.5 shows a real, weak speech-semantic signal ASR can't recover. Release checkpoint @best (76,000); @Final ties, LAWA rejected (paired bootstrap). - docs/v0.3-eval-report.md (new): full report — emergence curve, final @best table, checkpoint selection, FLEURS (acoustic shift), the 8 harness fixes, reproducibility + disclosures. - docs/v0.3-release-plan.md (new): the release plan/checklist. - docs/hf-model-card-...md: model-index populated + Evaluation section rewritten (data-efficiency framing, figures, status complete). - docs/v0.3-reval-report.md: Addendum A.7 pointer; docs/do.md: eval + blog closed. - docs/assets/v0.3/: 5 figures (emergence-curve hero + training-dynamics, per-codebook, eval-vs-topline, see-it-learn.gif), dataviz-styled. Prepare-for-review: NOT pushed. W&B eval/* backfill, hub eval/ push, model-card hub push, PR#10 comment, and the blog PR are the land-it pass on sign-off. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The v0.3 eval report lives on feat/v0.3-implementation (PR #10, not yet merged to main per team-review flow). Point the card's two eval-report links at the branch so they resolve now; they become blob/main/... on merge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

cataluna84 · 2026-07-22T20:09:18Z

v0.3 release evaluation — the data-efficiency answer

TL;DR — The GPU release-eval is complete and I'm framing it as the answer to
the blog's open question ("how much training on the full 840K dataset before
translation quality emerges, not just language identity?"). v0.3 is that
full-corpus run — 2.07 epochs / 76,250 steps / 6.59B tokens — and capability
emerges in a clear order: language identity → text translation (strong, ~25
chrF++) → audio synthesis (the remaining frontier). Release checkpoint =
@best (step 76,000). Eight eval-harness bugs found by running end-to-end
were fixed along the way (ea05a9a 2608dff 39bc2bf 5a6813b 71a76ee
b887fd9 82d030e fca3e23). Full report: docs/v0.3-eval-report.md. All local / prepare-for-review — nothing pushed
to the hub, W&B, or posted yet.

The finding

The text-pretrained-backbone bet pays off data-efficiently in the text
inner-monologue, and the honest gap is audio synthesis fidelity, not
translation understanding.

Teacher-forced emergence (W&B xzcb60bl):
text-token accuracy 25.6% → 96.6% (crosses 90% by step 27.5k), cb0 semantic
accuracy 10.4% → 41.5% then plateaus (frozen depth-decoder ceiling),
composite 6.72 → 2.82.

End-task @best (step 76,000, 500-row v03-val-500, greedy):

metric	hi→tr	tr→hi
free-run text chrF++ (inner-monologue)	25.7	25.1
generated-audio ASR-chrF++	3.7	9.6
GT-audio topline chrF++ (ceiling)	92.1	86.6
BLASER-2.0 QE (ASR-free, 1–5)	2.53	2.49
GEMBA adequacy (gemini-3.6-flash, 1–5)	1.08	1.10
DNSMOS Δ(gen − GT)	−1.34	−1.34
RTF / TTFA	0.95 / 76 ms

The BLASER-QE 2.5 > ASR-chrF++ ~4 / GEMBA 1.1 gap is the informative bit:
BLASER embeds source and generated speech directly (no ASR), and finds more
similarity than a transcript recovers — the audio is acoustically degraded,
not semantically empty. The model has learned what to say faster than how to
render it as clean speech; the bottleneck is the frozen Moshi depth decoder.

Checkpoint selection — `@best` ≈ `@final`; averaging rejected

Paired bootstrap (1000 resamples, per direction):

candidate	hi→tr ASR-chrF++	tr→hi ASR-chrF++	text chrF++	vs `@best`
`@best` (76,000) — released	3.7	9.6	25.7 / 25.1	—
`@final` (76,250)	3.8	9.8	25.9 / 24.8	NS (p = 0.26 / 0.19)
LAWA (66k–76k avg)	3.7	8.6	26.2 / 25.9	tr→hi worse (p = 0.006)

FLEURS — real-speech acoustic shift (winner only)

⚠️ Acoustic shift only (texts 200/200 seen, not held-out). On real human
recordings even the text stream collapses: text chrF++ 9.8 / 7.1, ASR-chrF++
1.2 / 9.3, topline 60.8 / 66.7 → v0.3 is distribution-bound to its
synthetic-TTS acoustics.

Eight harness fixes (all in the eval/reporting path; weights untouched)

commit	fix
`ea05a9a`	reproducible CUDA torch (cu128) via uv conflicting groups
`2608dff`	stage only the 3.77 GB checkpoint bundle, not the 51 GB suite mirror
`39bc2bf`	Hindi Whisper judge empty `suppress_tokens` IndexError (tr→hi rows)
`5a6813b`	`ckpt_step` tolerates LAWA metadata (no scalar step)
`71a76ee`	`load_checkpoint` tolerates LAWA metadata
`b887fd9`	normalize wavs (16 kHz, ≤1) for the BLASER SONAR encoders
`82d030e`	GEMBA `max_output_tokens` 4 → 1024 (gemini-3.x is a thinking model)
`fca3e23`	isolate BLASER ref-mode so QE+cosine survive a corrupt text encoder

Links & status

Report: docs/v0.3-eval-report.md
Model: https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3 · Run: https://wandb.ai/cataluna84/tinyaya-stage2-tpu/runs/xzcb60bl
W&B emergence report (text/cb0 + backfilled eval/* on the training-step axis): https://wandb.ai/cataluna84/tinyaya-stage2-tpu/reports/TinyAya-v0.3-Emergence-and-Data-Efficiency--VmlldzoxNzU1OTU1NQ==
Blog v0.3 revision: Cohere-Labs-Community/blog#14 (answering the post's own open question).
Prepare-for-review: these 8 commits + the new docs are local; W&B eval/*
backfill, hub eval/ push, model-card hub push, and the release promotion are
the land-it step on sign-off. (Note: not claiming a globally-green /verify —
a pre-existing lazy torch_xla import in src/model/scan_utils.py trips the
stricter seam gate; the module-level check_backend_seam.sh guard passes.)

🤖 Generated with Claude Code

Remove the 46-file configs/tpu/reval/ sweep-arm + hardening tree and 7 superseded standalone configs (v6e8_reval, v3/valfix smokes, the b256/b2048 batch-scale arms). Arm-D's winning recipe is already baked into the canonical stage2_tpu_v6e16_full_v03_mh.yaml; provenance is retained in git history + the eval report. Also drop now-dead deleted-doc pointers from two kept configs (full_v03, v6e_v2). Kept: the canonical run configs, the CI-test-pinned sweep configs, and the launch-default single-host config. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Remove 11 provenance / one-off-ops / orphaned docs (reval report + sweep/release/anneal plans, do.md, eval-session-plan, the CDN-unblock trail, sweep-coordinated runbook, tpu-capacity-log, tpu-trc-allocation which held verbatim internal TRC email content, and the orphaned 540 KB memory-system.png). Repair every resulting dangling link in kept docs (AGENTS/CONTRIBUTING/hf-card/public-release-plan/eval-report/evals-runbook/evals-plan/tpu-runbook) and kept scripts. Git history retains the trail. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Replace the prior author's hardcoded /home/alperiox/... paths and alperiox/* hub id with portable placeholders in the GPU repro template (configs/gpu/stage2_26k_parallel.yaml) and two kept scripts (gen_parallel.py, validate_full_pipeline.sh). No behaviour change; those paths were always environment-specific. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…aluated The root card still described v0.2-era reality (audio-only, text_weight 0, 'corpus has no alignments', weights + eval pending, effective batch 256). Rewrite to v0.3: text+audio (text_weight 0.2; corpus ships word-level alignments), run complete (best val 2.8199 @ 76,000, 2.07 epochs), release eval done (free-run text chrF++ ~25.7/25.1; ASR-chrF++ 3.7/9.6 vs GT-audio topline 92.1/86.6 -- audio synthesis is the frontier), arm-D exclude_top:0, populated model-index, fixed the typo'd Code URL, added the W&B report link. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The FLEURS subset embedded absolute /home/claudeuser/...phase-3-data-generation-pipeline/... paths in every row (val-500 already used relative encoded/... paths). Rewrite pt_path/src_align_path/tgt_align_path to the portable encoded/... form via the frozen subset's own write_subset()/rows_digest(), so the self-verifying digest stays consistent. The draw (samples/texts/overlap-audit) is unchanged; only the path encoding is portable now. Digest 5105afa6...->cd556805...; the two doc references (eval-report, evals-plan) follow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Remove launch_reval_arms.sh + reval_supervisor.sh + launch_reval_supervisor.sh -- they orchestrated the deleted configs/tpu/reval/ arm sweep and had no external referrers. This clears the last dangling reference to the pruned reval config tree. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Trim .claude/PROGRESS.md from 12,591 lines (500 KB) to ~110 (8 KB) for the public release: drop ~2,650 hook-auto-logged exec/edit/verify/session entries (2026-07-06 to 07-22), keeping the 5 curated summaries (release x2, hardening, decide x2 incl. the text+audio pivot). Full prior log remains in git history. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

cataluna84 · 2026-07-22T21:46:50Z

Public-release de-bloat — review requested

@alperiox — could you review this PR when you have a chance? On top of the v0.3
eval-release work (earlier comment),
I ran a cleanup pass so the repo isn't bloated when it goes public.

TL;DR — pruned 67 files (276 → 209) + trimmed the internal work-log, on a
strict rubric: keep everything needed to use and reproduce v0.3, drop
process/experimental clutter. No functional code changed; both CI gates + all
184 tests pass on the head (3089abd), CI green.

What was removed

Area	Removed	Why
Configs	the 46-file `configs/tpu/reval/` sweep-arm + hardening tree, and 7 superseded standalone configs (`v6e8_reval`, `v3`/`valfix` smokes, `b256`/`b2048` batch arms)	arm-D's winning recipe is already baked into the canonical `stage2_tpu_v6e16_full_v03_mh.yaml`; the trail lives in git history + the eval report
Docs	11 executed planning/ops docs (reval report + sweep/release/anneal plans, `do.md`, session/CDN/capacity/TRC-allocation notes, an orphaned diagram)	process paper-trail; the release-facing docs + git history retain it
Scripts	3 vestigial reval-orchestration scripts	they drove the deleted reval configs and had no other referrers
Work-log	`.claude/PROGRESS.md`: 12,591 → 111 lines	dropped ~2,650 hook-auto-logged `exec`/`edit`/`verify` entries; kept the 5 curated summaries

What was fixed (not removed)

Genericized prior-author paths/ids — replaced hardcoded /home/alperiox/…
paths + the alperiox/* hub id with portable placeholders in the GPU config,
gen_parallel.py, and validate_full_pipeline.sh. (Your attribution stays in
NOTICE.)
Corrected the root MODEL_CARD.md — it still described v0.2-era reality
(audio-only, text_weight 0, "eval pending"); now v0.3 text+audio, run
complete (best val 2.8199 @ 76,000), release eval done.
Made the FLEURS eval subset path-portable — rewrote the embedded absolute
paths to relative encoded/… (matching v03-val-500) via the subset's own
digest tooling; the self-verifying sha256 + the two doc references were
updated to match.
Repaired every cross-link the doc deletions would otherwise have dangled
(README/AGENTS/CONTRIBUTING/eval-report/runbooks + a few code comments).

Kept deliberately (coupling / reproducibility)

A few things I'd otherwise have pruned but kept because CI or the launch flow
depends on them: the two sweeps/* grids + v6e_v2/v6e_v3/v6e_proxy configs
(hard-read by test_sweep_promote.py), the single-host full_v03.yaml (the
default CONFIG_FILE for the launch scripts), the smoke/overfit configs (cited
by README/AGENTS), and all training/eval/inference code.

Verification

scripts/ci/check_backend_seam.sh + check_docs_sync.sh ✅
pytest tests/ → 184 passed, 1 skipped ✅
CI ci workflow green on 3089abd ✅
No secrets, no foreign private paths (only the NOTICE attribution), zero
dangling references.

… tooling Review of all .claude files for the public release: the archive (614 KB) held a personal email (x2) + 1,530 /home/cataluna84 paths + infra refs, and the live session-state files (PROGRESS/memories/PLAN/VERIFY) are internal working notes. Untrack them (kept locally via .gitignore) so only the reusable machinery ships -- hooks, skills, agents, commands, orchestration playbooks, settings. Also genericized the one /home path in the tpu-redeploy skill. Remaining tracked .claude is clean: no PII, no home paths, no dangling refs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The GCS log sanitizer hardcoded /home/cataluna84 as the path to redact, so it only scrubbed one user's home. Use os.path.expanduser('~') -> works for whoever runs it and drops the hardcoded username. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…B, blog, license) Bring every release-facing .md in line with the completed v0.3 release: flip status pending->complete across README + both model cards + the release plan + the eval runbooks; land the headline eval (free-run text chrF++ 25.7/25.1; ASR-chrF++ 3.7/9.6 vs a 92/87 GT-audio topline) and link docs/v0.3-eval-report.md; add the W&B emergence-report + blog links to README, MODEL_CARD, the HF card, the eval report, and the release plan; fix the stale README recipe (exclude_top 2->0 = arm-D winner; run 110,463 planned / 76,250 trained) and the 'audio-only' v0.3 narrative errors in the release plan (v0.3 is text+audio); fix the misspelled-org GitHub URL (simulatenous->simultaneous) in README + the v0.1/v0.2 cards and advance the v0.1 new_version pointer to v0.3. LICENSE: verified CohereLabs/tiny-aya-base is cc-by-nc-4.0 on HF, so the derivative weights inherit non-commercial terms. Reconciled the released-weights license to CC-BY-NC-4.0 everywhere (the apache-2.0 carried over stale in the root card would be legally invalid); Moshi/Mimi corrected to CC-BY-4.0 (verified). Code stays Apache-2.0. docs-sync + backend-seam gates pass; 184 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

cataluna84 · 2026-07-25T02:00:20Z

@alperiox — This PR is ready to be reviewed and merged to main, since the blog has been published

https://labscommunity.cohere.com/blog/2026/adapting-moshi-low-resource-speech-translation/

cataluna84 and others added 10 commits June 28, 2026 21:09

cataluna84 and others added 2 commits June 29, 2026 11:04

cataluna84 and others added 2 commits June 29, 2026 14:06

cataluna84 and others added 5 commits June 29, 2026 17:13

cataluna84 and others added 5 commits June 30, 2026 23:37

cataluna84 and others added 4 commits July 20, 2026 19:11

docs(runbook): W&B crashed-badge gotcha on resumed shared-mode runs

d62af33

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

cataluna84 and others added 12 commits July 22, 2026 11:16

cataluna84 and others added 7 commits July 23, 2026 02:43

cataluna84 and others added 3 commits July 23, 2026 03:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

v0.3 training implementation (Phases A–E): regularize, weight-averaging, deep-codebook, multitask#10

v0.3 training implementation (Phases A–E): regularize, weight-averaging, deep-codebook, multitask#10
cataluna84 wants to merge 185 commits into
mainfrom
feat/v0.3-implementation

cataluna84 commented Jun 28, 2026 •

edited

Loading

Uh oh!

cataluna84 commented Jun 29, 2026

Uh oh!

cataluna84 commented Jun 29, 2026

Uh oh!

cataluna84 commented Jun 29, 2026

Uh oh!

cataluna84 commented Jun 29, 2026

Uh oh!

cataluna84 commented Jun 30, 2026

Uh oh!

cataluna84 commented Jul 22, 2026

Uh oh!

cataluna84 commented Jul 22, 2026 •

edited

Loading

Uh oh!

cataluna84 commented Jul 22, 2026 •

edited

Loading

Uh oh!

cataluna84 commented Jul 25, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

cataluna84 commented Jun 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⏱️ Revised execution order (owner)

A lot already exists — we build, not rebuild

Implementation plan (Phases A–E)

Run / experiment sequence (owner-revised)

Open decisions (owner)

Uh oh!

cataluna84 commented Jun 29, 2026

✅ Phase D — Multitask balance (commit 91d4bf2)

Uh oh!

cataluna84 commented Jun 29, 2026

🔬 Smoke: Phase A + C + D all fire on v6e-8 (commit 13b0a2f)

Uh oh!

cataluna84 commented Jun 29, 2026

✅ Phase E — Sweep redesign + SpecAugment (commits 4c58892, af2eb6e)

Uh oh!

cataluna84 commented Jun 29, 2026

✅ 8× v6e-8 parallel sweep fleet (commit bde08a9)

Uh oh!

cataluna84 commented Jun 30, 2026

v0.3-scale: capacity sweep + long-horizon hardening (plan + status)

🖥️ TPU spun up

🔬 Part A — the capacity sweep (2-stage W&B, on the v6e-16)

🧱 Part B — long-horizon hardening (gate before the full production run)

⚙️ Execution model — detached & workstation-independent

📦 This commit (ec5ba86)

⏭️ Next

Uh oh!

cataluna84 commented Jul 22, 2026

v0.3 training complete — plateau + WSD anneal (final numbers)

Uh oh!

cataluna84 commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

v0.3 release evaluation — the data-efficiency answer

The finding

Checkpoint selection — @best ≈ @final; averaging rejected

FLEURS — real-speech acoustic shift (winner only)

Eight harness fixes (all in the eval/reporting path; weights untouched)

Links & status

Uh oh!

cataluna84 commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Public-release de-bloat — review requested

What was removed

What was fixed (not removed)

Kept deliberately (coupling / reproducibility)

Verification

Uh oh!

cataluna84 commented Jul 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cataluna84 commented Jun 28, 2026 •

edited

Loading

✅ Phase D — Multitask balance (commit `91d4bf2`)

🔬 Smoke: Phase A + C + D all fire on v6e-8 (commit `13b0a2f`)

✅ Phase E — Sweep redesign + SpecAugment (commits `4c58892`, `af2eb6e`)

✅ 8× v6e-8 parallel sweep fleet (commit `bde08a9`)

📦 This commit (`ec5ba86`)

cataluna84 commented Jul 22, 2026 •

edited

Loading

Checkpoint selection — `@best` ≈ `@final`; averaging rejected

cataluna84 commented Jul 22, 2026 •

edited

Loading

cataluna84 commented Jul 25, 2026 •

edited

Loading