Add data-dependent noise and multiscale training scripts#1
Open
yifanc96 wants to merge 20 commits into
Open
Conversation
New training scripts for data-dependent noise (Allen-Cahn, Gaussian fields, Navier-Stokes) and multiscale interpolation/wavelet approaches for Gaussian fields. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ion tests The Sampler in both mask and wavelet trainers had a critical bug: _make_tgrid concatenated per-phase time grids into one continuous array, so RK4 steps near phase boundaries used k1,k2,k3 from phase k but k4 from phase k+1 — mixing two completely different velocity fields. This produced spurious high-k energy in generated samples. Fix: integrate each phase independently. Each phase has its own time grid strictly inside (k+t_min, k+1-t_min), and the final state of phase k feeds phase k+1 as initial condition. Uniform and cosine grid modes are equivalent for well-conditioned flows (verified by oracle). Changes: - train_multiscale_interpolation.py: rewrite Sampler with per-phase integration - train_wavelet_multiscale_v2.py: new wavelet trainer (replaces v1) with proper Haar DWT, per-sub-band noise, multi-net support, and fixed sampler - Remove train_wavelet_multiscale.py (superseded by v2) - Add test scripts for screening effect verification (test_screening.py, test_mask_correct.py), analytical oracle tests (test_oracle_mask.py, test_oracle_wavelet.py), and spectrum comparison (test_spectrum_oracle_vs_net.py, test_compare_trained.py) Key findings from oracle tests at G=64 s=3: - Screening effect gives bounded conditional cond ~700 per band (mask K=6) - With consistent noise variance, oracle matches truth E(k) to ~2% (k≤30) - Mask K=6 multi-net trained model: 500x better spectrum than 1-mask FM - 12 RK4 steps sufficient (fully converged, step-independent) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…l + forecasting) Gaussian fields: - Per-band independent training with scale-appropriate UNets - Raw variant with loss/sigma^2 reweighting (no centering) - Mask K=3 and Haar L=3 both achieve all bins<1 with 4 RK4 steps/band - Mask: mean_rel<=30=0.21, Haar: 0.30 Navier-Stokes unconditional (128x128): - 1-channel mask (v3) works as well as 2-channel (v1) - t-dependent normalization does not help (v2 slightly worse than v3) - Multiscale K=3 beats standard 1-mask at all NFE budgets - At 16 NFE: 4x better than standard Navier-Stokes conditional forecasting (lag=2): - Predicts innovation r=x_hi-x_lo, Gaussian base at sigma_innov^2 - Multiscale wins at low NFE (0.23 vs 0.92 at 4 total RK4) - Plateaus at 0.21 due to coarse bands subsampling x_lo conditioning 256x256 experiments: Haar works, mask finest band OOM (future fix needed)
…bin eval
Adds:
- multiscale_paper.tex: clean paper-style writeup focused on spatial
multiscale factorisation; theory split into well-conditioned Schur
complement (load-bearing for numerical efficiency) vs screening effect
(parameterisation efficiency under simple-network constraints); appendix
on RG flows (Carosso/Wegner-Morris/Polchinski/Bauerschmidt).
- Gaussian-fields/train_multiscale_perband_v2.py: training script with
EMA, logit-normal time sampling, gradient clipping, LR warmup,
per-band channel widths.
- Gaussian-fields/eval_existing_mask.py: per-bin spectrum diagnostic
showing low/mid frequencies (k<=10) are at 5% rel error and the
headline 0.21 is dominated by k=15..30 mid-band degradation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds:
- multiscale_experiments.tex: working experimental log documenting
training settings (architecture, batch, lr, steps, noise, loss),
best results with error bars across n_seeds=5 inference seeds,
spectrum and per-bin error figures, and honest discussion of which
rows are training-budget-limited vs method-limited.
- Gaussian-fields/eval_gf_unified.py: unified inference eval producing
seed-averaged spectrum and rel-error arrays for all GF methods.
- Gaussian-fields/plot_gf_results.py: produces the 3 figures.
- figs/{gf_spectrum_compare,gf_relerror_perbin,gf_nfe_sweep}.png
- Gaussian-fields/eval_*.py: per-method diagnostic eval scripts.
GF G=64 status: trained results filled in with error bars from 5 seeds.
NS G=128 status: TBDs throughout pending eval_ns_unified.py.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… launched
Adds:
- Navier-Stokes/eval_ns_unified.py: unified inference eval for NS G=128.
Handles v1 (2-channel) multiscale FM, perband meanflow, perband shortcut.
- Navier-Stokes/plot_ns_results.py: NS spectrum, per-bin error, NFE sweep.
- figs/ns_{spectrum_compare,relerror_perbin,nfe_sweep}.png
Updates multiscale_experiments.tex:
- NS G=128 table populated with seed-averaged numbers (n_seeds=3, 400 samples).
- NS figures referenced; captions reflect the surprising NS finding
that multiscale FM (~0.06) beats multiscale meanflow (~0.23) and
shortcut (~0.15) at the available training budget — opposite of GF.
- GF figure captions sharpened to discuss Nyquist behaviour: vanilla FM
spikes at k=31 (mask edge), flow-map methods bypass the artifact.
- Open-issues section updated: GF single-scale meanflow re-trained
at 250k steps (gauss + data-dep) currently running.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…plained
NS results extended:
- Added single-scale FM (gauss base + standard 1-mask) rows to NS table.
Single-scale FM at NFE>=32 plateaus near 0.115, multiscale FM at 0.063 -
multiscale wins by ~1.8x at large NFE and ~4x at small NFE.
- Added finer NFE grid (16, 24, 32, 48, 64, 96, 128) for multiscale FM.
Confirmed: NFE=16 and NFE=24 give identical results because both round
to 1 RK4 step/band; NFE=32 (2 steps/band) has a known RK4 pathology
(midpoint at t=0.5 coincides with drift over-shoot region in this
checkpoint); NFE>=48 recovers monotonically.
- Tex notes the under-training hypothesis for NS multiscale meanflow /
shortcut (plateaus across NFE are characteristic of under-fitting).
Single-scale data-dep noise eval intentionally not done (flagged out of scope).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… rows
Adds:
- Gaussian-fields/train_gaussian_field_shortcut.py: single-scale (1-mask)
shortcut trainer for Gaussian field. Same MeanFlowVelocity architecture
as the meanflow trainer, but with bootstrap-based shortcut self-consistency
loss instead of JVP. Recipe identical to per-band shortcut for clean
single-scale-vs-multiscale comparison.
Tex updates:
- Collapse NS single-scale FM rows: ns_gauss_base and ns_standard_1mask_G128
are the same method (vanilla 1-mask FM, z0~N(0,I)), reported as a single
row using the better (standard) checkpoint. Footnote explains.
Currently training (250k steps each):
- gf_meanflow_gauss_long (GPU 1)
- gf_shortcut_gauss_long (GPU 2)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds:
- Navier-Stokes/train_ns_shortcut_gauss_base.py: single-scale (1-mask)
shortcut trainer for NS at G=128. Same MeanFlowVelocity architecture
as ns_meanflow_gauss_base, with bootstrap shortcut self-consistency loss.
Currently training (6 concurrent jobs):
GF: gf_meanflow_gauss_long (250k, GPU 1), gf_shortcut_gauss_long (250k, GPU 2)
NS: ns_perband_mf_K3 long (500k total, GPU 0)
ns_perband_shortcut_K3 long (500k total, GPU 3)
ns_meanflow_gauss_base_long (500k, GPU 4)
ns_shortcut_gauss_base_long (500k, GPU 7)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New training scripts for data-dependent noise (Allen-Cahn, Gaussian fields,
Navier-Stokes) and multiscale interpolation/wavelet approaches for Gaussian fields.
Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com