Semantic-Aware, Physics-Informed, Geometry-Grounded
Weather Video Synthesis

Chenghao Qian¹  ·  Nedko Savov²  ·  Lingdong Kong³  ·  Yeying Jin³  ·  Rui Song⁵
Wenjing Li^1,4  ·  Zhun Zhong⁴  ·  Jiaqi Ma⁵  ·  Gustav Markkula¹  ·  Luc Van Gool²

_{¹University of Leeds  ·  ²INSAIT, Sofia University “St. Kliment Ohridski”  ·  ³National University of Singapore  ·  ⁴Hefei University of Technology  ·  ⁵UCLA}

TL;DR — We steer an off-the-shelf video diffusion editor with three structured priors — semantics (what the weather looks like), dynamics (how it evolves), and geometry (where it appears) — to synthesize diverse, physically realistic weather on real videos, without any finetuning.

---

📰 News

• Jun 2026  🎉 Paper accepted to ECCV 2026!
• Jun 2026  🌐 Project page is live, with the supplementary demo video.
• Soon  💻 Code & pretrained models — stay tuned.

---

✨ Highlights

• 🧩 Tri-prior interface — a single, structured conditioning space that factorizes weather into semantics · dynamics · geometry, giving precise and interpretable control.
• 🌦️ Diverse appearance — a semantic-aware strategy binds the intended weather to scene semantics via a VLM + LLM, producing varied, realistic global appearances.
• ❄️ Physical particle dynamics — a physics-informed Gaussian particle field evolves under gravity, wind, and turbulence, activating latent weather priors in pretrained editors for dense, coherent particles.
• 📐 Geometry grounding — particles are gravity-aligned and projected with camera intrinsics/extrinsics into particle-augmented depth, ensuring spatially accurate, temporally consistent placement.

---

🎬 Results

_{Diverse weather synthesized across varied real-world scenes.  ▶️ Full demo video on the project page}

---

🧠 Method

From an input video, three modules build structured conditioning — semantic-aware appearance anchoring (VLM/LLM reasoning → appearance anchor), physics-informed dynamic simulation (a Gaussian particle field under gravity, wind, and turbulence), and geometry-grounded video synthesis (geometry assets, alignment, and particle projection). The resulting semantics, dynamics, and geometry signals jointly steer a frozen video diffusion model.

---

📚 Citation

If you find our work useful, please consider citing:

@inproceedings{qian2026weathervid,
  title     = {Semantic-Aware, Physics-Informed, Geometry-Grounded Weather Video Synthesis},
  author    = {Qian, Chenghao and Savov, Nedko and Kong, Lingdong and Jin, Yeying and
               Song, Rui and Li, Wenjing and Zhong, Zhun and Ma, Jiaqi and
               Markkula, Gustav and Van Gool, Luc},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}

_{🌐 Project Page  ·  📄 arXiv  ·  ECCV 2026}