forked from hehao13/projects-CameraCtrl
-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathindex.html
More file actions
170 lines (160 loc) · 11.8 KB
/
Copy pathindex.html
File metadata and controls
170 lines (160 loc) · 11.8 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>FinePhys</title>
<link href="./finephys/style.css" rel="stylesheet">
<script type="text/javascript" src="./CameraCtrl_files/jquery.mlens-1.0.min.js"></script>
<script type="text/javascript" src="./CameraCtrl_files/jquery.js"></script>
<script>
MathJax = {
tex: {
inlineMath: [['$', '$'], ['\\(', '\\)']]
}
};
</script>
<script id="MathJax-script" async src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
</head>
<body>
<div class="content">
<!-- <h1>FinePhys: Fine-grained Human Action Generation by Explicitly Incorporating Physical Laws for Effective Skeletal Guidance</h1> -->
<h1><b>FinePhys: Fine-grained Human Action Generation by Explicitly Incorporating Physical Laws for Effective Skeletal Guidance</b></h1>
<p id="authors" class="serif">
<span style="font-size: 1.0em">
Dian Shao<sup>†</sup>, Mingfei Shi, Shengda Xu, Haodong Chen, Yongle Huang, Binglu Wang
</span>
<br>
<a style="font-size: 1em"><sup>†</sup>Corresponding Author.</a>
<br>
<span style="font-size: 0.9em; margin-top: 0.6em">
Northwestern Polytechnical University
</span>
</p>
<font size="+1">
<p style="text-align: center;" class="sansserif">
<!-- <a href="https://arxiv.org/abs/2404.02101" target="" style="font-weight: bold;">[arXiv Report]</a> -->
<a href="https://github.com/SmartDianLab/FinePhys" target="" style="font-weight: bold;">[Code]</a>
<!-- <a href="#bibtex" style="font-weight: bold;">[BibTeX]</a> -->
<!-- <a href="https://huggingface.co/spaces/hehao13/CameraCtrl-svd" style="font-weight: bold;">[HF Demo]</a> -->
</p><br>
</font>
<!-- <div style="text-align:center;">
<img src="./finephys/teaser.png" width="100%" alt="teaser_figure">
</div> -->
<!-- <div style="text-align:center; margin-bottom:1em;">
<video class="clickplay" width="80%" controls autoplay muted loop>
<source src="./finephys/Demo_FinePhys.mp4" type="video/mp4">
</video>
</div> -->
<div style="text-align:center; margin-bottom:1em;">
<img src="./finephys/demo_video.gif" width="100%" alt="Demo GIF">
</div>
</div>
<!-- <div class="content">
<p style="text-align:center; font-size: 2em; font-weight: bold" class="sansserif">Abstract</p>
<p style="font-size: 1.2em; margin-left:5em; margin-right:5em;" class="serif"> Although remarkable progress has been achieved in video generation, synthesizing physically plausible human actions remains an unresolved challenge, especially when addressing fine-grained semantics and complex temporal dynamics. For instance, generating gymnastics routines such as \textit{“two turns on one leg with the free leg optionally below horizontal”} poses substantial difficulties for current video generation methods, which often fail to produce satisfactory results. To address this, we propose \textbf{FinePhys}, a \textbf{Fine}-grained human action generation framework incorporating \textbf{Phys}ics for effective skeletal guidance. Specifically, FinePhys first performs online 2D pose estimation and then accomplishes dimension lifting through in-context learning. Recognizing that such data-driven 3D pose estimations may lack stability and interpretability, we incorporate a physics-based module that re-estimates motion dynamics using Euler-Lagrange equations, calculating joint accelerations bidirectionally across the temporal dimension. The physically predicted 3D poses are then fused with data-driven poses to provide multi-scale 2D heatmap-based guidance for the video generation process. Evaluated on three fine-grained action subsets from FineGym (FX-JUMP, FX-TURN, and FX-SALTO), FinePhys significantly outperforms competitive baselines. Comprehensive qualitative results further demonstrate FinePhys's ability to generate more natural and plausible fine-grained human actions.
</div> -->
<div class="content">
<p style="text-align:center; font-size: 2em; font-weight: bold" class="sansserif">Abstract</p>
<p style="font-size: 1.2em; margin-left:5em; margin-right:5em;" class="serif">
Although remarkable progress has been achieved in video generation, synthesizing physically plausible human actions remains an unresolved challenge, especially when addressing fine-grained semantics and complex temporal dynamics. For instance, generating gymnastics routines such as <em>"two turns on one leg with the free leg optionally below horizontal"</em> poses substantial difficulties for current video generation methods, which often fail to produce satisfactory results. To address this, we propose <em><strong>FinePhys</strong></em>,
a <em><strong>Fine</strong></em>-grained human action generation framework incorporating <em><strong>Phys</strong></em>ics for effective skeletal guidance. Specifically, FinePhys first performs online 2D pose estimation and then accomplishes dimension lifting through in-context learning. Recognizing that such data-driven 3D pose estimations may lack stability and interpretability, we incorporate a physics-based module that re-estimates motion dynamics using Euler-Lagrange equations, calculating joint accelerations bidirectionally across the temporal dimension. The physically predicted 3D poses are then fused with data-driven poses to provide multi-scale 2D heatmap-based guidance for the video generation process. Evaluated on three fine-grained action subsets from FineGym (FX-JUMP, FX-TURN, and FX-SALTO), FinePhys significantly outperforms competitive baselines. Comprehensive qualitative results further demonstrate FinePhys's ability to generate more natural and plausible fine-grained human actions.
</p>
</div>
<div class="content">
<p style="text-align:center; font-size: 2em; font-weight: bold" class="sansserif">Demo Video</p>
<div style="text-align:center; margin-bottom:1em;">
<video class="clickplay" width="80%" controls>
<source src="./finephys/FinePhys_final_demo.mp4" type="video/mp4">
</video>
</div>
</div>
<!-- <div class="content">
<p style="text-align:center; font-size: 2em; font-weight: bold" class="sansserif">Framework</p> <br>
<img src="./finephys/architecture.png" style="width:90%;" alt="architecture_figure" class="summary-img"> <br>
<p style="font-size: 1.2em; margin-left:5em; margin-right:5em;" class="serif">
<strong>Overview of Finephys.</strong>
FinePhys addresses the challenging task of generating fine-grained human action videos by explicitly incorporating physical equations exploiting pose modality.
The pipeline begins with online extracting 2D poses, then transforms them into 3D using an in-context learning module, achieving the data-driven 3D skeleton sequence $S^{3D}_{dd}$.
To incorporate the physical laws of motion, we introduce a Phys-Net module to re-estimate the 3D positions of each human joint by accounting for second-order temporal variations (<em>i.e.</em>, accelerations) in both forward and reverse directions, yielding physically predicted 3D poses$S^{3D}_{pp}$.
Subsequently, $S^{3D}_{dd}$ and $S^{3D}_{pp}$ are fused, projected back into 2D space, encoded into multi-scale latent maps, and integrated into 3D-UNets to guide the denoising process.
</p>
</div> -->
<div class="content">
<p style="text-align:center; font-size: 2em; font-weight: bold" class="sansserif">Framework</p> <br>
<img src="./finephys/architecture.png" style="width:90%;" alt="architecture_figure" class="summary-img"> <br>
<p style="font-size: 1.2em; margin-left:5em; margin-right:5em;" class="serif">
<strong>Overview of FinePhys.</strong>
FinePhys addresses the challenging task of generating fine-grained human action videos by explicitly incorporating physical equations exploiting pose modality.
The pipeline begins with online extracting 2D poses, then transforms them into 3D using an in-context learning module, achieving the data-driven 3D skeleton sequence <span class="math inline">$S^{3D}_{dd}$</span>.
To incorporate the physical laws of motion, we introduce a Phys-Net module to re-estimate the 3D positions of each human joint by accounting for second-order temporal variations (<em>i.e.</em>, accelerations) in both forward and reverse directions, yielding physically predicted 3D poses <span class="math inline">$S^{3D}_{pp}$</span>.
Subsequently, <span class="math inline">$S^{3D}_{dd}$</span> and <span class="math inline">$S^{3D}_{pp}$</span> are fused, projected back into 2D space, encoded into multi-scale latent maps, and integrated into 3D-UNets to guide the denoising process.
</p>
</div>
<!-- <div class="content">
<p style="text-align:center; font-size: 2em; font-weight: bold" class="sansserif">Visualization Results</p> <br>
<p style="font-size: 1.3em" class="serif"> <code>CameraCtrl</code> for general text-to-video generation</p> <br>
<div style="text-align:center; margin-bottom:1em;">
<video class="clickplay" width="90%" controls>
<source src="./CameraCtrl_files/generat_t2v_object.mp4" type="video/mp4">
</video>
</div>
<div style="text-align:center; margin-bottom:1em;">
<video class="clickplay" width="90%" controls>
<source src="./CameraCtrl_files/general_t2v_scene.mp4" type="video/mp4">
</video>
</div> <br>
<p style="font-size: 1.3em" class="serif"> Same text prompt + Different camera trajectories</p>
<div style="text-align:center; margin-bottom:1em;">
<video class="clickplay" width="90%" controls>
<source src="./CameraCtrl_files/different_traj_same_prompt.mp4" type="video/mp4">
</video>
</div> <br>
<p style="font-size: 1.3em" class="serif"> <code>CameraCtrl</code> for personalized text-to-video generation</p> <br>
<div style="text-align:center; margin-bottom:1em;">
<video class="clickplay" width="90%" controls>
<source src="./CameraCtrl_files/realistic_vision.mp4" type="video/mp4">
</video>
</div>
<div style="text-align:center; margin-bottom:1em;">
<video class="clickplay" width="90%" controls>
<source src="./CameraCtrl_files/toonyou.mp4" type="video/mp4">
</video>
</div> <br>
<p style="font-size: 1.3em" class="serif"> <code>CameraCtrl</code> for image-to-video generation</p>
<div style="text-align:center; margin-bottom:1em;">
<video class="clickplay" width="90%" controls>
<source src="./CameraCtrl_files/i2v_object.mp4" type="video/mp4">
</video>
</div>
<div style="text-align:center; margin-bottom:1em;">
<video class="clickplay" width="90%" controls>
<source src="./CameraCtrl_files/i2v_scene.mp4" type="video/mp4">
</video>
</div> <br>
<p style="font-size: 1.3em" class="serif"> Integration <code>CameraCtrl</code> with other video control methods</p> <br>
<div style="text-align:center; margin-bottom:1em;">
<video class="clickplay" width="96%" controls>
<source src="./CameraCtrl_files/integrate_with_others.mp4" type="video/mp4">
</video>
</div> <br>
</div>
<div class="content" id="bibtex">
<p style="text-align:left; font-size: 2em; font-weight: bold" class="serif">BibTeX</p>
<code>
@misc{he2024cameractrl,<br>
title={CameraCtrl: Enabling Camera Control for Text-to-Video Generation},<br>
author={Hao He and Yinghao Xu and Yuwei Guo and Gordon Wetzstein and Bo Dai and Hongsheng Li and Ceyuan Yang},<br>
year={2024},<br>
eprint={2404.02101},<br>
archivePrefix={arXiv},<br>
primaryClass={cs.CV}<br>
}
</code>
</div> -->
<div class="content">
<p class="serif">
We borrow the source code of this project page from <a href="https://hehao13.github.io/projects-CameraCtrl/">CameraCtrl</a>.
</p>
</div>
</body>
</html>