4D video is here, and it runs on your own computer — Transcript

Full transcript

1. 0:00You can now create 40 video of yourself
2. 0:02locally for free. Only a few months ago,
3. 0:04you needed expensive studios with dozens
4. 0:06of cameras [music] filming you at the
5. 0:07exact same time. Now, all you need is
6. 0:10one phone. I spent the last few weeks
7. 0:12building a pipeline around some amazing
8. 0:14new research that lets you do this all
9. 0:15on consumer hardware. And to show you
10. 0:17what's possible and how it works,
11. 0:19[music] I combine it with our 3D world
12. 0:21generator from our last video. So, you
13. 0:23can put yourself in pretty much any
14. 0:24world you can imagine. But to properly
15. 0:26put it to the test, we need to try this
16. 0:28with the shot that this technique was
17. 0:30basically made for, [music] the Matrix
18. 0:31bullet time effect. Now, the research
19. 0:33paper explicitly states that weird poses
20. 0:35and loose clothes are basically the
21. 0:37breaking point for this technique, but
22. 0:38let's see how far we can push it.
23. 0:42Powering all of this is a research paper
24. 0:44called 4D Anyone. Normally, to capture
25. 0:46someone in 4D, you would need dozens of
26. 0:48cameras all pointing at you, filming you
27. 0:50at the exact same time. 40 anyone skips
28. 0:52this and instead of buying dozens of
29. 0:54cameras and sacrificing an entire room
30. 0:56to build a rig for them, we simply use
31. 0:58AI to generate these new views. Now,
32. 1:00plenty of AI models can already generate
33. 1:02new angles, but the reconstruction for
34. 1:04each frame only works if every angle
35. 1:07shows the exact same person as pixel
36. 1:09perfect as possible. And most models
37. 1:11cannot do that, especially not for like
38. 1:1348 different views. To fix this, 40
39. 1:16anyone first extracts a 3D skeleton of
40. 1:18the person. The skeleton locks the pose
41. 1:20for every angle, and the fine-tuned
42. 1:22video model fills the closed face and
43. 1:24the details from your original footage.
44. 1:27The trick is that the model keeps all
45. 1:28the angles it has already generated on
46. 1:30hand, so each new one is created with
47. 1:33the previous views inside and also it
48. 1:35keeps reshuffling which angles get
49. 1:37generated together. Early on, it mixes
50. 1:39views from all around the body so they
51. 1:41settle on the same overall shape. Then
52. 1:44later it groups neighboring angles
53. 1:46together so the transitions between them
54. 1:48stay smooth. And once we have all these
55. 1:50consistent views, we now need to turn
56. 1:52them into an actual 3D representation of
57. 1:55myself here that we can move around. And
58. 1:57that's where gshian splatting comes in.
59. 1:58Gosh splatting represents a 3D scene
60. 2:01using millions of tiny semi-transparent
61. 2:03blobs called gshions. Each one has a
62. 2:05position in 3D space, a size, a
63. 2:07rotation, a color, and transparency. And
64. 2:10to train a splat, you start with images
65. 2:13of the scene from different angles and
66. 2:15their camera positions. The training
67. 2:17process then adjusts the gshions,
68. 2:20moving, resizing, rotating, adding, and
69. 2:22removing them until rendering the scene
70. 2:25from those camera angles closely matches
71. 2:27the original images. To use this
72. 2:30workflow, you first need a video of
73. 2:31yourself. And ideally, it should be
74. 2:33filmed on a tripod. You can also decide
75. 2:35whether you want to film yourself full
76. 2:37body or closeup. Of course, if you film
77. 2:39closer, your legs and stuff will be
78. 2:41missing, but you'll also get more detail
79. 2:42in the face, for example. I ported 4D
80. 2:45anyone to Comi, a free node-based
81. 2:47interface for AI models, so it's a bit
82. 2:49easier to set up. If you don't have Comi
83. 2:51yet, we have a free guide showing you
84. 2:53how to set it up. For porting this to
85. 2:55Comi, I ran into two challenges. The
86. 2:57first problem was that the original
87. 2:58pipeline relied on code for
88. 3:00reconstructing the skeletons that isn't
89. 3:02allowed to be used for commercial
90. 3:03production, but I was able to swap it
91. 3:05out for SAM 3D body by Meta, which gives
92. 3:08us the same kind of data, but with a
93. 3:11more permissive license. And a second
94. 3:13problem was that G-splat, the renderer
95. 3:15that actually trains the gshion splats.
96. 3:17Normally, you have to compile it
97. 3:18yourself, but I wanted to keep it
98. 3:20simple. Drag and drop the workflow JSON
99. 3:22file into the CompuI interface. Then you
100. 3:25need to install the missing custom nodes
101. 3:28and it's only these two notepads right
102. 3:30here. Next, you need to copy these two
103. 3:32files into your Comfy UI folder and run
104. 3:35this file. This will set up Gsplit for
105. 3:38you. You basically just throw in your
106. 3:41video here, and this should be a
107. 3:42vertical video, either like a close-up
108. 3:44shot or a full body view. And then you
109. 3:46need to download these models and put
110. 3:48them in the correct folder. In the next
111. 3:49group, the synthetic data set will be
112. 3:51generated. And you don't have to set a
113. 3:53lot here. You can select the camera
114. 3:55preset and for highest quality I
115. 3:57recommend this one. For quick testing
116. 3:59you can also do like 16 two rings but
117. 4:02quality will be not as good of course.
118. 4:04The next important options are Turbo.
119. 4:06You should turn this on. This one will
120. 4:08not really cost you any quality and it
121. 4:10will be a lot faster. Another way to
122. 4:11speed this up is to install Sage
123. 4:13intension for comi. In this case you can
124. 4:16activate Sage. And then we also have
125. 4:17this low VRM setting right here. So if
126. 4:20you have anything less than 24 GB um
127. 4:23definitely make sure to uh turn this on.
128. 4:25In the next group here we have some
129. 4:27settings for gshion splatting. So first
130. 4:28of all we can select how many frames we
131. 4:31want to train. So you could lower this
132. 4:33number right here for a quick preview.
133. 4:35Down here you can switch the quality and
134. 4:38I usually use standard. Like all the
135. 4:40clips that you've seen are standard and
136. 4:42once it's done you have this cool
137. 4:44preview where you can check out your
138. 4:45results. But like talking characters in
139. 4:473D, I mean it's cool, but can we push it
140. 4:50even further? So now let's try to create
141. 4:52the Matrix bullet time effect using this
142. 4:53tool. So in the original Matrix movie,
143. 4:55they had this entire studio with a line
144. 4:57of around 120 cameras all firing in
145. 5:00rapid succession to create this amazing
146. 5:02camera move that also appears to happen
147. 5:04in slow motion. I really tried my best
148. 5:06to recreate this in my office. I
149. 5:08recorded multiple angles of myself
150. 5:10repeatedly jumping onto a mattress to
151. 5:12see which one would resolve best. And I
152. 5:14used the iPhone's slow motion setting
153. 5:16for this. So to my neighbors wondering
154. 5:18why their ceiling was shaking, that was
155. 5:19what I was doing for half an hour. In
156. 5:21the end, I ran all the clips through the
157. 5:23pipeline with varying degrees of
158. 5:25success. Most of them completely break
159. 5:27once I hit the mattress. This one was
160. 5:28the first one where I was thinking this
161. 5:30is sort of okay. At least you can still
162. 5:32understand the movement, but yeah, it's
163. 5:34not great. So then I started
164. 5:36experimenting whether I could use Miniax
165. 5:38H3, a free open-source video model to
166. 5:41clean up those broken renderings. And
167. 5:43that one worked pretty much first try.
168. 5:45So now pretty confident that I could at
169. 5:47least get something viewable uh out of
170. 5:49this workflow, I wanted to create the 3D
171. 5:52environment to put myself into. Of
172. 5:53course, you could scan a real
173. 5:55environment, but I relied on the
174. 5:57workflow from our latest video. There we
175. 5:59developed a pipeline that lets you take
176. 6:01an image or prompt and turn it into a
177. 6:03full 3D gshian splat environment using
178. 6:06virtual AI drones that fly through your
179. 6:08scene to map everything out. You should
180. 6:10check it out. It's a lot of fun. So, I
181. 6:12tried importing the environment and the
182. 6:14splat sequence into Blender using an
183. 6:17add-on called 3DGS render engine by
184. 6:20Kiri, but unfortunately it doesn't
185. 6:23support splat sequences yet. I tried
186. 6:26creating a script that loads each frame
187. 6:27individually, but it was a bit too heavy
188. 6:29for my computer and a bit messy. So,
189. 6:31instead, I created the camera move in
190. 6:33Blender around the static splat and then
191. 6:35asked Claude code to directly merge the
192. 6:37splats in G-splat. So, I had it add
193. 6:40bullets as well, rendered out the
194. 6:42sequence, and it looked like this. Not
195. 6:44quite Matrix yet, but then I added my
196. 6:47Miniax H3 cleanup workflow, and it
197. 6:50worked so well that I was even able to
198. 6:52prompt for this helicopter. Now, I would
199. 6:54have preferred to add this one in
200. 6:56Blender, but I think that we'll have to
201. 6:58wait until 3D sequence support gets a
202. 7:00bit better there. If you have any tips
203. 7:02for tools that can do that, please tell
204. 7:04me in the comments. Oh, by the way, the
205. 7:064DN1 pipeline and the workflows are all
206. 7:08completely free. But if you want to get
207. 7:10your hands on this H3 cleanup workflow
208. 7:12and the example files we created for
209. 7:14this video, consider becoming an
210. 7:15advanced supporter on Patreon. All
211. 7:17supporters also get access to exclusive
212. 7:19devlogs, beta workflows, and our Discord
213. 7:22community. So, thank you so much for
214. 7:23making weeks of R&D possible. But now,
215. 7:25without further ado, let me present the
216. 7:27final shot.
217. 7:33Now, I'm pretty happy with it, even if I
218. 7:35had to do this cleanup pass and couldn't
219. 7:37use the 40 gshion splats completely on
220. 7:39their own. But keep in mind, I also
221. 7:40really challenged this technique with
222. 7:42this type of shot. I mean, the intro
223. 7:44footage worked much better straight out
224. 7:46of the box. And of course, we can also
225. 7:48use the cleanup workflow there for some
226. 7:50more amazing results. Also, keep in mind
227. 7:52that this is like one of the first
228. 7:53synthetic data set workflows of this
229. 7:55kind, and it's already working quite
230. 7:57well with all the other improvements in
231. 7:59special AI like World Labs Atlas. I bet
232. 8:02the quality is going to improve
233. 8:04dramatically over the coming months. So,
234. 8:06make sure to subscribe to not miss any
235. 8:08developments in this field. See you next
236. 8:10time.