AI Can Now Animate Any Monster. Opus 5.5 Built the Tool — Transcript

Full transcript

1. 0:00Hey guys, have you ever tried to add a
2. 0:02cool creature to a game you make with
3. 0:04AI? Something that moves like in triple
4. 0:07A games and every time the AI tried to
5. 0:09animate it in Blender or anywhere else,
6. 0:11it was just bad. Well, what if I told
7. 0:14you I found a way to fix it and still
8. 0:17animate any creature I want. For people
9. 0:19it's easy. You have motion capture,
10. 0:21Mixamo, Kimodo because there are a huge
11. 0:23[snorts] databases of real people
12. 0:25moving. So, AI has plenty to learn from.
13. 0:28But for a dragon, a troll, or anything
14. 0:30with a strange body, [music] there is no
15. 0:32database like that. So, there is no tool
16. 0:34like that either. Not yet at least,
17. 0:36nothing public. And I bet it's coming
18. 0:39soon.
19. 0:40My way is honestly a bit crazy, but look
20. 0:42at this. A brand new monsters, the
21. 0:45lizard and the rat. And not a single
22. 0:47frame was fixed by hand. And it works
23. 0:50for any kind of creature. Four legs, two
24. 0:52legs, wings, a tail, a sword. And if you
25. 0:55watched my previous videos, you know
26. 0:57I've been fighting with this for a long
27. 0:59time. And most of the methods we tried
28. 1:01just broken our monsters. So, here is
29. 1:03the whole story. Every method we tried,
30. 1:06why it broke, and how the tool works
31. 1:08now. And if you're not subscribing, hit
32. 1:10subscribe button. So, let me take you
33. 1:12back to the very beginning. All right,
34. 1:14so it all started with our game. I
35. 1:17wanted to add some animals into it. So,
36. 1:19we [music] picked deer. And then Opus
37. 1:20wrote the walk itself, keyframes by
38. 1:22code. And well, it looked like this,
39. 1:25just blocky. Then much better model from
40. 1:27Tripo. And same walk by code and still
41. 1:30it was not real animal. So, we had this
42. 1:32crazy idea. Don't animate the deer at
43. 1:34all, just film it. Well, an AI video of
44. 1:37our own deer. And then copy that video
45. 1:39onto the skeleton. So, for every frame a
46. 1:41script written by Opus posed our model,
47. 1:44rendered it, and checked [music] does
48. 1:45the outline match. And for the deer it
49. 1:47kind of worked, but it had two big
50. 1:49problems. From the side the legs crossed
51. 1:51all the time.
52. 1:52>> [music]
53. 1:52>> And the script lost track of which leg
54. 1:54is which. So, Opus had to go back and
55. 1:56mark the feet itself [music] point by
56. 1:57point. And the peasant has the same
57. 1:59issue. He mixed up its near and far leg.
58. 2:01So again, Opus had to go through every
59. 2:03frame. And the second issue was the
60. 2:04depth. A man lifting a stone matched the
61. 2:07video almost to pixel, but you clearly
62. 2:09see a lot of issues with it. Because the
63. 2:11video is flat like a photo, and script
64. 2:13had to guess how far everything is. And
65. 2:15of course, it guessed wrong.
66. 2:16>> [music]
67. 2:17>> But first, one thing about Sidance. Give
68. 2:18it the same picture or the same words
69. 2:20twice, and you will get two different
70. 2:22videos. [music]
71. 2:23So, if we filmed the front and the side
72. 2:25as two separate videos, they would just
73. 2:27never match. So, for the troll, we gave
74. 2:29Sidance two cameras in one picture, the
75. 2:31front and the side. With two cameras,
76. 2:33[music] a point can only be in one spot.
77. 2:35The two lines crossed right here, and
78. 2:37the depth problem was gone. And then the
79. 2:39script said 84% match, like all checks
80. 2:42passed. And then I opened the lab, and
81. 2:44the majority of frames were just [music]
82. 2:45broken. So again, Opus went through
83. 2:47every frame itself. The video on top and
84. 2:50our model right under it. It found every
85. 2:52broken frames and just fixed it.
86. 2:54>> [music]
87. 2:54>> And that's when I said, "Okay, this can
88. 2:56be how we make every monster." Because
89. 2:59it takes a lot of time and a lot of
90. 3:01tokens for Opus to fix [music]
91. 3:02everything by hand. So, we needed one
92. 3:04tool for any monster and any move with
93. 3:06no hand fixes. So, we went shopping. And
94. 3:09here is every tool we tried and how it
95. 3:11[music] results. First, point tracking
96. 3:14from Google DeepMind. The first frame of
97. 3:16the video is our own render. So, we know
98. 3:18exactly which pixel is [music] which
99. 3:20part of the body. And the tracker just
100. 3:22follows every one of them through the
101. 3:23video. Sounds perfect, right? Well, look
102. 3:26at the legs. After a few steps, the
103. 3:28points [music] light up onto the body,
104. 3:29and the legs have nothing left. Then we
105. 3:31tried outlines from Meta AI M2. One
106. 3:34outline for every leg. On a walk, it
107. 3:37wasn't bad, but in a run, the legs blur,
108. 3:40and the far legs just disappear or jump
109. 3:42onto the chest. And sometimes the back
110. 3:44leg gets the color of the front leg, so
111. 3:46we don't know which leg is which
112. 3:48anymore. Then we tried an AI that
113. 3:50guesses depth from one picture to tell
114. 3:52the new leg from the far one. And bright
115. 3:55means close and dark means far. And here
116. 3:57the new leg and the far one get the
117. 3:59[music] same score.
118. 4:01We checked every frame and it was right
119. 4:02only four times out of 10. The two legs
120. 4:05are just 10 cm apart and that is too
121. 4:08small for this AI. So, we thought,
122. 4:10"Okay, let's make it easy." And we
123. 4:12[music] painted the legs, every leg its
124. 4:14own color like socks. And it worked.
125. 4:16Cidence kept the colors and in almost
126. 4:18every frame we knew which leg is which.
127. 4:21But here is the catch. Now everything
128. 4:22[music] depends on the Cidence keeping
129. 4:24the paint right. And remember, Cidence
130. 4:26never makes the same video twice. And
131. 4:29this time it kept the paint, next time
132. 4:31it might [music] not. And this is just a
133. 4:33wolf, four legs, four colors. What about
134. 4:35a dragon with wings or a centaur with
135. 4:38six limbs? So, we needed different
136. 4:40method. And then we tried animal pose
137. 4:42detectors trained on thousands of real
138. 4:44animal photos. From the side it draws
139. 4:47the far legs right on the belly and from
140. 4:49the front it's just a mess. And then the
141. 4:51newest research, Mockup Anything,
142. 4:53trained on a real animal motion capture.
143. 4:56And [music] our wolf is just running
144. 4:58inclined and the gallop is not perfect
145. 5:01either. And also it's just for animals,
146. 5:03so what about monsters? So, after all of
147. 5:06those tries, I was thinking, "Okay,
148. 5:08let's stop for a moment and just rethink
149. 5:10everything over." And then we had some
150. 5:13totally different idea. If both cameras
151. 5:15know exactly where a joint is in the
152. 5:17picture, the 3D part is just a simple
153. 5:19map. The two lines cross, done. So, the
154. 5:22whole problem is one thing, find every
155. 5:24joint in the picture in every frame and
156. 5:26know which one is which. But first,
157. 5:28every new monster needs a skeleton, so
158. 5:31that's now the first step of the tool.
159. 5:33We use Uni Rig and AI that looks at a 3D
160. 5:36model and draws the bone inside it. And
161. 5:39it's good, but not perfect. Here are
162. 5:41four tries of our rate, bones sticking
163. 5:43out of the body and ones no legs at all.
164. 5:46So, our tool checks each try against the
165. 5:48model itself. Every leg on the ground
166. 5:50needs a bone and left and right must
167. 5:52match and it keeps the best one. Then it
168. 5:55folds the fingers into the hand and cuts
169. 5:57off whatever sticks out of the body. So,
170. 5:59now the skeleton is inside the 3D model,
171. 6:02but finding those joints in the video is
172. 6:04the hard part and [music] no tool out
173. 6:05there can do it for our troll because no
174. 6:08AI has ever seen our troll, but we have
175. 6:10something better. We have the troll. So,
176. 6:13we [music] take Dino, an AI from Meta
177. 6:15that has already seen millions of photos
178. 6:17and we teach it to only a job finding
179. 6:20every joint of our troll. And for that
180. 6:22we render the troll in 3,000 random
181. 6:24poses from both [music] cameras. And
182. 6:26because we made the pictures, we know
183. 6:28exactly where every joint is and we make
184. 6:31it hard on purpose. About half of the
185. 6:33poses move fast, so the arms and legs
186. 6:35are [music] blurred because that's
187. 6:36exactly what Sidon's gives us. Every
188. 6:38picture is made like this. We take our
189. 6:40troll, bend every joint at random, but
190. 6:43only as far as real joint can go, turn
191. 6:45him, blur what moves fast, and then make
192. 6:48it a bit worse like Sidon's [music]
193. 6:49does. The light, the colors, and the
194. 6:51blurry arm. Then Dino learns this one
195. 6:54job on this one troll, about 30 minutes
196. 6:56on my graphics card. And now it reads a
197. 6:58Sidon's video like this. Every joint
198. 7:00lights up in both pictures frame by
199. 7:02frame in [music] just seconds. But how
200. 7:05do we know the tool is right without me
201. 7:07checking every frame? We give it an
202. 7:08exam. For every monster, even a brand
203. 7:11new one, we make up a motion ourselves.
204. 7:14Every joint moves at random, but only
205. 7:16inside its limits. And we film it in our
206. 7:18studio with the same two cameras. And
207. 7:20because we made that motion, we know
208. 7:21exactly where every joint is in [music]
209. 7:23every frame. The tool reads that video
210. 7:26like any Sidon's clip and we compare its
211. 7:28answer with our motion bone by bone. So,
212. 7:31our new lizard took this exam before we
213. 7:33paid for a single Sidon's video and
214. 7:35every change to the tool has to pass
215. 7:37this exam for every monster or it's out.
216. 7:39And before a Sidon's video even goes in,
217. 7:42the tool checks it. Do both pictures
218. 7:44show the same move? Because sometimes
219. 7:46Cidence throws two different moves in
220. 7:48the two halves, and then it's just
221. 7:50faster to make a new video than to fix
222. 7:52it. Okay, but remember depth anything? I
223. 7:55thought maybe we can use a depth map
224. 7:57here, too, and combine the two methods
225. 7:59to make the results even better. So, we
226. 8:01taught our own AI depth, as well,
227. 8:03because in our render we know exactly
228. 8:05how far every joint is. And on the same
229. 8:07move run, it picked the nearer leg right
230. 8:10every single time. And [music] Depth
231. 8:11Anything got it four times out of 10,
232. 8:13but combined it with our two cameras, it
233. 8:15changed almost nothing, because two
234. 8:17cameras already see the depth. So, can
235. 8:19it replace the second camera? Not yet.
236. 8:22It knows which leg is nearer, but not
237. 8:24exactly how far. So, here is the whole
238. 8:26thing from A to Z. A model from Tripod,
239. 8:29the skeleton in a few minutes, our
240. 8:31monster from two cameras in one picture,
241. 8:33and Cidence make it move. Then the tool
242. 8:36checks the video, reads every joint,
243. 8:38puts the move on the skeleton, and it
244. 8:40goes right into Blender and [music] into
245. 8:42the game. Four legs, two legs, wings, a
246. 8:44tail, a sword, it doesn't matter. And
247. 8:46before one troll used to take [music]
248. 8:48four days with Opus validating every
249. 8:51frame and a lot of tokens, now it's
250. 8:53[music] just a few minutes, because the
251. 8:55learning process is just one time per
252. 8:57monster. Then you can create any
253. 8:59animations you want with it. And it's
254. 9:01not magic, though.
255. 9:03If Cidence throws two different moves in
256. 9:05the two pictures, the tool can only
257. 9:06follow one.
258. 9:07>> [music]
259. 9:07>> And a hand hidden behind the body is
260. 9:09still a guess. And let me know in
261. 9:11comments what do you think about it and
262. 9:13if you want to try it out. As always,
263. 9:15thanks for watching and see you in the
264. 9:17next one.