YouTube2Text

How To Animate a Short Film with Blender + Higgsfield (Breakdown) — Transcript

by Higgsfield AI · 5,154 words · 791 segments · language en · Watch on YouTube

Full transcript

  1. 0:00Hi, I'm Amina, and welcome to the first
  2. 0:02episode of Hixeld Animation.
  3. 0:11This channel is dedicated to showcase
  4. 0:12our projects and dive deeper into our
  5. 0:15processes and pipelines in a form of a
  6. 0:17tutorial.
  7. 0:18Today, we're looking at our upcoming
  8. 0:20animated short called Passport Crush.
  9. 0:23It's a 5-minute action comedy about a
  10. 0:25girl who arrives at the airport and
  11. 0:27suddenly realizes she has left her
  12. 0:29passport across town.
  13. 0:31Now, she needs to go back for it and
  14. 0:33things get a bit out of hand. We played
  15. 0:35around on a relatable experience with
  16. 0:37absurd exaggerated moments of how the
  17. 0:39whole world seems to suddenly be against
  18. 0:41you when you're in a rush, especially
  19. 0:43when you're trying to make your flight.
  20. 0:45We also wanted to show you how we're
  21. 0:47combining traditional animation with AI
  22. 0:49and what that means during an actual
  23. 0:51production. Our team comes from
  24. 0:53traditional animation and we hadn't
  25. 0:54worked with AI before joining Hixeld.
  26. 0:57There's been a lot to learn, to tell you
  27. 0:58the truth, and we're still figuring out
  28. 1:00how these tools fit into the way we want
  29. 1:02to work. If you're exploring them, too,
  30. 1:04you might be working through some of the
  31. 1:05same questions and we'd like to share
  32. 1:07our experiences and hear what you're
  33. 1:09learning as well.
  34. 1:10So, on this channel, we'll break down
  35. 1:13our own projects through practical
  36. 1:15tutorials showing how we're fitting AI
  37. 1:17into our animation process without
  38. 1:19losing the craft. We'll explain the
  39. 1:21creative decisions behind the results,
  40. 1:22too,
  41. 1:23because our artistic voice is the most
  42. 1:25important part of our work and that
  43. 1:27continues to guide how we use these
  44. 1:29tools. The film is still in production
  45. 1:31and once it's finished, I'll share a
  46. 1:33full breakdown. For today, though, let's
  47. 1:35watch the car sequence and go through
  48. 1:37how we made it.
  49. 2:24>> Pre-production starts with an idea, a
  50. 2:26script, and storyboards as it usually
  51. 2:27does. For the sequence, we wanted to
  52. 2:29test how 3D previous and AI could work
  53. 2:32together in an action scene. Our
  54. 2:34storyboard artists began with the part
  55. 2:36where the girl goes into the taxi and
  56. 2:38the driver takes her through the busy
  57. 2:39traffic. We wanted to bring in some
  58. 2:41visual comedy and make the car feel
  59. 2:43jumpy and unpredictable, moving in a way
  60. 2:46a real car couldn't. We also needed to
  61. 2:48work out the art direction and prepare
  62. 2:50the assets, the character sheets, prop
  63. 2:52designs, and locations so that they
  64. 2:54would become our image inputs during
  65. 2:56generation, and we spent really good
  66. 2:58amount of time getting those right.
  67. 3:00The assets needed to read clearly and
  68. 3:02feel like they belonged in the same
  69. 3:04world. The production structure was
  70. 3:06familiar to how it is in traditional
  71. 3:07pipeline. We had a small team though, so
  72. 3:10people were wearing multiple hats and
  73. 3:12taking on several responsibilities.
  74. 3:14Everyone wrote prompt as part of their
  75. 3:16own work, whatever department they were
  76. 3:17in. We didn't have a separate prompt
  77. 3:19engineer, so the artists working on the
  78. 3:21characters, environments, and animation
  79. 3:24were writing the prompts for those parts
  80. 3:25of the project themselves. Let me show
  81. 3:27you the boards. You'll notice some
  82. 3:29panels are marked orange. Orange means
  83. 3:31Blender. Those are the shots we were
  84. 3:34planning to have a previous stage for,
  85. 3:36and the ones that are left blank will go
  86. 3:38straight to video generation. Our
  87. 3:40thought process was this: complex shots
  88. 3:42with specific movement, camera angle, or
  89. 3:44anything that takes longer to explain in
  90. 3:46words to model goes to Blender. Less
  91. 3:48complicated shots go straight to video
  92. 3:50generation.
  93. 3:52And one thing worth mentioning, boards
  94. 3:54are just a guide, not something set in
  95. 3:55stone. So, video generation tends to be
  96. 3:58very unpredictable, and we tried to
  97. 4:00leave some shots up to a chance,
  98. 4:02sticking to the story beats and main
  99. 4:04scenes rather than planning out every
  100. 4:06single shot because we expected things
  101. 4:08to change as we go.
  102. 4:10Now, let's talk about art direction.
  103. 4:12Here's the one rule that this whole
  104. 4:14pipeline is built on. We call it
  105. 4:16asset-first approach. Essentially, the
  106. 4:18quality of the image inputs dictates the
  107. 4:20quality of the overall film. If your
  108. 4:23image inputs are unclear, sloppy, or
  109. 4:25inconsistent, your overall film is going
  110. 4:27to be sloppy. You might be thinking,
  111. 4:29"Amina, what in the world are image
  112. 4:32inputs?" Well, those are character
  113. 4:34sheets, prop sheets, backgrounds, and so
  114. 4:36on. For people who prepare those assets,
  115. 4:39we came up with a rule called asset
  116. 4:40managers. Their work essentially
  117. 4:42includes making sure that characters,
  118. 4:44props, and backgrounds all feel like
  119. 4:46they belong in the same film and the
  120. 4:48same world, which is equivalent to what
  121. 4:50an art director does. For this project,
  122. 4:52we went with stylized 3D characters and
  123. 4:55watercolor backgrounds. Getting those
  124. 4:57two looks to work together well can be
  125. 4:59tricky in a traditional pipeline.
  126. 5:01However, that was a great challenge for
  127. 5:02this project. When it comes to our
  128. 5:04characters, we wanted them to be cute
  129. 5:06and funny with proportions that suited
  130. 5:08the exaggerated movement we had planned
  131. 5:10for the film. The comedy needed to come
  132. 5:12through in the design as well as the
  133. 5:14character performance.
  134. 5:16Our project schedules are usually very
  135. 5:18tight, sometimes only a month long.
  136. 5:20To get started, we generate some initial
  137. 5:22designs from prompts, then edit them by
  138. 5:25hand to save some time. That gives us
  139. 5:27different options to compare and develop
  140. 5:29before we settle on a single direction
  141. 5:31and idea. I'm going to show you an
  142. 5:32example of how we write a prompt for our
  143. 5:34character sheets. Usually, we stick to
  144. 5:37Claude to write prompts, but you can use
  145. 5:39any other AI chatbot that you like. Let
  146. 5:41me show you one of our requests followed
  147. 5:43by an example of a a character prompt.
  148. 5:46So, let's look at what's in that long
  149. 5:49prompt that describes the character's
  150. 5:52appearance, clothes, and pose, along
  151. 5:54with the framing and lighting.
  152. 5:56Claude helps us organize all of that
  153. 5:58information, but we still read through
  154. 6:00it before generating from time to time.
  155. 6:02We check that the details match our
  156. 6:04designs and correct anything that
  157. 6:06doesn't match. Those choices affect what
  158. 6:08comes out, so we want to be clear about
  159. 6:10what we're asking for.
  160. 6:12When it comes to image generation, we
  161. 6:14use our own image model called Soul 2.0
  162. 6:17that is available at Hicksfield. That's
  163. 6:19the model we went for to create all of
  164. 6:21the assets for this project. Our
  165. 6:23creative department, alongside our
  166. 6:25engineering team, developed this model.
  167. 6:27It was created by people with
  168. 6:29backgrounds in film, animation,
  169. 6:31photography, fashion, and graphic
  170. 6:33design, who helped shape the model's
  171. 6:35visual direction and the quality that we
  172. 6:37have. They wanted it to be useful for
  173. 6:39the kind of work they do in their own
  174. 6:41industries with attention to taste,
  175. 6:43style, and high visual quality that
  176. 6:46matters in those fields. Let me take you
  177. 6:48through the generation step.
  178. 6:50I open Cinema Studio on Hicksfield and
  179. 6:52choose Soul 2.0 in the model picker.
  180. 6:55Then,
  181. 6:56I paste the prompt, set the aspect
  182. 6:58ratio,
  183. 6:59choose a batch of four, and hit
  184. 7:01generate. Here's what came back. First
  185. 7:03one looks good, but she was a bit too
  186. 7:06goofy. We wanted funny, but also
  187. 7:08something still cool.
  188. 7:10And the hair and outfits don't quite
  189. 7:12feel right in my opinion, especially it
  190. 7:13comes to main character. I think they
  191. 7:15need to stand out more.
  192. 7:17This one is cute, but a little too
  193. 7:18realistic for the film style. We want a
  194. 7:21more expressive face and exaggerated
  195. 7:23proportions. This one was much closer.
  196. 7:25The ginger hair really stands out
  197. 7:27against the watercolor backgrounds,
  198. 7:28which I really like. Her proportions,
  199. 7:30however, still need some work.
  200. 7:33And I think she looks a little too
  201. 7:34dressed up for a flight. I don't know
  202. 7:36about you, but I usually pull up in
  203. 7:38sweatpants and most comfy clothes to the
  204. 7:40airport. So, this is the version we went
  205. 7:43with. We brought back the hoodie and
  206. 7:45sneakers from the first design and added
  207. 7:48the skirt for the silhouette.
  208. 7:49The contrast between her ginger hair and
  209. 7:52blue hoodie looks really great, too, and
  210. 7:55I kind of like it.
  211. 7:56Now,
  212. 7:57here comes the fun part, tweaking the
  213. 7:59details by hand.
  214. 8:01First thing that caught our eye was that
  215. 8:02her color palette seemed too dull,
  216. 8:04especially for that comedic energy that
  217. 8:06we're trying to go with for our story.
  218. 8:08So, we tweaked the brightness and
  219. 8:10saturation a bit in Photoshop to bring
  220. 8:12back all that color and brightness.
  221. 8:14We also decided to add some paint
  222. 8:16strokes around her silhouette to give
  223. 8:18off that hybrid look. I think those
  224. 8:20details add some dynamic and expressive
  225. 8:23hand-drawn feeling, but otherwise we
  226. 8:25kept the characters majorly
  227. 8:27cel-stylistically 3D.
  228. 8:29Once the main character was locked, it
  229. 8:31was easier to develop the rest of the
  230. 8:33characters by the same method, like our
  231. 8:35taxi driver.
  232. 8:37Once our main character rushes out of
  233. 8:39the airport, she gets into a taxi.
  234. 8:41The taxi driver is a middle-aged man
  235. 8:43with round, kind face who helps her get
  236. 8:46through the busy traffic fast.
  237. 8:48He looks very calm, but his driving
  238. 8:50style turns out to be very reckless. To
  239. 8:53be honest, I find that this character is
  240. 8:55the driving force, no pun intended, of
  241. 8:57the majority of our action scenes. We
  242. 9:00used the same process to develop his
  243. 9:01design, starting with a short
  244. 9:03description in Claude. Here's our
  245. 9:05request, followed by an example of the
  246. 9:07full character sheet prompt. In this
  247. 9:09one, the cheeks were a bit much here, I
  248. 9:11think, and I'm not sure what happened
  249. 9:13with his hair. This one is a different
  250. 9:16hairstyle in every view. This one was
  251. 9:18closer, but the skin and facial hair
  252. 9:20still felt too plasticky for our style.
  253. 9:22He also looks a little tired and
  254. 9:24worried, and we wanted him to look much
  255. 9:26more relaxed considering what he's about
  256. 9:29to put this girl through. This is the
  257. 9:30one we chose eventually. I love the cap
  258. 9:33and the big mustache. He looks friendly
  259. 9:35and very pleased with himself. I'll get
  260. 9:37in his car.
  261. 9:39Maybe once.
  262. 9:41Probably only once. Here he is after the
  263. 9:43hand-drawn details. We added little
  264. 9:45strokes around his face, on the cap, and
  265. 9:48through the clothes, just like we did
  266. 9:49with the girl. I especially like how
  267. 9:52they bring out the shape of his
  268. 9:53mustache. It looks very expressive. He
  269. 9:56still has that 3D look, but the John
  270. 9:58details helped the both characters feel
  271. 9:59like they belong in the same film.
  272. 10:02Main characters down. Now, off to
  273. 10:04locations. Now, since this sequence is
  274. 10:07right out of the airport, our location
  275. 10:08had to be the road where our characters
  276. 10:10drive through to get to the city. It's a
  277. 10:12long highway away from any buildings.
  278. 10:15Same process as with the characters, a
  279. 10:17short request sent to Claude, and it
  280. 10:19comes back as the full location prompt.
  281. 10:22Locations gave us the project's biggest
  282. 10:24style lessons by being quite challenging
  283. 10:26to execute. We wanted to go with pure
  284. 10:28watercolor visuals, but they ended up
  285. 10:30being sloppy, looking dirty, and
  286. 10:32soulless during tests. Our team kept
  287. 10:35hitting a dead end, and at some point we
  288. 10:37even seriously discussed dropping the
  289. 10:39watercolor look altogether and pushing
  290. 10:41the whole film towards something more
  291. 10:423D. But, we didn't give up yet and kept
  292. 10:45testing out both options to find our own
  293. 10:47balance between watercolor and 3D looks.
  294. 10:50Sometimes, you have to make a lot of
  295. 10:51mistakes before finding the right
  296. 10:52direction to go with. We still went
  297. 10:54through plenty of versions, but we could
  298. 10:56explore more ideas in the time we had.
  299. 10:59Even the ones we didn't use helped us
  300. 11:00rule things out and settle on an art
  301. 11:02direction sooner before we carried those
  302. 11:04choices into the rest of the production.
  303. 11:06The initial time of our film was midday,
  304. 11:08and that resulted in flat and dull look.
  305. 11:11It seemed like lighting was a big factor
  306. 11:12that we had to consider. Duh. So, we
  307. 11:15decided to keep the location, but settle
  308. 11:17on late afternoon lighting instead,
  309. 11:19adding a more dynamic light through
  310. 11:20prompt. Warm evening light and harsh
  311. 11:23shadows brought more volume and depth to
  312. 11:25our shots.
  313. 11:26This fix became the film's visual
  314. 11:28formula. Essentially, you generate
  315. 11:30location in watercolor style, then add
  316. 11:32in the golden hour lighting and prompt
  317. 11:34during production.
  318. 11:36That's why testing out assets in video
  319. 11:38early on in pre-production is very
  320. 11:40crucial. It allows us to face visual
  321. 11:42problems while we still can fix them
  322. 11:44quick. That is equivalent to animation
  323. 11:46test I'd say that we do in
  324. 11:47pre-production stage to test out
  325. 11:49characters. Except in this pipeline, we
  326. 11:52can test backgrounds too because due to
  327. 11:54AI's nature, backgrounds can influence
  328. 11:56the scenes in interesting ways like it
  329. 11:59did with some of our props like the taxi
  330. 12:01car. The taxi's present in a big chunk
  331. 12:03of the movie so we took our time with
  332. 12:05designing it in a way that made it feel
  333. 12:07like a character of its own.
  334. 12:09Our very first 3D tests were of this
  335. 12:10very taxi car driving through the road
  336. 12:13where our animators explored the way
  337. 12:14it's going to move. The car needed to be
  338. 12:17jumpy, have a good amount of squash and
  339. 12:18stretch, and complement its owner's
  340. 12:20personality almost like a horse and its
  341. 12:22rider. For the shots that have this car,
  342. 12:24we planned to make a 3D previous first
  343. 12:27and then attach it as a video input
  344. 12:28during production to keep the movement
  345. 12:31and bring more control to the animation
  346. 12:33itself. That way, we could work out the
  347. 12:35movements ourselves and give the model
  348. 12:37something to follow. When we already
  349. 12:39know how we want something to move, it
  350. 12:41can be quicker to animate it than to
  351. 12:43explain every single detail with words
  352. 12:45in a prompt. We are visual artists. The
  353. 12:48taxi also needed to fit into our
  354. 12:49watercolor world. So at first, we gave
  355. 12:52it a lot of painted texture. During our
  356. 12:54first attempts, the car was in a very
  357. 12:56obvious watercolor style. However, in a
  358. 12:59video, it looked like these textures
  359. 13:01laid onto the 3D form incorrectly which
  360. 13:04made it look cheap like a PNG stretched
  361. 13:06over 3D geometry. Not good at all.
  362. 13:09So we kept making iterations. Here's our
  363. 13:11taxi request so far.
  364. 13:13A watercolor drawing of an old yellow
  365. 13:15taxi, 3/4 front left view, single car in
  366. 13:18frame, rich watercolor washes with drips
  367. 13:21and highlights, soft shadow under the
  368. 13:23car.
  369. 13:24Claude turns that into the full prompt
  370. 13:26for a prop sheet, which we ran through
  371. 13:29Soul 2.0 inside Higgsfield. So, why did
  372. 13:32the watercolor car keep falling in
  373. 13:34motion?
  374. 13:35Here's what we figured out after a lot
  375. 13:37of takes. Heavy watercolor texture is a
  376. 13:40lot of information for a model to hold
  377. 13:42once things start moving. Put a
  378. 13:44watercolor prop on top of a watercolor
  379. 13:46background, and that's double the load.
  380. 13:48Two dense textures fighting in every
  381. 13:50single frame. So, we decided to lighten
  382. 13:53it up a bit. The car became clean 3D,
  383. 13:55and the paint stayed in the world around
  384. 13:57it. After locking the car, we also made
  385. 13:59a roofless version of it. Here's an
  386. 14:01example of how you can edit an existing
  387. 14:03image instead of generating from
  388. 14:05scratch. This time, we'll go into SeaArt
  389. 14:075.0 Pro, which we use for edits like
  390. 14:10this. Here's the request and the full
  391. 14:12prompt. And we also needed an inside
  392. 14:14view of the backseat for some of our
  393. 14:16shots where we used the same workflow of
  394. 14:18attaching our main car design. To
  395. 14:21populate the highway, we speed ran the
  396. 14:23rest of the props. Trucks stayed
  397. 14:25watercolor since they are on screen
  398. 14:26briefly and didn't require too much
  399. 14:29editing.
  400. 14:30We don't usually build assets for
  401. 14:31everything. Only for specific props that
  402. 14:34have to reappear from scene to scene and
  403. 14:36have to look consistent throughout. For
  404. 14:38example, we had three trucks that would
  405. 14:41keep appearing in more than one shot.
  406. 14:43So, they needed their own prop sheet.
  407. 14:45The rest can be achieved through text
  408. 14:47video. Very simple. We also made an
  409. 14:49effects sheet for our explosion scene at
  410. 14:51the end of the sequence. That part was
  411. 14:53very fun and that was an interesting
  412. 14:55process. You see, our initial request
  413. 14:58said mushroom-shaped explosion. Little
  414. 15:01did we know that it will literally
  415. 15:03generate a mushroom-shaped explosion
  416. 15:05with smoke.
  417. 15:07So, we changed the wording to nuclear
  418. 15:09blast instead, which made it fix, more
  419. 15:12or less. But, something was still
  420. 15:13missing. So, we had an idea.
  421. 15:16We took the base and added some
  422. 15:17hand-drawn textures to it to match the
  423. 15:19playful art direction of the film that
  424. 15:21you saw before on our characters. And
  425. 15:23thankfully, it made it to the final cut.
  426. 15:31With most assets locked, we were ready
  427. 15:33to go into production.
  428. 15:36Most of the time, production here means
  429. 15:38one thing. We generate video from text
  430. 15:41with our assets attached as image
  431. 15:42inputs.
  432. 15:44That covers the comp shots, but some
  433. 15:45shots need one extra stage first, a
  434. 15:48preview stage, like this scene. Our car
  435. 15:50gets blocked by two trucks and there's
  436. 15:52no way around it. But who said we can't
  437. 15:55pull off a stunt and squeeze right
  438. 15:57between them? Totally optional, by the
  439. 15:59way. The preview stage covers two of our
  440. 16:01needs here. It helps us lock the camera
  441. 16:03angle and also lock the timing, which is
  442. 16:06super important. We added some squash
  443. 16:09and stretch into the wheels, just enough
  444. 16:11to get rid of the stiffness and add some
  445. 16:13character to the car. Then, we export
  446. 16:15the scene as a plain Blender preview. No
  447. 16:18color, no materials, only the gray geo,
  448. 16:21motion, and the camera. Now, how do we
  449. 16:23write the prompt for a video? The way
  450. 16:26you write your prompt can make or break
  451. 16:28your scene. It's a whole science of its
  452. 16:30own, believe me. Let's look at one of
  453. 16:32the prompts we made.
  454. 16:33Yes, it's very long. I know you're
  455. 16:36thinking, "Who writes an essay for a
  456. 16:384-second shot?" Well, we do, because
  457. 16:41every detail we don't write, the model
  458. 16:43decides for us. And trust me, you don't
  459. 16:46want that. It is a terrible guesser. One
  460. 16:49thing before we go block by block, the
  461. 16:51order matters. The first 20% of the
  462. 16:54prompt matters a lot because the model
  463. 16:56reads it from top to bottom, like we
  464. 16:58humans do. And the further it reads, the
  465. 17:00less strictly it follows the prompt. So,
  466. 17:03let's go over the blocks that we have in
  467. 17:05the front. First, we have scene context
  468. 17:07and style block, which is where we
  469. 17:09establish what happens in the shot in
  470. 17:11plain sentences. Plus, this is where we
  471. 17:13add the style rules, the physics of the
  472. 17:15world, and so on. Very important. Then
  473. 17:17we have active references block where
  474. 17:19your assets are present in the scene in
  475. 17:21the form of an attached image or videos,
  476. 17:23like a call sheet. It's important to
  477. 17:25specify what is what, otherwise the
  478. 17:28model will try to guess it and again,
  479. 17:30it's a terrible guesser. This is where
  480. 17:31you can add the personality, the quality
  481. 17:33description for the assets, but do not
  482. 17:36add visual description. That's what the
  483. 17:38image inputs are for.
  484. 17:40You can also add this line, characters,
  485. 17:43environments and props, 100% match the
  486. 17:45reference. That line carries the
  487. 17:47consistency so that it sticks to the
  488. 17:49visual input that you have. Then we have
  489. 17:51the short structure block. This is where
  490. 17:53we write down each shot that is divided
  491. 17:54by hard cuts.
  492. 17:56This is where we describe the action
  493. 17:58beat by beat with timing. What happens
  494. 18:00first, what's next, how long each
  495. 18:02movement holds.
  496. 18:04This is kind of the same job as what
  497. 18:05storyboard panel does, just put in
  498. 18:08words. Then we have the block for locks
  499. 18:11and constraints.
  500. 18:12Video models love adding their own ideas
  501. 18:15and the constraints keep them on track.
  502. 18:17These are the rules the model has to
  503. 18:19follow strictly.
  504. 18:20Now, constraints vary from project to
  505. 18:22project, so this is for you to decide
  506. 18:24what you want to use, but we will always
  507. 18:26use no music, SFX only, because that
  508. 18:29helps us with post production. Then we
  509. 18:31have a block for animation principles.
  510. 18:34The 12 principles of animation have
  511. 18:35their own section in the prompt, too. We
  512. 18:38include squash and stretch,
  513. 18:39anticipation, follow through and all the
  514. 18:41other rules and ask for them to be
  515. 18:43applied in our scenes. The model doesn't
  516. 18:45follow all of them consistently though,
  517. 18:47but including them helped us get closer
  518. 18:49to the movement we wanted. Do we type
  519. 18:51all of this every single time? Of course
  520. 18:54not.
  521. 18:55We wrote the structure down for Claude
  522. 18:56once and it applied it to every single
  523. 18:58prompt after.
  524. 19:00We call it a skill, also known as a
  525. 19:02system prompt. We tend to share around
  526. 19:04the same system prompts with the team,
  527. 19:06so everyone follows a similar structure,
  528. 19:08more or less. Let's go through the whole
  529. 19:10process using a shot where the taxi
  530. 19:12squeezes between two trucks again. We
  531. 19:14started with the storyboards. Then the
  532. 19:16taxi approaches the gap, tips onto two
  533. 19:18wheels, squeezes through, and lands back
  534. 19:21on all four once it's clear of the
  535. 19:22trucks. We needed that order to read
  536. 19:24clearly, especially with everything
  537. 19:26happening so quickly. Then we worked out
  538. 19:29the movement in Blender. This is where
  539. 19:31our animators had fun. We could adjust
  540. 19:33when the car starts tilting, how long it
  541. 19:35stays on two wheels, and how it settles
  542. 19:37after landing. The camera needed to
  543. 19:39follow all of that without losing the
  544. 19:41action, too. So working in Blender
  545. 19:43allowed us to play with the shot back
  546. 19:45and forth and adjust the camera
  547. 19:46alongside the animation.
  548. 19:48The control was super important here.
  549. 19:50And when we exported a simple gray
  550. 19:52preview from Blender, at this stage
  551. 19:54we're using it to show the movement and
  552. 19:56framing only. Our image references then
  553. 19:59came over and would provide the final
  554. 20:01designs.
  555. 20:02In Hicksfield, we attached that preview
  556. 20:04as the video input. Then we added the
  557. 20:06image references for the taxi, both
  558. 20:08trucks, the characters, the road, and
  559. 20:10the light. All the necessary stuff. Here
  560. 20:13we use Claude to help put the scene
  561. 20:15prompt together. It describes the action
  562. 20:17and explains what each reference is for.
  563. 20:20The video guides the movement and
  564. 20:21camera, while the images show how
  565. 20:23everything should look.
  566. 20:25We also specify details like keeping
  567. 20:27both characters in their seats and
  568. 20:29waiting until the taxi clears the trucks
  569. 20:31before it lands. Otherwise, it just
  570. 20:34keeps breaking. Once we check the
  571. 20:35prompt, we generated the shot.
  572. 20:38Then we watched it back against the
  573. 20:40previews.
  574. 20:41We checked the timing, the camera, and
  575. 20:43whether the car and characters still
  576. 20:44matched our designs, which
  577. 20:47thankfully it did. Remember the orange
  578. 20:49marks on the boards?
  579. 20:50We used Blender for shots where we had
  580. 20:52specific movement or camera work in
  581. 20:54mind. If it seemed quicker to animate an
  582. 20:56action than to explain it in words,
  583. 20:59we made a preview. For simpler shots, we
  584. 21:01went straight from boards to video
  585. 21:03generation, text to video.
  586. 21:06We still reviewed and adjusted the
  587. 21:07generations in the same way. Those shots
  588. 21:10just didn't need that extra blender
  589. 21:11stage. That's how we approach the
  590. 21:13individual shots. But, let me go back a
  591. 21:16little because some of those decisions
  592. 21:18came from tests we ran before starting
  593. 21:20the full sequence. Between
  594. 21:21pre-production and production, we ran a
  595. 21:23test stage. We took the assets we just
  596. 21:26made and put them straight into video to
  597. 21:28see whether it will work or not in the
  598. 21:30final shot. It is better to find out now
  599. 21:32and this way while we are still in
  600. 21:34pre-production stage and not in the
  601. 21:36middle of production when it gets harder
  602. 21:38to come back and make edits. You know
  603. 21:40what I mean.
  604. 21:41We didn't have any of our rules yet at
  605. 21:43that point. Most of them came from these
  606. 21:45three lessons.
  607. 21:47The lessons and learnings we find during
  608. 21:49this stage later become our project's
  609. 21:51visual rules. Here's what we learned.
  610. 21:53Lesson one,
  611. 21:54the watercolor car.
  612. 21:56We talked about this in the taxi section
  613. 21:58and these are some of our early tests.
  614. 22:00We used them to work out how much
  615. 22:02texture do we need to keep on the car,
  616. 22:04then carry that visual direction into
  617. 22:05our prompt. That helped us to keep the
  618. 22:08surfaces consistent so we didn't
  619. 22:10suddenly get a photorealistic car in the
  620. 22:12middle of a painted scene.
  621. 22:14Lesson two, we started testing before
  622. 22:16the character sheets were ready, so the
  623. 22:18car at first was empty.
  624. 22:20Random people kept showing up inside for
  625. 22:22some reason even though they weren't in
  626. 22:24our references or prompts or anywhere
  627. 22:26else. Once we added our characters in,
  628. 22:28that stopped. Lesson three, the
  629. 22:30lighting.
  630. 22:31One of our teammates added evening light
  631. 22:33to a prompt just to see what would
  632. 22:35happen. And the shots suddenly started
  633. 22:37looking warmer and had more depth. So,
  634. 22:39we kept using it. Eventually, low warm
  635. 22:42sun and long shadows became the
  636. 22:44direction we kept going for. We carried
  637. 22:46that lighting lock into production along
  638. 22:48with the car and character references we
  639. 22:50had tested.
  640. 22:54There are 18 scenes in this sequence. I
  641. 22:56want to show you three that took some
  642. 22:58extra work. In this shot, the girl is
  643. 23:00right next to the truck's door. We had
  644. 23:02already animated a reference in Blender,
  645. 23:05but the generations weren't coming out
  646. 23:06right. We figured that the camera was
  647. 23:09too close to the action in our Blender
  648. 23:11preview. It looked fine to us because we
  649. 23:13knew the scene and could tell what
  650. 23:15everything was supposed to be, but
  651. 23:16without textures or facial details, most
  652. 23:19of the frame was just large gray shapes
  653. 23:21with very little of the surroundings
  654. 23:23visible. That made it difficult for the
  655. 23:25video model to understand how the
  656. 23:26characters and objects were positioned.
  657. 23:28So, we went back into Blender and pulled
  658. 23:31the camera back a little. More of the
  659. 23:33scene was visible this time. The action
  660. 23:35was easier to follow, and the next
  661. 23:36generation came out right. Rule one, if
  662. 23:39the model doesn't understand your
  663. 23:40reference, you probably can't see enough
  664. 23:42of it. Zoom out. In this scene, she
  665. 23:44fastened both seat belts so they formed
  666. 23:46an X over her chest. We thought it would
  667. 23:49be fairly straightforward, especially
  668. 23:50since we'd already tested the action and
  669. 23:53it worked. Those first tests used an
  670. 23:55earlier character design. And when her
  671. 23:57final design was ready, we regenerated
  672. 23:59the scene with it again, keeping the
  673. 24:01same prompt and the other references.
  674. 24:04But this time, the camera kept changing
  675. 24:06angles between takes, and we couldn't
  676. 24:08get the framing we wanted consistently.
  677. 24:10We had expected to update her appearance
  678. 24:13and keep the rest of the shot as it was.
  679. 24:15Instead, we had to spend more time
  680. 24:17getting the camera right again. We took
  681. 24:18screenshots from the takes that had the
  682. 24:20camera angle we wanted and attached them
  683. 24:22as the references for the next attempts.
  684. 24:25That helped keep the framing consistent,
  685. 24:27and we were getting much closer to the
  686. 24:29shot we wanted. Then we noticed her
  687. 24:30headphones were missing in every new
  688. 24:33take. If you look at the screenshot we
  689. 24:35used, you can see they're missing there,
  690. 24:36too. We had picked it for the camera
  691. 24:39angle and missed that detail. Her
  692. 24:41character sheet still had the
  693. 24:42headphones, but we were giving the model
  694. 24:44references that didn't quite match. So,
  695. 24:47that meant we needed to edit the
  696. 24:49screenshot that was causing the
  697. 24:51headphone disappearance and match it
  698. 24:52with the character sheet. We opened the
  699. 24:54screenshot in Photoshop, drew the
  700. 24:56headphones back in by hand, and used
  701. 24:58that version as the reference.
  702. 25:00In the next generation, she had them
  703. 25:02already, and the camera angles were
  704. 25:03still right. Now, before we finish,
  705. 25:06there's another shot I wanted to show
  706. 25:07you, and that one is my favorite.
  707. 25:10Unfortunately, it didn't make it into
  708. 25:11the sequence because we haven't managed
  709. 25:13to get it working yet.
  710. 25:15We planned a little gag inside the taxi.
  711. 25:17We thought this one would be easy.
  712. 25:20Mid-chaos, the girl grabs the ceiling
  713. 25:21handle, it rips off. She panics, grabs
  714. 25:24the other one, that one rips off, too.
  715. 25:26Looking at the boards, we weren't too
  716. 25:28worried about this shot. I mean, we're
  717. 25:30making a car chase with flying logs and
  718. 25:32explosions. Surely we could manage to
  719. 25:35handle ripping off, right?
  720. 25:37Now, here's the prompt from one of our
  721. 25:38attempts. This shot was very easy to
  722. 25:41underestimate since it required to keep
  723. 25:43track of the smallest details. A lot of
  724. 25:45it was about keeping the handles
  725. 25:47consistent. We had to describe where
  726. 25:49they were attached, how they came off,
  727. 25:51and how they should look once she was
  728. 25:53holding them.
  729. 25:54I mean, we were also struggling with her
  730. 25:56performance. She needs to grab the first
  731. 25:59handle in a panic, then have a moment to
  732. 26:01realize it's broken before reaching for
  733. 26:03the other one. That little pause
  734. 26:06mattered. That small bit of reaction was
  735. 26:08crucial to deliver the gag. We knew this
  736. 26:10as animators, but we couldn't get those
  737. 26:13beats right through text prompts alone,
  738. 26:15so we turned to Blender. We thought that
  739. 26:17animating the performance ourselves and
  740. 26:19using it as a video reference would help
  741. 26:21it. It should help it. But, that proved
  742. 26:24to be tricky as well. With a simple gray
  743. 26:26character in our Blender preview, the
  744. 26:28expressions and hand movements weren't
  745. 26:30coming through clearly in the
  746. 26:31generations. When we added more detail,
  747. 26:34or at least tried, the model started
  748. 26:36copying the Blender's geo shape instead
  749. 26:38of following our actual character sheet
  750. 26:40and her design. We called this a
  751. 26:43performance gap.
  752. 26:44We knew how she should react, where she
  753. 26:47needed to pause, but we couldn't get the
  754. 26:49performance we wanted while keeping her
  755. 26:51final character design consistent in the
  756. 26:53AI generated shot.
  757. 26:55What surprised us was how much trouble
  758. 26:57we had with a scene that seemed fairly
  759. 26:59straightforward to animate
  760. 27:01traditionally, to be honest. We had
  761. 27:03expected the bigger action scenes to be
  762. 27:04harder, but turns out AI struggles with
  763. 27:07the subtle character acting instead.
  764. 27:10Setbacks like this help us figure out
  765. 27:11what to work on next. We haven't solved
  766. 27:14every shot, but making the car chase
  767. 27:16sequence has given us a clear idea of
  768. 27:18where to focus our next tests.
  769. 27:21There's still a lot to explore as we
  770. 27:22finish the film, and we're looking
  771. 27:24forward to sharing what we discover in
  772. 27:26the full breakdown. For this sequence,
  773. 27:28we worked on 18 scenes in just 3 days. I
  774. 27:31know, crazy timeline.
  775. 27:34There's still more to figure out as we
  776. 27:35work on the full film. Once it's
  777. 27:37finished, we'll share the full Passport
  778. 27:38Trash breakdown, including all the
  779. 27:40production numbers, the total number of
  780. 27:42generations, how much we spent in terms
  781. 27:44of budget, and how long the whole film
  782. 27:46took to make.
  783. 27:48If there's a part of your own workflow
  784. 27:49you're struggling with, tell us about it
  785. 27:51in the comments. Let us know what you've
  786. 27:53tried and where you're getting stuck.
  787. 27:55That way, we can focus future videos on
  788. 27:58the things you need help with. Thank you
  789. 27:59so much for watching, and if you'd like
  790. 28:01to follow along, subscribe, and I'll see
  791. 28:03you in the next one.

About this transcript

This page contains the full transcript of How To Animate a Short Film with Blender + Higgsfield (Breakdown) by Higgsfield AI, generated from the public captions YouTube serves with the video. The transcript has 5,154 words across 791 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.