YT YouTube2Text

YouTube2Text → Transcripts → How to Turn a Storyboard into a Full AI Video in Minutes

How to Turn a Storyboard into a Full AI Video in Minutes — Transcript

by Jack Vs. AI · 3,379 words · 478 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00AI filmmaking just got ridiculously easy
0:02because you can now take a single
0:04storyboard image and use AI to transform
0:07it into an entire sequence. So, you can
0:10forget about generating every shot
0:12separately, struggling with dozens of
0:14manual tweaks, and burning through
0:16credits. I'm Jack. I've spent over a
0:18decade creating visual effects for many
0:20of the biggest brands on the planet. And
0:22now I'm using AI to reinvent creative
0:25workflows. Used correctly, this method
0:27can be incredibly powerful because it
0:29completely changes the speed at which
0:31you can prototype ideas, create short
0:33films, and test product ads. In this
0:36video, I'll teach you the entire
0:38workflow, how to generate storyboards AI
0:40actually understands, and how to bring
0:42your vision to life faster than ever.
0:45And if you do enjoy today's video, be
0:46sure to whack a like on there for me and
0:48consider subscribing. All right, let's
0:50get into it. I'm sure you guys have seen
0:52a bunch of these videos over on social
0:54media recently where you have the
0:56storyboard stacked on top of the
0:58finished AI video. A load of it is going
1:02viral right now, and for good reason
1:03because it's a pretty interesting
1:05approach to using generative AI tools in
1:08a different way. The problem is no one
1:10is really showing you exactly how to put
1:13this workflow into practice. So, I'm
1:15going to set that right today and give
1:17you all of the information and materials
1:20that you need to do this yourself. So,
1:22everything I've generated using this
1:25workflow was made over on Higgsfield AI.
1:28And if you guys did want to follow
1:29along, you'll find a link down below in
1:31the description. So, you can see that
1:32I've got all of my storyboards generated
1:35here. These were made using GPT image 2.
1:39I've actually come to really like this
1:41image model now. And I really wanted to
1:43make sure that I didn't just test out
1:45one specific style with this workflow.
1:49We've got some photoreal stuff here, so
1:51like a cooking scene. Uh we've obviously
1:53got the anime of the snowboarders going
1:56down the mountain you would have seen
1:57already. Then we've got this kind of
1:59Pixar-inspired
2:00short story as well. And of course,
2:02we've also got all of our finished video
2:04generations where we're using those
2:06storyboards as [music] image references
2:09and generating the videos through C
2:12Dance 2. But let's just rewind a little
2:14bit and go back right to the start of
2:17the process because I want to show you
2:18guys a step [music] that some creators
2:20might not really be sharing in detail.
2:23Now this is only really necessary if you
2:25want to have a really high degree of
2:27control over your storyboards. But
2:29something I found that made this
2:31workflow even better was actually
2:33creating these character sheets before
2:36generating the storyboards. So this is
2:38actually pretty simple to do inside of
2:41Higgs Field, you first want to go ahead
2:43and select GPT image 2. You can see that
2:46I've uploaded a few different image
2:48references of myself. These were just
2:50shot on my phone, super simple. And for
2:53the prompt itself, which of course I
2:55will go ahead and share with you guys,
2:57it might look complicated, but the only
2:59part of it you guys actually need to
3:01tweak is this bottom section here. All
3:04of this is just a framework, so you can
3:06just leave it as is. The bottom part is
3:09where you're going to kind of embed the
3:11style of what you're after. So for this
3:13particular example, I made myself as a
3:16chef. I'm pretty much just wearing what
3:18I normally would, but I've just got that
3:19brown apron. So I wanted to start with a
3:21really basic, also photoreal example. So
3:24you can see I'm just leaving the prompt
3:26framework exactly as is, uploading my
3:29image references with a few different
3:30angles of myself with good lighting, and
3:33I'm just saying neutral expression
3:35wearing a brown leather apron, black
3:37cargo trousers, and dark unbranded
3:40Gore-Tex trainers. Let me also go ahead
3:42and show you guys how I built this
3:44Pixar-style version of myself as well.
3:47If you tap on this icon here, that's
3:50essentially to recreate what you've
3:53already built. So, it's going to just
3:54throw in all of your image references
3:56and the prompt that you use. So, if I do
3:58that for the Pixar version of me, you
4:01can see that the prompt has been tweaked
4:02ever so slightly, but the image
4:04references are exactly the same. All I
4:06added to the end of the prompt was Pixar
4:08animation style. So, just really simple.
4:11And you can also go ahead and create
4:12these product reference sheets as well.
4:15This is just using a tweaked version of
4:17the same framework, which again I will
4:19share with you guys. And you can see
4:21that I actually didn't need to use any
4:23image references here at all. So, this
4:26is for me that first step that some
4:28other creators aren't really going into
4:30detail with. And ultimately, it's just
4:32going to give you a lot more control
4:34over your final result. Now that we have
4:37our character and product references
4:39built, we can look at actually
4:40generating those all important
4:42storyboards. And to do this, I want to
4:44make an example with you guys to take
4:47you through the entire process from
4:49scratch. So, the first thing I'm going
4:50to do is just click on the recreate icon
4:53because I know that I want to use all of
4:54these image references again, and I want
4:57this prompt framework here, but I'm just
4:58going to remove this kind of style
5:00reference at the end. And instead, I'm
5:02going to put in rubber hose animation
5:05style. So, before we send this off, we
5:07want to make sure that we've got high
5:09selected for the visual fidelity. In
5:12terms of resolution, 2K is going to be
5:15good enough for this specific [music]
5:17process, but you can go all the way up
5:19to 4K if you want to. 16 by 9 is the
5:22aspect ratio that we want. And let's go
5:25ahead and generate two different options
5:27just so we have a bit of a point of
5:28comparison. Let's send this off. Looking
5:31at our results, these are pretty cool. I
5:33think I prefer this one here to this
5:37particular version just cuz it's got the
5:38kind of Mickey Mouse gloves in there,
5:40and the face is a little bit strange on
5:42this one. So, we're going to go ahead
5:44and use this here. You want to get this
5:47downloaded to start. And the next thing
5:49we need is our text prompt. The
5:51framework itself is actually going to be
5:53really quite complex, but thankfully,
5:55[music]
5:56everything that we have to do is pretty
5:58simple. And that's all thanks to an MD
6:00file that I'm going to share with you
6:02guys as well. The great thing about this
6:05is that you can use it in Claude,
6:06[music]
6:06Gemini, ChatGPT, any large language
6:09model that you [music] prefer. So, what
6:11you want to go ahead and do is click the
6:13upload button, and then I will, of
6:15course, share with you guys the
6:16storyboard prompt builder, so you'll
6:19drop that straight into your LLM of
6:22choice. And we also want you go ahead
6:24and upload that image of the character
6:28that we just created. So, let's drop
6:30that straight in here as well. And like
6:32I said at the start of the video, the
6:34character design step was completely
6:37optional. So, if you haven't made one of
6:38those up, just don't upload it. It's
6:40completely fine. You're still going to
6:42get a really good quality text prompt,
6:43you're just going to have a little bit
6:45less control over your end result. And
6:47literally, all that we need to do now is
6:49describe the type of scene that we're
6:52after. So, I'm going to say,
6:54"Ultra dynamic cartoon scene where the
6:57bearded man is making a latte in a
7:00coffee shop, finishing with some
7:02stunning latte art." We can just go
7:04ahead and send this off, and Claude's,
7:07or whatever LLM you're using, is going
7:10to write you out this really detailed
7:12prompt. You can see here it's saying,
7:14"Reading the storyboard prompt builder
7:16skill." It's going to flesh everything
7:18out for us. And as you can see, we get
7:21back this incredibly detail-rich prompt
7:24here, which is going to give GPT image 2
7:27everything that it needs to create our
7:29storyboard. So, all you want to go ahead
7:31and do is copy this prompt here. We can
7:34head back over to Higgs Field, and we
7:36can drop [music] this into the text
7:38prompt field. We don't have to make any
7:41changes to this whatsoever. The only
7:43thing we do want to go ahead and do is
7:45select this as our image reference.
7:48Again, if you skip this step, no worries
7:50at all. You are just going to get a
7:51slightly more generic result. So, I'm
7:53going to click this icon here, and then
7:56this kind of rubber hose style animation
7:58is going to be used as an image
8:00reference for our GPT image to
8:04storyboard. Now, for the storyboard
8:06itself, I would suggest [snorts] bumping
8:08up to 4K here because the more kind of
8:11detail and fidelity we give to Seed ants
8:142, the higher quality animation you're
8:16going to get at the end of it. Again, we
8:18want 16x9 aspect ratio, and I'm going to
8:20have a batch size of two. So, let's
8:22generate our storyboard. [music] Okay,
8:25so we have back both of our storyboard
8:27generations here, and I think both of
8:30them are looking really pretty cool. The
8:32style has stayed nice and consistent
8:34throughout. There's just one minor
8:36problem here, and that is that,
8:39interestingly, for both generations, the
8:41storyboard panel five has become like
8:45almost photoreal, especially like the
8:48kind of coffee grounds that falling into
8:50the puck. I think that's what it's
8:51called. But, what's really useful about
8:53an image model like GPT image 2 is that
8:56we can go ahead and I'm just going to
8:58clear out the prompt field here and
9:00remove my current image reference. If I
9:03click on the reference icon here, we're
9:05now going to be able to make edits to
9:07our storyboard. And I'm simply going to
9:09say, "Ensure that panel five has the
9:12same rubber hose animation style [music]
9:15applied to it as the rest of the film."
9:18We're going to leave the resolution at
9:194K, and let's just go for a batch size
9:23of one here and send off this generation
9:25to tweak our existing storyboard. And
9:28you can now see that the issue we had
9:30with our panel five [music] has been
9:32completely rectified. This feels a lot
9:34more in line with the rest of our short
9:37story here. So, now what we want to go
9:39ahead and do is hover over the video
9:41tab, and we can go across to C Dance 2.
9:45The first step is that we want to upload
9:48our media. So, if we head across to
9:51image generations here, you can see that
9:53we've got all of the storyboards and
9:55character references that we built up
9:58using GPT image 2. First step you want
10:01to do is click on check eligibility, and
10:03this is basically just going to scan
10:05your images checking for any kind of
10:07copyright infringement or IP issues. You
10:10can also go ahead and do the same thing
10:12to our character reference sheet as
10:15well. And just like that, both our
10:17character reference sheet and storyboard
10:19have been deemed eligible. So, we can
10:21click on the storyboard for starters to
10:24select this. And if I just go over to
10:27one of my previous generations and click
10:29on top of the prompt, it's just going to
10:31fill this field here with the exact same
10:33text prompt, which is really useful. And
10:35all that we're saying for our text
10:37prompt, really quite simple, is use the
10:40reference storyboards to make a full
10:42animation [music] movie. The audio,
10:44we're saying diegetic sound only,
10:47natural ambience, environmental foley,
10:49and subject-driven sound. Now, you could
10:52leave this off your text prompt if you
10:54didn't mind C Dance 2 adding music for
10:56you, but definitely make sure that you
10:58leave on no text. And the reason that
11:01this prompt can be so kind of basic is
11:04the fact that if we go ahead and open up
11:06our storyboard in full screen, C Dance 2
11:09has got all of this information embedded
11:13into the storyboard itself. So, it's
11:15able to kind of extract the text from
11:17this and understand what it is that
11:19we're after. And one extra step, and
11:21again, this is optional, but if you want
11:24a kind of greater level of control over
11:26your end result, is you might want to
11:28add
11:29>> [music]
11:29>> the character sheet as well. I've had
11:31good results from just using the
11:33storyboard, but also adding the
11:35character sheet can give you a little
11:37bit more control, bit more consistency,
11:40and ultimately leads to you spending
11:42less credits over like regenerations and
11:44things like that. So, both workflows are
11:47viable. I'm going to throw in the
11:49character reference sheet to work
11:51alongside our storyboard. I think we'll
11:53test that out here. And for the shot
11:55duration, I'm going to go for 15
11:57seconds. If you guys wanted to generate
12:00a shorter video, that's completely fine.
12:02All you'd want to do is go back across
12:04to Claude or the large language model
12:07that you're using and just say, "I want
12:09you to rework the prompts to make it
12:11work for a 10-second generation." And it
12:14will just tweak that framework and give
12:16you back a fresh prompt you can go ahead
12:18and drop straight into SeaArt 2. Aspect
12:21ratio is completely up to you guys. If
12:23you wanted to use this for social media
12:26content, 9 by 16 is going to be the way
12:28to go. In my case, I'm going to select
12:3116 by 9. And then for resolution, I
12:33generally go for 1080p. My general
12:36approach is that I want the best
12:38possible quality available. But if you
12:41guys did want to spend less of your
12:43credits and you're kind of just using
12:44this as a proof of concept or to test,
12:47you could go for 720p. I'm sure that's
12:49fine for most use cases. 480p is
12:52probably getting a little bit low. In my
12:55case, let's test out 1080p so we can
12:58send off this generation. Like I say,
13:01we're using the storyboard in
13:03combination with the character reference
13:04sheet and this incredibly simple prompt
13:07here all through SeaArt 2. [music] Let's
13:10take a look at what we get back.
13:24So, whilst it's not perfect, we do
13:26[music] have back this really nice
13:29dynamic sequence that has that
13:31consistent cartoon style throughout.
13:34What I want to go ahead and do now is
13:36show you guys some other examples. We
13:38have two cooking videos here which have
13:40been presented in a very cinematic way.
13:43You can see the storyboards in the top
13:46left-hand corner [music] and again, they
13:48feel nice and dynamic and there's a good
13:51level there of consistency. Also pretty
13:54impressed by some of the textural
13:56details inside of both of these videos
13:59as well. And I just wanted to try out
14:01something completely different to what
14:03you might expect to show you that this
14:06workflow is pretty versatile. You can
14:07use it for all sorts of approaches. We
14:10also have this kind of 1970s style anime
14:14version of a trio snowboarding down the
14:17side of a mountain. I really liked how
14:20this came out and how stylized it is,
14:23but it isn't without its problems. For
14:25instance, it shows who's supposed to be
14:27me with the goggles off and then I put
14:30them down and the illusion is kind of a
14:32little bit broken. The problem there is
14:34that Cinar's 2 doesn't have the context
14:36as to what the anime version of me
14:39should look like without goggles. Now to
14:41get around this, you could just remove
14:43that shot in your editing software or
14:45just regenerate the same sequence again
14:47and you may not run into that problem.
14:50But if you really want to get deep into
14:51it, you could create a character sheet
14:53where you can actually see what's the
14:55skier or the version of me in this looks
14:58like [music] without his goggles on. Now
15:00we have our Pixar-inspired
15:02animation video as well. I did really,
15:06really like this one. I actually ended
15:07up running this twice and I find this to
15:10be a really powerful workflow as well,
15:12especially when it comes to multi-shot
15:14generations like these where you have a
15:1715-second sequence which has various
15:19different cuts with different camera
15:21angles and pieces of action happening
15:23within them. If you rerun the exact same
15:26prompt with the exact same image
15:28references two or three times in your
15:30edit, you can kind of pick and choose
15:32between which moments do and don't work
15:35and build out a sequence that feels
15:37really great. [music] And then we also
15:38have some product ads here that I made
15:41for the iPhone 17 [music] Pro Max. And
15:44this is a great example here as well
15:47because all that was used for this was
15:50the storyboard rather than having to add
15:53the product reference sheet to get this
15:55level of output. So, as we take a look
15:58at these, you can see we've successfully
16:00got that very kind of premium feeling
16:02[music] CG aesthetic that we were going
16:05for. Again, it isn't perfect, but I do
16:08think this storyboard workflow is just a
16:10really quick way to test out your ideas
16:15very fast. Now, what I do briefly want
16:17to touch on is some of the downsides of
16:20this particular workflow. And the
16:22eagle-eyed among you will have noticed
16:24that it is not a frame-for-frame
16:27replication of the storyboard. By that,
16:29I mean every single frame that is in the
16:32storyboard is not going to be laid
16:34across into your SeeDance 2 generation.
16:37What you'll notice instead is that
16:38SeeDance 2 is more just using these as
16:41kind of general inspiration and trying
16:43to hit the most important story beats.
16:47But the style of the storyboard does
16:49have a very meaningful effect on your
16:51final output. So, I'd say the main
16:53positive for this workflow is speed and
16:56simplicity. You are able to kind of test
16:59out your ideas and concepts incredibly
17:02quickly, especially if you're going to
17:04be using the MD file that I've supplied.
17:07You guys haven't got to worry about any
17:08kind of complicated prompting or
17:10generating single shots and stitching
17:13them together. You can get a really
17:15great result very quickly. Now, the
17:17downside is accuracy. You aren't going
17:20to have quite as much control as using
17:23more complex AI filmmaking techniques. I
17:26think if you want to get kind of ultra
17:28specific about the AI video generations
17:31you're going to get back, let's say you
17:32have a super super specific idea for
17:36individual shots, you may well be better
17:38to generate those separately. You could
17:40potentially use this as your main
17:42workflow and just regenerate certain
17:44shots if you want to have that level of
17:46control. And I do believe that as these
17:48omni reference video models get even
17:50better over the course of time, as we
17:52get new versions of them, this workflow
17:54is going to become even more powerful.
17:57It's going to get that little bit more
17:59accurate. I'd love for you guys to test
18:00out this storyboard workflow yourself.
18:03If you get any results that you're
18:04particularly proud of, come across to my
18:06school community and post them over
18:08there so we can all take a look. Like I
18:10say, all of the prompts and the work
18:13materials that you'll need, including
18:14that all important MD file, will be over
18:17there as well. You'll find the link down
18:18below in the description. And if you
18:20have enjoyed today's video, be sure to
18:22whack a like on there for me and
18:24consider subscribing. I'll catch you
18:25guys on the next one.

About this transcript

This page contains the full transcript of How to Turn a Storyboard into a Full AI Video in Minutes by Jack Vs. AI, generated from the public captions YouTube serves with the video. The transcript has 3,379 words across 478 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.