YT YouTube2Text

YouTube2Text → Transcripts → wk07 2 vizAI iat111

wk07 2 vizAI iat111 — Transcript

by ivizlab · 6,448 words · 1,007 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00Welcome everybody to week seven. Uh
0:03we've been doing a lot with large
0:04language models and now we're going to
0:06start moving out to other kinds of
0:09systems, generally generative
0:11visualization systems like for stills
0:14and images and then we'll be moving in
0:18later ones into image understanding
0:20that's used in robots. We'll be talking
0:22about robot technology and other things
0:25always bring up ethics along the way. So
0:28a quick recap. Uh we talked a lot about
0:31how we're moving from reasoning
0:34uh with our LLMs into endogenetic
0:37multimodal where now we have web access
0:40uh it can go out and do file and data
0:42handling image and audio understanding,
0:45code execution. So this is what's
0:47currently now happening in our modern
0:51large language models.
0:53So as we've discussed a lot has changed.
0:56Now there's percents persistent context
0:59it remembers things in your prior chats.
1:01It's doing that tool orchestration that
1:04I mentioned like and web retrieval and
1:06multimedia
1:08fusion through everything it does.
1:11It changes of course now how you use it.
1:13You're no longer just asking it a
1:15question or chatting. You're in the best
1:17sense co-orchestrating
1:19which
1:20in my lab we've done a lot with using
1:23the FURC model
1:25but there's many other kinds of ways to
1:27do this level of orchestration.
1:29So just want to remind you that an LLM
1:32is no longer just one neural network but
1:35but many different smaller neural
1:38networks doing different tasks.
1:42And we also talked about the context
1:44window which is how much the context can
1:47hold and in and in general there's a
1:50very large model that model is looking
1:52at your current prompt which we call the
1:54context window. Um but you could only
1:57fit so much in it. So, we've been
1:59talking about rags or retrieval
2:01augmented generation
2:02um as a way to put in a document and
2:05then it uh chunks that into small
2:08meaningful,
2:10let's say, paragraphs or chunks. And
2:12then only when um
2:15at execution time does it put that in
2:18that smaller part in the context window.
2:21I bring up how important this is for so
2:23many types of things. Um, so you did
2:26your prompt engineering exercise and we
2:28talked about it a lot using the this
2:31technique. Uh and then we've now moved
2:34on past that into um
2:38full collaboration. So, no longer now
2:41you conduct and orchestrate like a in
2:44some human AI collaboration where the AI
2:47now is not the oracle but in fact just
2:50your assistant where you frame, explore,
2:52refine, and commit. And many other
2:55methods. So, you just don't ask for an
2:56answer, but you direct the process in
2:59many ways. Getting back to the rag and
3:01some of these techniques that I talk
3:03about
3:05from last week, there's there's now a
3:07different way of working where you're
3:09summarizing into a document, you're
3:11putting that document back in using a
3:13hostile critic, you're using uh
3:16tools that make assumptions. All
3:18throughout you're making the human
3:20decisions about what to keep and what to
3:23lose.
3:24So, that was FURC. We talked about it
3:26last week.
3:28Uh and then moving on, we're starting to
3:31get towards meaning. So, I'm going to
3:33start sliding to talking about visual
3:36systems. But to do visual systems, we
3:38would definitely have these text image
3:41pairs that I've talked about um
3:44when you when you get these
3:46uh image plus text, you get a perfect
3:49one when it's in fact exactly explaining
3:52what's going on, but in other cases you
3:54get a number and eventually you could
3:56build up a system where you're starting
3:58to get meaning that these things have
4:00more to do than other things.
4:02Um, this gives us an embedding space
4:06um, [clears throat]
4:07where in general um,
4:09there's some bit of knowledge already
4:11ready to go. And in fact
4:14most of us working with these systems
4:17are are working with some kind of
4:20embedding space already. So we build on
4:22top of other embedding spaces. So
4:25embedding space is a vector space where
4:27similar meanings are positioned close to
4:29each other in some simple way, but in
4:31the end of the day it's a latent space
4:34on a knowledge set and now we just tend
4:36to
4:37um, in different fields grab on that. So
4:40if I was to do which I am um, trying to
4:42have a very smart system on on on art
4:46and art imagery, that would be based on
4:49a different embedding space on just
4:53vision in general and all the image just
4:55regular photography. So that I can put
4:58art on top of that.
5:00Um, we've talked about vibe coding.
5:04Um,
5:05and how we're moving towards that. I'll
5:07be doing more on vibe coding, but the
5:09idea that you could have an idea
5:11[clears throat]
5:11and and as long as you can explain that
5:15um, to a system in different kinds of
5:16ways. And this way you're the final
5:18specialist step. You know, you're using
5:20the AI as the specialist step in this
5:23case coding, but there's other ideas
5:25that go with that. Um, [clears throat]
5:27so you could be using that in your in
5:30your final projects as well, but I you
5:33know, from a wider notion we are
5:36coming up to the if you have a good
5:38idea,
5:39uh, AI [clears throat] can help you
5:41create the website or the movie script
5:44or anything with that in mind.
5:47So, why visuals
5:50>> [clears throat]
5:50>> um
5:52uh we're going to move now to visual AI
5:55a bit more, and it does use a lot from
5:58what we've already learned from large
5:59language models. Surely, it is deep
6:01learning. It It is trained on billions
6:03of images rather than words. It's using
6:06pattern recognition. So, many visual
6:08models learn to create by removing
6:11noise, and we'll be talking about that
6:13in a second. But But as we get uh more
6:16into it, it's doing that plus what is in
6:19it with what is in an LLM, which is a
6:21transformer. So, [snorts]
6:23we definitely have a neural net, as you
6:25know,
6:26um
6:27>> [clears throat]
6:27>> where we have low level to high level
6:29with image understanding.
6:31Um [snorts and clears throat] generative
6:32though, to generate a visual, which is
6:34mainly what we're talking about from now
6:37on, is we go back through the system.
6:40So, we go up through the system to
6:41understand things. That would work for
6:44writing text and things, but then
6:46certain nodes are are fired
6:49uh and we're able to do that. So, what
6:52we might see in that case is each layer
6:54um this notion that once it goes up, um
6:58certain things are fired like if we're
7:00putting in uh an image of Van Gogh,
7:04uh we have some firing going on. We
7:06could put in this image as well, too.
7:09Different things fire, and then when we
7:10go back down, it will bias the picture
7:12towards those these two. So, let's just
7:15look at that in a video.
7:21>> This is how Stable Diffusion works.
7:23Training begins with billions of images.
7:26Consider this sample as representative
7:27of all our data. However, this grid does
7:30not contain all possible images that
7:32exist. In reality, they make up a tiny
7:34fraction, with the vast majority being
7:36random noise.
7:37>> So, you can imagine then that the neural
7:39net
7:40um
7:41has every type of pixel representation
7:45possible. Some of those pic pixel
7:47representations actually are images that
7:50we would understand.
7:52We have them all go to noise and come
7:54back, which I'll show you in a second.
7:56Um and this
7:58>> our data within this massive image
8:00space, we can then define clusters based
8:02on keywords. Take for example dancer or
8:04Van Gogh. These are two disparate ideas,
8:07but by drawing connections between these
8:08nodes, we can conceptually define a
8:10greater space of all normal-looking
8:12images. If we then choose a spot on this
8:14grid that is not defined and as such
8:16random noise and provide a prompt that
8:18connects these terms, we can
8:20progressively move from that spot
8:22towards this connection between nodes
8:23and arrive at a new image that is not
8:25part of our original data set.
8:29>> So,
8:30um
8:31I'd like to mention again, not only are
8:34we then able to do what I was just
8:36mentioning that there are certain things
8:38fired to get us the dancer and then Van
8:40Gogh and a way to then put those
8:43together
8:44to give us even video and image stills
8:48that are a new. But, it's also a meaning
8:50space that they are clustered in in
8:53areas together. So, we're both getting
8:55meaning and the the gen viz tools out of
9:00this work.
9:03Um
9:03>> [clears throat]
9:03>> so, let's talk about this diffusion
9:05process.
9:06Um we start with a real image from a
9:08data set and gradually add noise to it.
9:12Um
9:13uh and then we're able to go backwards
9:16through that. So, um diffusion, which is
9:19the main technique used in the image
9:22side rather than transformers in words,
9:26turns an image into noise in steps, uses
9:29the knowledge to make a new image from
9:31noise. So, by um slowly moving to a
9:37noise space
9:39um um
9:39we're using a particular technique
9:42where we can then go we can go backwards
9:45through that process. This was an
9:47amazing in a number of ways because we
9:49can move everything to noise and then
9:51whether it's Van Gogh or the dancer, you
9:54can see how we could build that up from
9:56the noise space in the way that we just
9:58showed, but it also means that we no
10:00longer have to kind of save uh
10:04the images in the data set. The data set
10:07is much smaller because we're only
10:09saving um
10:11the parameters of of its noise. So
10:14again, we call this a latent or
10:16embedding space where we only have we
10:18can move through the world that we just
10:21saw um in in this image here by moving
10:26through the latent space and these don't
10:28have to be now the the original images.
10:31They are uh just the parameterized of
10:34how you would get out of the noise to
10:36get to Van Gogh. So this is really uh
10:39it changed the world of visual
10:42generation amazingly.
10:46So
10:47um LLMs then are mainly for text, right?
10:50Um
10:52and the idea then is it predict it
10:55predicts the next token or word, but
10:58what we're now seeing is we're able to
11:00do a little bit of both. So we use the
11:03diffusion
11:04um to and refine from noise, but the
11:07hybrid is we could use the same token
11:11system by cutting up the image and then
11:15using the diffusion throughout. So
11:17originally everything was just diffusion
11:20for image space and transformers were
11:23used more with this kind of next token
11:26prediction in LLMs, but we've now
11:29speeded that up greatly by realizing
11:32that diffusion could be uh smaller parts
11:35of an image and then when we're
11:37predicting the next diffusion area. So,
11:41it's a way to put the two of them
11:43together. And as I said, the meaning
11:45builds up the meaning builds up this way
11:47and we're starting to really understand
11:49things. So, learning that a cat, you
11:52know, uh about a cat, you know, you
11:54could be training on millions of images
11:57of these image caption pairs. Some of
12:00them are cats, cat sleeping. Um and then
12:03you could see like a a golden retriever
12:05playing and we have a we have a image
12:07and a and a caption of that. And by that
12:09we're able to learn all these different
12:11kinds of pieces and then their nearness.
12:16So, diffusion then is this process of
12:19going to
12:20pure noise, learning the knowledge to
12:23predict and remove the noise, repeating
12:26that many, many steps. And as you start
12:28using these systems, you'll see that you
12:31you play with this this steps
12:33controller. Um
12:35and then we can go from that and
12:37gradually come back the other way.
12:42Um so, we call this high-dimensional
12:45space where text and images share
12:47meaning and embedding space like I just
12:49talked about. In general, um we do that
12:52with mathematical representations called
12:54vectors. We no longer have to put the
12:57images in there and this way we can put
12:59all that together.
13:02Um so, we tend to now call these vision
13:04transformers that are turning an image
13:06into tokens. So, that's the new idea
13:09that we're able to put these both
13:11together now. So, we cut up the image
13:13into grids. That's where we're still
13:15doing diffusion, but we turn each patch
13:18into now a visual token and then we can
13:21use the large language model type
13:23transformer system to do that. So, once
13:26an idea an image is in a token, the same
13:29transformer system that we use for text,
13:32we can apply to images.
13:38Um
13:41So, that there we then have that, you
13:43know, we're trying to make a cat
13:44astronaut. The text encoder, which is
13:47increasingly like a full large language
13:49model, can convert that prompt into
13:51embedded things that has the now the
13:53concept of a cat, the concept of an
13:57astronaut, and using diffusion as the
13:59last step can put that together. So, um
14:03this is now how modern
14:06visual systems work.
14:09Understanding and generating uh
14:12our way of merging this multimedia data.
14:14And we have all these particular systems
14:16out there to do it. Nano banana is the
14:18Gemini system. They now have a new one
14:21called Argon that I'll be showing you
14:22some examples of.
14:24Uh so, both the text encoder,
14:26um
14:27and then decoding using the diffusion
14:29model is one of the ways we do it. So,
14:32images become tokens or patches, uh so
14:35areas of picture.
14:37And we use both the next token technique
14:39and the diffusion technique to put them
14:42both together. It's still doing deep
14:44learning, large data sets, and
14:46transformers now. So, they're really
14:47coming together. So, let's look at at
14:50examples. So, this would be just
14:52blending two images. So, you could just
14:54say that out of the text and all of a
14:56sudden you could put that together.
14:57That could be an image of you
14:59>> [clears throat]
15:00>> um and your dog um
15:06Um people can be using it in interior
15:09design or designing houses. So, add a
15:12green sofa with a modern style beneath
15:14the window, add a Persian rug.
15:18Uh how about a mid-century table? And
15:20you can see now how you can quickly put
15:22all this together.
15:27We could also visualize dense data. So,
15:30in this case, this is um
15:32uh wildfire data. I have the actual data
15:35set if you want to see it. And not only
15:38can we look at it in a chart or in some
15:40ways, but we could really put in the
15:42notion of the Earth and the globe and
15:44three-dimensionality and really thinking
15:46about that way. This could be really
15:48amazing for some of your final projects.
15:54So, uh I already mentioned it design.
15:57Sorry to go backwards there.
15:59Um and very, very recently, we're now
16:02moving into three-dimensionality and
16:04interaction. So, similarly, the the same
16:07system now is able to let's say with
16:10with making your product and product
16:12[clears throat] design, showing off um
16:16or in gaming, showing off um a
16:18three-dimensional tool, and giving you
16:21control. So, this control to change the
16:23lighting and the way it looks, and you
16:25can see I'm I'm fooling [clears throat]
16:27with the buttons here. Um
16:29So, this is what the level is now
16:31possible with these systems. So, I think
16:34I'm thinking about you now and how you
16:36might want to use it in uh the tools
16:39that you might make.
16:42Um
16:43>> [clears throat]
16:44>> somehow I left out my my graphic here,
16:47which is um doing uh interactive
16:50[clears throat] visualization. So, from
16:52one picture in a schematic, I could get
16:55uh a a web page on financial data or in
16:58this case on everything about this
17:01Victorian cal- carriage, or with the med
17:04school, we're looking at uh
17:06visualizing anatomy and biology. So,
17:11think about all these possibilities,
17:12come back to the slide, and what you
17:14might think about you might even want to
17:17do for your final project. Some of these
17:19are using this Gemini Argon system,
17:21which is a new system. Fortunately, you
17:23would need a paid Gemini account. Um but
17:27there are other systems out there. I do
17:29have a paid Gemini account, so there's a
17:31team that wants to use something like
17:33that and doesn't have it, they can be
17:35talking to me.
17:41So,
17:42um
17:43the big shift then is you used to just
17:46whatever it just made something and you
17:48had to take it or leave it. And the only
17:50thing you do is like run it again. But
17:52now you can do very surgical in painting
17:55and direct control over everything.
17:59Um
17:59and uh with you could have a
18:04you could use uh
18:06you used to be able to just use text
18:07prompts, but now it could be
18:09multi-references, brand guides, data,
18:12anything you want to explain the
18:15visualization you want. We have more
18:17consistency with faces uh and it used to
18:20be terrible at at doing text and now it
18:23can do text.
18:24Um let's look at
18:26>> Great.
18:27That's why you can say, "Make this mug
18:29look like it's on a beach at sunset."
18:31And it nails the vibe in one go. Inside
18:34the model lives a huge team of mini
18:36networks or experts. One expert knows
18:38faces, another knows lighting, another
18:41knows text. When you give a prompt, a
18:43tiny manager quickly assigns each part
18:46of the job to the right experts. So you
18:48get high-quality results fast without
18:50overloading your computer. It's like
18:52ordering different dishes and having a
18:54kitchen brigade cook them in parallel.
18:55>> We'll talk more about mixture of
18:56experts.
18:57>> does it actually paint a picture? It
18:59starts with pure TV static noise, then
19:01erases the noise step by step until an
19:04image appears guided by your
19:06instructions. Picture a Polaroid
19:08developing in reverse. First you you
19:10blur, then shapes, then a crisp photo.
19:13This noise-to-art process is called
19:15diffusion, and it's why the outputs look
19:17so clean. Subject consistency. Take one
19:20selfie and spawn dozens of matching
19:22product shots. Multi-image fusion. Drop
19:26your logo and a cafe photo and get
19:28realistic signage.
19:29Conversational tweaks. Nice. Now, make
19:32the cup red. Instant update.
19:35Every image gets an invisible watermark
19:37baked into the pixels called SynthID. It
19:40survives cropping or filters and tells
19:42future software, "Yep, this came from
19:45Google AI." Helping fight fakes. So,
19:48Nano Banana isn't magic. It's smart
19:50architecture. If you learned something
19:52new, hit like.
19:53>> So, we'll talking be a lot more about
19:55about watermarking and provenance um
19:58a little bit later.
20:02I have one more that I want to show you
20:03here.
20:06>> Hey everyone. Welcome back to Code
20:08Viral. Today, we're unpeeling net
20:10>> No, that's the wrong one. I'm sorry.
20:13Uh
20:14I like maybe I'll just leave out this
20:16other one for the time being. Let's go
20:18back to our
20:20work. So, getting full control now.
20:24Um
20:26So, this is now extending uh I'm going
20:28to move a little bit more to the video
20:30side. We were kind of looking a little
20:32bit more stills there. So, this is
20:34extending those tokens into space and
20:37time.
20:38Uh
20:39there's a lot of issues there like
20:40temporal consistency. This has been
20:42terrible. You weren't able to make a
20:45character from scene to scene that
20:47looked the same. We're way past that
20:48now. You weren't able to have it move.
20:52Well, now we can. Uh we now are able to
20:55do lip sync and many many other things.
20:58So, now we have a video autoencoder that
21:00compresses the clip into small latent
21:02space that we talked about. And then we
21:05cut that into patches, but now there's
21:07space-time patches rather than just
21:09image patches.
21:11And these are some of the systems out
21:13there for that. Uh video though is very
21:16expensive. If you just count the tokens,
21:18so to speak, there's a lot more to do.
21:20So self-attention tries to compare every
21:22token with every other one, so that cost
21:24grows
21:26um
21:26with the square of the token count. So
21:28it's very It's a huge number. This is
21:31why
21:32uh video can can be very slow to
21:35generate on your home computer. Um By
21:37the way, I've been showing a lot of um
21:39tools I've been talking about are
21:41cloud-based tools that you could use,
21:43often have to pay for or you get a
21:45limited amount, but many professionals,
21:47including my lab, and we're doing it for
21:49a number of reasons, do not use
21:51cloud-based tools. We use open tools. So
21:53if you have a gaming computer, you can
21:56do things You can do things using that,
21:59and then it's free, unlimited. You can
22:01use it all night long. Uh and that's
22:03what many professionals are doing. They
22:05can't deal with the cost, the privacy,
22:08the energy uh waste that these other
22:12systems
22:13um use. However, if you really
22:16interested, and you don't have a choice,
22:17you can, you know, uh
22:20use one of these cloud
22:21uh
22:23commercial tools as well.
22:25So we definitely have character
22:26consistency. We have so much that we're
22:28doing. So how do you use these tools?
22:30Well, you could just use text and write
22:32that in, but now you could use reference
22:34images to keep the character and the
22:36look consistent. Let's take a look at
22:38that.
22:44>> Run an image has dropped, and it's by
22:45far the best character animation and
22:47replacement AI model on the market.
22:50>> Take a look at this video that I made.
22:52>> Friends, listen, what you're seeing
22:53right now is crazy.
22:55>> We are crossing a line in human history
22:57where the boundary between reality and
23:00fiction is vanishing. this is not
23:02>> Hollywood billion-dollar high production
23:05studio. This is open source. This is
23:08from your computer click of a button and
23:09most people will not be able to even
23:11tell the difference now. So, be safe out
23:14there. Question what you see. Verify
23:16what you believe.
23:17>> and follow and stay ahead of the curve.
23:19>> And of course, we'll be talking much
23:21more next week uh cuz this is literally
23:23week one on the issues of deep fakes.
23:27I've been on the CBC talking about it as
23:29well, too. From everything from deep
23:31fake wildfires that looks like there's a
23:33wildfire in your area to to every other
23:37kind of deep fake. I have a lot to show
23:39you next week on that and tools to
23:41combat it. Um Another way you would do
23:44it is first frame last frame is a very
23:46big one. Let me just show you that.
23:47That's where you get one frame and
23:49another frame inset and then use a text
23:51prompt to say what to do in between.
23:53>> job. So, that is really really
23:55impressive. So, our second use case for
23:57first frame and last frame is product
24:00shots. So, I have two quick shots here.
24:03We have this close-up of a soda and then
24:06a wide shot of a soda and you can
24:07imagine we would put text inside of that
24:10blank area. Well, we put this inside of
24:13Google VEO and we got this video here.
24:17>> So, ways to really just take two images
24:20>> Obviously, it's not a crazy dynamic
24:22movement, but that's not what we're
24:23looking for. We're looking for something
24:25that can be that hero shot inside of our
24:27advertisement.
24:27>> be so expensive.
24:29>> darn good. Now, the reason why I did it
24:31with the text being into that video is
24:34because AI models tend to not do a great
24:37job when it comes to text being on the
24:40screen. It's almost always better to
24:42just composite in that text inside of
24:44your video editing application or maybe
24:46Adobe After Effects just to kind of
24:48bring everything together. But, this
24:50helps save a lot of time in the
24:52texturing and modeling environment
24:53because you wouldn't have to bring this
24:55into Cinema 4D or Blender to pull this
24:58off. For our third example, I want to
24:59take a look at a really cool VFX
25:01process. So, think of that book
25:03Animorphs. Remember the covers where the
25:05character would like change into a
25:06different character? Well, I want to
25:07show you how you can pull that off
25:09utilizing AI. So, AI systems are really
25:12good for transitioning between two
25:14different states and that extends into
25:16organic examples like this one here. So,
25:19we have this image of this woman. Let's
25:21say we want to change her into an alien.
25:24Well, using Nano Banana Pro, we can just
25:26upload the image and then prompt to
25:28change her into the alien character and
25:30you can see it looks pretty good. Like
25:32the lighting matches pretty well. It
25:34definitely is pulling from the both the
25:36fluorescent lighting and the
25:38>> So, these video these URLs are around.
25:41So, if you want to look at them more,
25:43please do. So, we now have full camera
25:46as well.
25:48Um
25:48there's a very famous artist also I'll
25:50show you next week who does this these
25:52frame things where she uses a still
25:56tool, I think Midjourney, to get a
25:58really good looks and then has them
26:00dance or animate with this first last
26:03frame or number you know, or frames in
26:06between. We now have full audio as well,
26:09too and a lot of people I'm using using
26:12AI music and now just recently using um
26:17being able to lip sync uh uh
26:19uh a a character that doesn't exist to
26:22actually sing their songs. So, I would
26:24tell you we used to not be able to have
26:27consistent looking characters
26:29shot-to-shot. Now, we have that. Then,
26:31we got multiple characters shot-to-shot.
26:34We could not do lip sync at all. Then,
26:37we were able to do lip sync on one
26:39character. Now, we could do lip sync on
26:41multiple characters. Now, we're able to
26:43do um
26:45a motive voice lip sync and
26:49uh
26:49>> [clears throat]
26:49>> um
26:50full director
26:52controls telling a virtual actor what to
26:54do, what to say. So, all these things
26:57are coming up pretty quickly.
26:59Let's show you music videos.
27:05>> I woke up early and you were still
27:07asleep. Sunlight on your face, I forgot
27:10how to leave.
27:12Made [music] you some coffee, but you
27:14slept right through it.
27:15>> There is one thing inside Sundance 2.0
27:18that most people building AI music
27:20videos
27:21>> This one explains exactly
27:26how to do it, the prompt you give. So,
27:28you can see it's it's this man in
27:30mid-20s standing outside an Italian
27:32villa, 3/4 angle camera, and on and on
27:35they're going to explain exactly how to
27:37do it, and then putting your song in it,
27:40and then understanding that. So, it kind
27:41of goes on and on if you're interested
27:43in in this area. Again, huge ethical
27:46issues that we will talk about next week
27:49along with some of the details of how
27:52this works.
27:53>> They say I'm crazy.
27:55>> Obviously, it doesn't
27:56>> In a world that's half asleep, [music]
27:58they say I'm crazy.
28:01>> It does not have to look realistic, so
28:03[laughter] you can be do other kinds of
28:04things. And this now doing multiple
28:08lip sync is is quite new. Um
28:15So, all these possibilities surely video
28:17to video. You have a video of yourself,
28:19like we already showed, um and then you
28:22could make it look like someone else or
28:24do very So, uh besides the deep fake and
28:28other problems here, some of this is
28:30just a new form of personal video
28:33making. If you have an amazing idea for
28:35a movie or short sequence, people are
28:39coming up with uh
28:40>> [clears throat]
28:40>> uh new uh movie making approaches that
28:43we'll talk a little bit more about next
28:45week. So, we definitely have
28:47photorealism, strong prompts, uh
28:50conversational editing now, you know,
28:53getting be dealing with text. We used to
28:55only be able to do 5 to 15 seconds, and
28:57we're now moving past that. Um so, these
29:00are some of the tools that are out there
29:02that you might want to uh play with.
29:05They usually give you a little bit of
29:07free usage. Um
29:09still everything is not perfect. It used
29:11to be hands and text. That's not the
29:14problem now. But, there's many other
29:16visual tells that just don't work well.
29:19Uh physical accuracy of liquid and
29:21clothing and hair, you know, people
29:23appreciate that
29:25because we can tell what's real and what
29:27isn't. I'm asked about that all the
29:29time. But, it's not it's just getting so
29:32much better that at some point
29:35it's going to be really hard to tell.
29:37And that's where, as you saw with the
29:39Google system and others, this notion of
29:41watermarking and provenance come in.
29:44These are very big techniques that are
29:45happening very quickly where um
29:48>> [clears throat]
29:48>> you'll be able to tell. Now, is it wrong
29:51to make an AI video? Um
29:53it could be, depending on what you're
29:55doing. But, one thing that I think is
29:57wrong is not labeling when something's
30:00AI and not. And we'll talk about this
30:02more next week. For some reason,
30:03Facebook has just decided, as other
30:06systems have, that um
30:08they'll just you can literally lie with
30:10things. And we're seeing in the US uh
30:13uh elections
30:14uh kind of the full-throat version of
30:16this where you have you make a video of
30:18your opponent saying something
30:20disgusting, and it's not real. The other
30:22way is it really was you saying
30:24something disgusting, and you say it's
30:26AI. So, we'll talk more about about that
30:29next week.
30:30Um there are definitely techniques that
30:32you could use to, just like we did with
30:36the first two assignments, about how you
30:38actually uh could um
30:41do the best version of making uh stills
30:45and videos so it isn't AI slop and it's
30:47conducting to what you want. We're
30:50seeing multi-shot video, native audio,
30:53complete 3D worlds. So we're moving
30:55quite quick into this and it's not just
30:59models of video. At some point it's the
31:01model of of worlds, understanding
31:04physics and many other things. So that's
31:06the last part I'm about to talk about.
31:10So okay, so current state I think I've
31:12explained it to you.
31:15So we can do all this kind of it's
31:18really affecting these kinds of areas
31:20and markets and people really switching
31:23over for in product visualization and
31:25marketing materials and medical imaging
31:28enhancing and a number of other areas.
31:31Some of it very amazing, some of it
31:33rather dangerous.
31:35So it's it's going from an interesting
31:37tool to a
31:38to a full production
31:41technique where people having real
31:43budgets and using this in a number of
31:45ways.
31:48Here's some notions about how to do it
31:49right. So maybe you can look at that on
31:51your own still scripting and
31:52storyboarding, thinking through every
31:54piece. I teach a computer graphics class
31:57where we're using some of the techniques
31:59that you would use at Pixar to make a
32:01good animated movie with with these
32:03kinds of tools. But
32:05the other side of it is now
32:08almost everyone can do this. So if you
32:10have a great idea, but maybe you don't
32:12have that animation artistic experience,
32:14but there's ways to do that. So students
32:17can illustrate their projects more,
32:19educators can generate materials. Well,
32:22again, we'll talk about the goods and
32:23bads. The one of the things seems to be
32:26happening now and some of this red is is
32:29really coming directly from me, which is
32:31domain domain experts, people who are
32:33good at this to begin with. It is their
32:34job, whether that's programming or
32:37making marketing videos,
32:39are getting a boost from these tools
32:41with speed and quality and control like
32:43never before. Um, also beginners get to
32:46work with this smart assistant that can
32:48do it, but there's just huge issues with
32:52with is it going to take jobs away, the
32:55deep fakes
32:56to the point where we're losing truth,
32:58AI slop, um, we have papers on AI slag
33:02and energy of these systems. Slag means,
33:05you know, all the attempts you take
33:07um, or all the prompts you make that
33:10don't really matter, but still have to
33:12get saved somewhere and the energy
33:14issues there. So, we talked a little bit
33:16about all these different areas. You can
33:17look through it a little bit more.
33:19Um, so yes, I'll I'll bring up next week
33:22all these areas about deep fakes and
33:24copyright and authentic
33:27Sorry, authenticity and the watermarking
33:30as a way to dealing with it and the
33:32problems there.
33:33Um,
33:34so if it's becoming indistinguishable,
33:37we're definitely into misinformation and
33:39other issues. Again, I'll I'll get into
33:41that more next week, but for you, I
33:44think the idea is for jobs and things,
33:47you this this field is changing
33:48incredibly fast and you should be
33:51thinking about about all these issues.
33:53We'll talk more about it next week.
33:56So, I have some questions if you want to
33:58put them on the forum. What what what is
34:00this good for? And I'll just leave it at
34:02that. Um, I did want to move on to this
34:04other area quickly about video. So,
34:07we're starting um,
34:09uh,
34:09>> [clears throat]
34:10>> to move away from large language models
34:13because with video generators and video
34:15training now, um, that world models are
34:18starting to show up. So, world model is
34:20an AI system that learns from an
34:23internal representation and can
34:25understand physical dynamics,
34:26environmental constraints, cause and
34:28effect, um, and it allows it to predict,
34:32uh, uh,
34:33um, and reason about future states. So,
34:35you've definitely seen some of this
34:37with, um, the self-driving cars
34:40um,
34:41lecture that I showed you. Um, so, this
34:44is the new area that people are looking
34:46into, you know, that, uh, the head of,
34:50uh, of
34:50AI research at at Meta Facebook, he left
34:54and he's very big on saying LLMs will
34:57never be AGI, but this world model is
35:00the way to deal with it. So, it's
35:01learning how the world works, not just
35:03how it looks. It's got to understand by
35:06watching phys- you know,
35:08physical real world. You understand how,
35:12you know, how someone comes off a curb
35:14and if they come off the curb
35:15wrong, they can fall and what happens to
35:17their phone and how liquids pour. So, a
35:21lot of this cause and effect, you push a
35:23cup, it slides, it drops, glass, it
35:25breaks. This is what we're starting to
35:27build. So, the goal then is a world
35:29model, which is an eternal eternal
35:31simulator of real rules, not just its
35:34appearance. That's the direction we're
35:36moving
35:37for, uh, to do that
35:40every company is trying to get video
35:43training down. So, I'm not talking so
35:45much about about video generation, but
35:48video understanding that we were looking
35:51at within the self-driving car world.
35:53So, the trouble is the computer's just
35:55really expensive to actually have these
35:58video encoders and these new kinds of,
36:02uh,
36:03transformer systems.
36:06The just the data alone is is huge. Just
36:10so much video. Uh, how much video a
36:12robot needs for first-person data and,
36:16uh, you know, now we're having
36:17multi-camera, so you can see it from all
36:19different views. Video needs to be
36:22labeled. It was pretty, you know, it was
36:25an issue, but much easier to label
36:28images. And uh, I'm still with my lab
36:31working on semantic labeling cuz some of
36:34the labels bring in bias and cultural
36:36flattening. So, we're still putting up
36:38labels for stills, but how do you
36:41actually label time-based work? This
36:44literally hierarchical labels just
36:46trying to label, for instance,
36:49a TV
36:50presentation of a basketball game is
36:53pretty wild with all the different
36:54things that happen.
36:56Um
36:57So, in conclusion, video can teach us
37:00about physics and causality, and it
37:03could help for systems like robotic
37:06self-traffic cars, simulation, and it's
37:08a step towards AI that understands
37:11reality.
37:12Um there's there's a lot to work on
37:14here,
37:16uh but world waters are are definitely
37:19in their own category now in 2026, and I
37:22think you can expect rapid change. So,
37:25very significant companies working on
37:28this. Again, people leaving Meta to to
37:31and getting billions of dollars to to
37:35now work on these world model-based
37:37systems.
37:39The idea is to move to artificial
37:41general intelligence, AIs that can
37:43understand, learn, apply, and so on.
37:47Um do current models understand the
37:49world? There are plenty people telling
37:50you So, I hear it's about Yanne LeCun
37:52leaving Meta. There's plenty people
37:55telling you that LLMs alone are just
37:58fine, and we're going to get to AGI.
37:59Surely, Elon Musk and others say this
38:01all the time. I'm of the belief that
38:04LLMs could never get us to that state.
38:07They're very good at the the kind of the
38:09front end, and I don't think the front
38:10end will go away. That LLMs are really
38:13good at conversation and understanding a
38:14bit, but at some point the full brain,
38:17like what you're thinking, what you're
38:18strategizing, what to do next, it really
38:21LLMs really don't do that too well.
38:24There's um again for next week there's
38:26some more and more my opinion is do we
38:29need AGI or as Trump says SI super
38:32intelligence or our current systems good
38:35enough and are these
38:37bigger systems
38:39too too dangerous and seem to more about
38:42like, you know, war and other kinds of
38:44situations where the models the simpler
38:47models we have might just be fine for us
38:51doing research helping in health and
38:54medicine and other advances. It's a
38:57ongoing current debate. Um okay, so I've
39:01now showed you
39:03the presentation for this week on moving
39:07away from large language models. We're
39:09still doing work in that space. How
39:10that's now combined with these diffusion
39:13systems where we're we're we're using
39:16kind of both to understand and generate
39:20video
39:21and text and we'll see more about this
39:24and more demos next week. One of the
39:27things you might want to do on Discord
39:29is if you find really good examples
39:32of how people are using these generation
39:35systems maybe put up a link. Thank you
39:37very much.

About this transcript

This page contains the full transcript of wk07 2 vizAI iat111 by ivizlab, generated from the public captions YouTube serves with the video. The transcript has 6,448 words across 1,007 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.