YT YouTube2Text

YouTube2Text → Transcripts → YouTube transcript (pwQ0l4hFCVI)

YouTube transcript (pwQ0l4hFCVI) — Transcript

12,541 words · 2,159 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:05Today we are going to move to um
0:07a sequence of um lectures on large
0:10language models. Not many, like probably
0:12three, four. And in between we also have
0:15to talk about reinforcement learning a
0:16little bit because otherwise we cannot
0:17talk about some of the modern reasoning
0:20models without understanding a little
0:22bit about reinforcement learning.
0:24So that's pretty much the rest of the
0:25lecture except uh
0:26two guest lectures. I think next
0:29um the upcoming Monday we'll have Simran
0:32to give us a lecture on system ML which
0:34is also quite about um
0:36transformers. Like how do you optimize
0:38the efficiency of transformers on a GPU
0:40so on and so forth. Um
0:42Okay, sounds good. So large language
0:45models, I guess you always probably all
0:46of you have heard of this word
0:49uh large language models. This is um um
0:52precisely I think in some sense this is
0:54all starting from GPT-3 when OpenAI
0:57released GPT-3. Um it has a um it used
1:00this new paradigm which uh used
1:02autoregressive
1:03large language models. Um so um so
1:06basically we're going to focus on that,
1:07you know, autoregressive large language
1:09models. We're going to skip some of the
1:11earlier versions of transformer which
1:12are um not autoregressive. Um so
1:17and um and an autoregressive basically
1:20means you generate tokens one by one and
1:22each of the new tokens depends on the
1:23previous tokens and you do this um
1:25um and you model probability in this
1:27kind of like a autoregressive way. I'm
1:29going to talk more about details. In
1:30some sense you model your language
1:32distribution by using a
1:34um um a sequentially sequential model
1:37one by one in some sense.
1:39Um anyway, so um let's um I think the
1:41point
1:43Our goal is to talk about some details.
1:44So I'll fill in more high-level stuff as
1:46we talk about the details.
1:48So the first thing we're going to talk
1:49about is tokenization.
1:53This is um you know, just uh
1:56to prepare us for to talk about the real
1:58meat in some sense.
1:59So, what does tokenization mean? It
2:01means that you need to
2:02you know, these transformers, they are
2:04numerical you know, architecture models,
2:06right? So, they take in
2:08uh in some sense numerical numbers,
2:10right? So, um but uh you have text. And
2:12how do you turn text into some kind of
2:14things that
2:15the models can understand? So, uh the
2:18first step is to tokenize. Tokenize
2:20means that you need to decide you what
2:22is the smallest unit uh of the input and
2:25you give to the large language model.
2:27And often it's not a
2:28a word, right? And the most natural
2:30thing you can think of is maybe the
2:32smallest unit is a character or a word.
2:34If you use character level tokenization,
2:36then the issue is that often you have so
2:38many characters, right? You you need so
2:40many tokens to so many steps.
2:42Um and um
2:44uh if you use word level uh
2:45tokenization, then often the issue is
2:47that you don't How do you deal with this
2:49long word, which are very rare? For
2:50example, you have a word called say
2:52internationalization,
2:55right? So, you have If you treat this
2:57word as a single word, then maybe it's
2:58too rare and you don't leverage your
3:01understanding about other words, right?
3:03So, if you treat internationalization as
3:04one word, then
3:06it is different from internationalize.
3:08It It's different from like a nation,
3:10right? And And they are all completely
3:12different kind of things and you cannot
3:14leverage
3:16um that you For example, if you
3:17understand internationalize, then you
3:19should probably understand
3:20internationalization to some degree
3:22without even seeing more data. And uh
3:24so, um so, that's why people don't
3:26always use um subword tokenization uh
3:29don't use word tokenization. Maybe
3:30another Okay, maybe let me give you
3:32another example, right? For example, if
3:33you have use this
3:34I guess in my notes there is this word
3:36LM
3:38ification.
3:39I actually don't know whether this is a
3:40real word, you know, like I think it's a
3:41new word that is completely new word.
3:43Nobody have used it before 2020
3:46uh 2020, I guess, right? So, uh even
3:48even right now it's not that like of
3:50like common.
3:52And if you have to treat this word as a
3:55unit, then you you have to see this word
3:58enough times to understand what it
3:59means.
4:00Right? However, if you treat it as sub
4:02words, then maybe you have a a token
4:04called LM. You have a token which is
4:07you have a two units. You know, this is
4:08a token, this is a token. So, LM is a
4:10token, ification is a token. Then in
4:13some sense, you have seen LMs many many
4:14times, you already understand it. And
4:16then
4:18ification has a special meaning in
4:20English, and then you combine these two
4:22understanding, you understand the word.
4:24So, that's much more efficient
4:26understanding of the word than just the
4:29treating this whole thing as a single
4:30unit.
4:32So,
4:33and then and this happens more often in,
4:35for example, biology, where you have
4:37this very long word. Or maybe in German,
4:39you have very long word, but actually
4:40they are fundamentally many combinations
4:42of many sub words. And uh and uh and uh
4:45using sub word tokenization allows you
4:47to understand each of the sub word, and
4:49and then you can understand the
4:50combination of the those sub words
4:52as a as a as a whole
4:54very efficiently. So, yeah, so basically
4:57tokenization, you know, just means that
4:58you have a predefined vocabulary.
5:02Vocabulary.
5:04And this vocabulary is a list of like a
5:05sub words, right? So, for example, you
5:07can have, you know, many many of this
5:08like maybe A is a sub word, you know, is
5:11the one of the token, you know,
5:12I think maybe like a dog is a token.
5:16Pretty much all common words these days
5:18is a token.
5:19And actually, if you look at the the the
5:21actually, you can play with the
5:22tokenization online if you search OpenAI
5:24Tokenizer. OpenAI give you a playground
5:26where you can see how the texts are
5:28tokenized. And and you'll you'll find
5:31that actually blank dog is also token.
5:35So,
5:36uh so, the blank is actually merged with
5:37dog as a token. And uh you know, and you
5:40have so many of this. I typically this
5:41kind of like a tokenization right now,
5:44um I think for the open source models,
5:45we know the number of tokens uh they
5:47have. I think it's, you know, often on
5:49the order of like 10 to the five, like
5:51100k or at least. Um I think QN QN 3.5
5:55um this is the newest version of
5:56open-source models, which are pretty
5:57good. I think they have like 250 tokens.
6:01Um uh I think 250 probably is mostly
6:03because, you know, I think 250 is is
6:05somehow you can represent this token
6:07with like a
6:08maybe
6:09eight eight bytes in some sense, like
6:11like a
6:12it also depends on how many bits you can
6:14present the tokens. Um so uh anyway, so
6:17you have this vocabulary, and then you
6:19just uh uh given a long uh string of
6:21text, you just uh um
6:23uh take all the subwords that are
6:24tokens, and you uh turn it into a
6:26sequence of like a segments, right? For
6:28example, if you say if I have a a dog
6:31runs, suppose this is the text you see,
6:33and then you you'll say this is a token.
6:36So, this is one part. And then I think
6:38blank dog is a token.
6:41And blank runs is also token.
6:43Uh and uh
6:45and comma, I think is a token by itself.
6:48So, and every token has an ID, right?
6:50So, let's say A is probably the ID
6:52number one, dog is maybe the ID
6:54number like 356, whatever, and and run
6:57is a some ID.
6:58Maybe I don't know, like a
7:00maybe 250.
7:01I'm just writing some random numbers.
7:03So, every everything has an ID. So,
7:04basically, you turn every text into a
7:06sequence of IDs, you know, each ID
7:08corresponds to one token.
7:11Um and um
7:12and I think So, in this case, you know,
7:14most of these words are actually one
7:15token. I think uh I just checked it
7:17recently. If you have the word
7:19happiness,
7:21I think the tokenization will be
7:23unhappiness, let's say.
7:25I think this part is a token.
7:27H is a token, and
7:29a p i n e s s is a token
7:31uh
7:32for some reason. Um and maybe another
7:34example is this Oh, I I already erased
7:37it. The I L M imbification.
7:39And this is another examples [laughter]
7:41so
7:42I come uh we come up with, you know,
7:44just a I don't know actually what these
7:45words means. If you have a blast
7:47oma, I think this part is a token, this
7:49part is a token, and this part is token.
7:51I think this is just because for this
7:52biological biology words, you know,
7:54biomedical words, you know, actually
7:56every part of the word has some special
7:58meaning and uh uh
7:59uh so, you tokenize them into um tokens.
8:02And uh and what is the exact vocabulary?
8:04It depends on the model.
8:06Uh um so, basically the the algorithm
8:09that turns the text into this uh tokens
8:11is called a tokenizer.
8:13Um and uh and every model has a
8:14different tokenizer. Not not everyone
8:16has a different one. Some Some of them
8:18shared the same tokenizers, but
8:20generally it's specific to the tokenizer
8:22to to the model.
8:23Uh and different companies have
8:25different um proprietary tokenization
8:27approach. Uh you probably heard of that
8:30uh recently, I think Cloud Code uh
8:32Cloud, they updated their tokenizer so
8:34that uh
8:35uh I think they make the tokenizers more
8:36granular uh to some degree, so that the
8:38same amount of text will be tokenized
8:40into more tokens. So, before it was like
8:435,000 tokens, now it's 1.5,000 tokens.
8:45Um you know, I didn't verify myself, but
8:47I read some news about this. Uh assuming
8:49they are true, uh that means that you
8:51you spend more tokens, you know, you you
8:52you pay more to Cloud Code.
8:55>> [laughter]
8:55>> With the same amount of text. Um anyway,
8:58so um cool.
9:01And there are some details about how to
9:03come up with the vocabulary and how to
9:05kind of like uh uh um run this algorithm
9:08to tokenize. Um the the keyword is
9:11called BPE. Um
9:13BPE
9:15uh tokenizer, byte um pair um uh
9:18encoding. So, I'm not going to the
9:20details because this is just a very
9:21small quick uh uh setup. Um it's not
9:24that important. Um but roughly speaking,
9:26the BPE uh tokenization idea is that you
9:29try to In some sense, you just try to
9:30find frequent
9:32subwords that are most frequent, right?
9:34And uh and you use them them in a
9:36vocabulary. But how do you decide which
9:38one is more frequent and and not? I
9:39think that there's a sequential
9:40algorithm. You first go through maybe
9:42like the the
9:44the smallest kind of like a
9:47combination of like you go through all
9:49the characters, you know, and you you
9:51try to find, you know, I think it's a
9:52greedy algorithm. First you try to find
9:54the most frequent, you know, one
9:55character thing and you find in the most
9:57frequent combination of two characters
9:58and you remove you know, it's it's a
10:00little bit complicated which I let me
10:02not try to give you the precise version,
10:04but it's kind of like a a greedy
10:06algorithm that decides what one which
10:08one is more frequent
10:10and so and so forth. And if you are
10:12interested, you can look up the exact
10:13algorithm.
10:15Um Okay, so that's the tokenization. So
10:18that means that from now on, you know,
10:20we just assume we have a tokenizer. So
10:22every text, every piece of text will be
10:24turned into a sequence of tokens. So
10:26from now on like a
10:29a text it just means a sequence of
10:30tokens and each of this xi is in the
10:33vocabulary. Let's call the vocabulary v
10:35v.
10:37And the size of the vocabulary could be
10:38like 250
10:40thousands.
10:42Um and you know, each of these is kind
10:44of like index in the vocabulary like you
10:46know, this is the fifth token, this is
10:48the
10:49105th token and so and so forth.
10:51Okay. So and now the question is, you
10:53know, now we go into the next question
10:56which is how do we model this
10:57distribution of sequence of tokens? How
11:00do we create a probabilistic model to
11:02model it? If you think about the support
11:05of
11:06this this distribution, like how many
11:08possible combinations
11:10that they are possible here? So the
11:12number of possible combinations is
11:14you have v choice.
11:17This is the cardinality of v. You have v
11:18choice, cardinality of v choice for each
11:20of the
11:21the
11:23tokens and you have a sequence of length
11:25t. So you take the power to the t,
11:26right? So everyone has
11:29v choice and the v to the power t is the
11:31total number of choices. So, you cannot
11:34assign one probability for each of this
11:36sequence
11:39like like a like a like a like a
11:41discrete categorical tokens.
11:43Categorical distributions, right? If you
11:44treat this as a categorical
11:45distributions, then every choice here
11:48will have one probability, and that's
11:50just way too many parameters to describe
11:52it. So, that's why
11:54what people do is that people decompose
11:56this distribution using the chain rule
11:58to write it this way. So, you say
12:00suppose I have a probability
12:01distribution
12:03over the joint of the sequence, then I
12:06can write it as the following. So, I can
12:08say it's equals to
12:09P of X1 * P of
12:12X2 given X1
12:14P of X3
12:15given X2 and X1
12:18so and so forth and P of XT given X1 up
12:21to XT - 1.
12:23And this is the the so-called, you know,
12:25conditional
12:27probability decomposition. You decompose
12:29the joint distribution into a product of
12:32conditional probabilities.
12:34And you model each of these conditional
12:36probabilities.
12:37The benefit here is that each of these
12:38conditional probabilities
12:40is a distribution is a conditional
12:42distribution, but the choice of X2, for
12:45example, here is only V. You have V
12:48choices here. You only have a You only
12:50have to have a kind of like a softmax or
12:52logits over V choices for each of this.
12:55Right? So, each of these will be
12:56eventually you'll see is a some kind of
12:58like a
12:59um
13:00um you have some logits and you apply
13:01some softmax, you get a probability
13:02distribution over
13:04V choice.
13:05So, that's what we're going to do.
13:08So, basically each of this um uh will be
13:10parameterized by a new network. Um the
13:13input will be uh what's after the
13:16conditioning. Right? So, input here is
13:17the X1, X2, and output is the
13:19distribution of X3. And the And this
13:21input-output relationship is modeled by
13:24um a
13:26uh uh network, which is the transformer.
13:29Okay, so that's the um
13:31what we are
13:32going towards. How do we model each of
13:34these conditional distributions?
13:37So, um
13:40So, basically we're going to describe a
13:41conditional distribution P of P sub
13:43theta Xt given X1
13:46up to Xt minus one.
13:48So, um
13:52Before going to details, let's treat
13:54this as a kind of like a black box
13:55first. And let's first kind of like
13:58discuss a little bit about, you know,
13:59the
14:00uh the boundary, right? In some sense,
14:02right? So, like it's kind of like a
14:03there are multiple layers here. So, the
14:05typical way to um um
14:07parameterize this is the following. So,
14:10uh use a transformer. So, basically what
14:12happens is that you have
14:14a sequence X1 up to Xt.
14:16And um
14:18um So, first of all, these are discrete
14:20IDs, right? So, each of these means a
14:22um ID for the token. You first have to
14:25turn them into some numerical numbers.
14:28So, the the the first step is that you
14:30say, I'm going to turn this into
14:32an embedding E of X1.
14:35And so, basically I turn each of the
14:37token X into
14:39uh Xt into embedding of Xt.
14:43And what is the embedding? Embedding is
14:44a
14:45let's say D-dimensional vector uh in um
14:49the Euclidean space.
14:50And this embedding is is trained. So,
14:52basically you don't know what the
14:53parameters are. Uh you will just uh
14:55parameterize it, you know, as unknown
14:57numbers, and you train the model to uh
15:00find the right numbers um
15:02uh for uh for these embeddings. So,
15:04every
15:05every um
15:07token in the vocabulary will be will
15:09have a corresponding embeddings. So,
15:11basically, um you're going to have a um
15:15a sequence of embeddings E1, E2. This is
15:17the embedding for the first token,
15:19embedding for the second token, up to
15:20the embedding for the last token.
15:22Um and in this lecture, I think we're
15:24going to use row vectors because I think
15:27um we are getting to the
15:28implementations, you know, every vector
15:30is actually row vectors in the uh in the
15:32Python code, even though in mathematics
15:34people often use column vectors. So, um
15:37so I I thought about this and I I I
15:38thought just the for this for large
15:40language model sections, we'll just
15:41always use row vectors, which is more
15:43consistent with the practice. It looks a
15:45bit weird to me, actually, uh from a
15:48mathematical perspective. And you if you
15:49read papers, many of the papers do use
15:51column vectors, but I think we'll just
15:53use row vectors to be consistent with
15:54the code. So, basically, every embedding
15:56is a row vector.
15:58And then you have this uh
16:00embedding matrix. Um so, this is um you
16:03have a
16:06V of them, and each of them is of
16:08dimension D. So, you have this matrix.
16:10And uh and to read off which row you
16:12are, you just have to multiply either
16:14you just read off the ID, right? So, if
16:15you have the fifth token, you just read
16:17the fifth row. Or you can multiply this
16:19matrix with
16:20um
16:21uh uh indicator vector to get that row,
16:23which is the same thing. Anyway, so
16:25we'll just read off from this embedding
16:27matrix the corresponding embeddings for
16:29each of the token. So, you get E of X2
16:31up to E of XT.
16:35Um and uh
16:36and I think in in many cases often
16:39people append um prepend um a uh
16:43beginning of sentence token, uh which is
16:45just a fixed token uh in the very
16:46beginning. This is a kind of like a not
16:48always necessary. In some cases, you
16:50just don't do it. In some cases, you do
16:51it. But let's um probably just introduce
16:54that. You have a beginning of sentence
16:55token, which is one of the special token
16:57in this vocabulary. You probably often
16:59actually what happens is that your
17:01vocabulary is bigger than what is
17:03needed. You always reserve a few tokens,
17:05you know, actually maybe a thousand, you
17:06know, a few thousand tokens for special
17:08tokens that you can introduce uh in in
17:12uh for special purposes. And you can
17:14introduce this beginning of sentence
17:15token uh as X0, so which is always the
17:18same.
17:19Um and then you turn them into
17:20embeddings.
17:21And then what you do is that you um go
17:23through this transformer, which I'll
17:25discuss in the later half of the
17:27lecture.
17:28So, this is a transformer.
17:32For now, you just think of this as a
17:33network which does a lot of computation,
17:35matrix multiplication, you know, uh
17:37tensions, which I'm going to discuss.
17:39A lot of operations, and then you output
17:41um
17:42uh some logits
17:44of
17:45like a you output some kind of logits
17:47at each of the locations.
17:51So, each of these logits is supposed to
17:53describe the conditional distribution um
17:56before the uh the softmax. So, basically
18:01um
18:05what you do is that you say
18:07um the probability of um
18:09So, let's just you say each of these U1
18:12is
18:13Let's call this U1
18:15>> [snorts]
18:15>> U of theta
18:16X0, and then P of X1
18:20is
18:21um
18:23modeled as, you know,
18:25just the softmax.
18:28This distribution P of XI is the
18:31X1 is modeled as the softmax of
18:34um
18:35F theta
18:36X0.
18:38So, the first uh
18:40distribution, which is not conditional,
18:41you just uh get the output from
18:44the X0, you get some logic. So, this U
18:47UI is in dimension
18:49V,
18:50and you take a softmax, you get a
18:51probability vector.
18:53So, this is a probability vector.
18:59Right? In this uh simplex of dimension
19:01V, right? So, the sum this this after
19:03taking a softmax, the sum of the entries
19:05is one, uh and all the all of the
19:07entries are non-active.
19:09So, then this is your probability for
19:10X1. This is the distribution.
19:12Technically, I think the distribution of
19:13this is equal to softmax.
19:15Okay? Um and then
19:17the same thing, you know, when you have
19:18X2
19:20given X1 you say this is equals to
19:22softmax
19:25of um
19:27U2
19:28where U2 is the output at the second
19:30position.
19:31And and U2, you know, we'll just call U2
19:33the same as, you know,
19:35so you call U2
19:38the output, you know,
19:40of um this model
19:43given X
19:45um
19:46zero and X1.
19:49So
19:50so each of these UIs, you know, um is uh
19:54basically the output
19:56um
19:58for
19:59um the sequence before it, right? So UT
20:01+ 1 is F theta X0 up to XT.
20:06So um so so basically generically, you
20:09know, P of XT given X1
20:12up XT - 1
20:14is the softmax
20:17of
20:19um UT
20:22uh
20:23Is it UT or UT + 1? UT +
20:26UT, yes. UT. Which is the softmax
20:31of F theta X0 up to XT
20:36T - 1.
20:44Okay?
20:45Any questions so far?
20:47>> Is your index for your U starting at
20:49zero or
20:51>> So I I I particularly chose it to index
20:52at one, yes.
20:53>> Okay. And so that's
20:55this
20:56on the right.
20:57>> Yes.
20:57>> Okay.
20:58I'm over here.
21:00You're looking at the same thing.
21:01>> U is just a vector, you know. I think
21:03you know, I can just call U I I uh
21:06it's kind of like uh I can just call you
21:07f theta x zero it's the same thing. I'm
21:09not
21:10Did I answer the question?
21:27So oh I think uh maybe maybe that's the
21:30what
21:31So
21:32So
21:32I'm just using f theta x zero x one as a
21:36more explicit way to represent UI
21:38because I'm trying to
21:40express the dependency.
21:42So
21:43So basically I'm saying like a basically
21:45you
21:46I I think they are the same thing. Like
21:48it's just there there are no difference
21:49at all in some sense. It's just how you
21:50view it, right? Like in one way you just
21:51say oh I have I need a an
21:54annotation for the vector and the other
21:55way is that I have I need to interpret
21:57this vector as a function of the inputs.
22:02Oh okay okay okay. So maybe I didn't get
22:03the question.
22:44Yes yes yes. So I think maybe
22:47I'm trying to figure out what's the
22:48question here. So
22:49um
22:50I think maybe what you're asking is the
22:52following. So
22:54So maybe let's take this one, right? So
22:56this is a So this is a vector.
22:59This is a vector in dimension
23:01V.
23:03So in in the vocabulary size. And you
23:05have to take a softmax, it's a it's a a
23:07sequence of numbers which are
23:09uh um
23:10all non-negative and they sum to one.
23:13So, and what this really means is that
23:15um
23:16P theta of XT is equals to one
23:20given X1 up to XT minus one.
23:23The chance to see the first token under
23:25this probabilistic model parameterized
23:26by theta is equals to
23:30This is a vector. The first
23:31entry of this vector, right? It's equals
23:33to softmax
23:36F theta X0 up to XT minus one
23:40indexed by one. This is the first entry
23:42of this vector. And you have many many
23:44other entries, right? Each entry is like
23:46you have another entry softmax
23:50F theta
23:51Sorry.
23:57two, and that is is equals to P theta of
24:01XT is equal to two. The chance of seeing
24:03the second
24:04second token given X1
24:07up to XT minus one.
24:09And you have so on and so forth.
24:12Does that answer the question?
24:14V is the
24:15uh the Yeah, this V absolute value is
24:17the size of the vocabulary.
24:21>> So, this is like a masked attention
24:23because the model is only conditional on
24:25like up to the previous
24:27ones after it, right?
24:28>> Yep.
24:29Yep. So, these are causal models so far.
24:31But I haven't defined the transformer
24:32yet. So, but I mean insisting that this
24:34is only So, guys, this is one of the
24:37reason I I'm using this notation. So, I
24:38mean insisting that UT
24:40is only a function of X0 and XT minus
24:42one, but not a function of the
24:45the the later uh inputs.
24:47So, but that will be um
24:50that will be guaranteed when we
24:51introduce the exact inner working of
24:54this transformer.
24:57>> So, functions of X are not the previous
24:59ones.
25:10>> Okay, good question. So the first the
25:12second question is easier to answer. So
25:14the embeddings is only a property of the
25:17this X, right? You don't have to see any
25:18other things, right? It's just actually
25:20it's literally if this is the first
25:21token then you just read the first row
25:23of this matrix. If this is the 10th
25:25token you use the 10th row and that's
25:26your embedding. No other dependencies.
25:28And while we are writing this as a
25:30as functions of the X but not embeddings
25:33that's just the
25:34it's just a
25:36It sometimes the embeddings is the inner
25:38working to some degree. I'm just trying
25:40to give you more I'm trying to unpack it
25:42a little bit, right? So So So So if you
25:44don't have to unpack it, you just work
25:45you can work with the text. Yeah. That's
25:47just the representation. Yeah.
25:50And it's one-to-one so in some sense
25:51there is no difference.
25:56Okay, so this is a full description of
25:58the probabilistic model. Once you have
26:00this probabilistic model you can compute
26:01the maximum likelihood given any data.
26:03We can say I have a data distribution I
26:05have a data point a sequence of data. I
26:07can see under this probabilistic model P
26:09theta what is the chance of seeing this
26:13this data. So that's that's what you
26:15need for computing the do the training,
26:17right? You You You need to compute the
26:18likelihood and then you maximize the
26:20likelihood as a function of theta,
26:22right? So
26:24and let's briefly discuss you know for
26:27generation, right? So suppose you
26:29already have the
26:30generation or decoding.
26:33Suppose you already have the model.
26:35Then
26:36a model theta then what do you do? So
26:38the
26:39as you can see the most natural thing is
26:41you just keep generating X1 X2 X3 so and
26:44so forth, right? You just say I'm going
26:46to generate X1 first from this you know
26:50P say softmax.
26:55Softmax, you know, F theta
26:58X1 up to X0 up to XT X, I guess I just
27:02rewrite
27:04uh, generically. So, you generate XT
27:05minus one from
27:07um, F theta X0 up to XT minus one.
27:10Right? You do this sequentially.
27:12Sometimes people draw this in this way.
27:13So, you say, "Oh, you have X0. You first
27:16go through the transformer
27:18and you get X1 and you feed X1 back to
27:21this transformer.
27:24Right? So,
27:25and um,
27:27and then you generate X2
27:29by sampling from the softmax. And then
27:32you you feed back. So, that's kind of
27:34why it's called autoregressive because
27:35you
27:36you have to like when you are
27:37generating, you don't have the sequence,
27:39right? When you are computing log
27:40likelihood, you see the whole you
27:41already see the whole sequence. You are
27:43just trying to figure out what's the
27:44chance of seeing this sequence. But,
27:46when you are generating, you you start
27:47with nothing. And then you have to
27:48generate your sequence yourself. And you
27:51That's why you start with a beginning of
27:52sentences sequence uh, token and then
27:54you generate first token and then you
27:56give the first token back to transformer
27:58and compute this um, softmax. And then
28:00you um, generate second token. You do a
28:02sampling to generate second token. You
28:04give it back and you keep doing this.
28:06So, that's the autoregressive
28:08generation.
28:09Um, the um,
28:12um, and uh, sometimes, you know, like
28:13uh, this generation only start from
28:16certain part because in many cases you
28:18are not generating from nothing. So, you
28:21are given maybe a few tokens. Maybe you
28:23are given So, the sometimes it's called
28:25prompt.
28:27So, prompt is like something the models
28:28are given. It's maybe like a question,
28:30you know, an instruction so and so
28:31forth. You say you are given X1 up to
28:34XK. And then the model is supposed to
28:37complete the generation. So, it start
28:39with XT plus one. So, XT plus one
28:42So, this is given.
28:44And then you start with XT plus one. You
28:45first generate XT + 1 from the softmax
28:49of the alpha theta X1
28:51up to X
28:53K
28:54and then you generate XT + 2
28:57and then you do the the same thing. So,
28:59it doesn't have to be always from the
29:00beginning.
29:02And then there's another small thing
29:03which is that when you generate, you
29:06don't necessarily have to always follow
29:08the same um model. And there's one thing
29:11that you can deviate from it which is
29:12that you can um add additional
29:15temperature
29:18where um you
29:19say as opposed to generating
29:22from the original
29:24um probabilistic model, I'm generate I'm
29:26going to divide this by a a temperature
29:28T tau or T sometimes. Um so, um and uh
29:33what this what what does this mean is
29:35that you change the
29:37you don't change the orders between this
29:39uh choices, right? So, this is a
29:41this this alpha theta is a vector,
29:43right? It's a a V-dimensional vector
29:45that describes the largest for each of
29:46the tokens in the vocabulary.
29:49And when you divide by tau, you don't
29:51change their relative orders. The bigger
29:52ones are still bigger, the smaller ones
29:53are small still smaller, but you change
29:56the
29:57the scale.
29:58So, um so, for example, before you could
30:00have like a
30:02So,
30:04suppose, you know, alpha theta
30:06X0 up to XK
30:09is
30:09is equals to, let's say,
30:12maybe like a 2 1
30:15-1, something like this.
30:17And if you divide by tau, let's say
30:19suppose you divide by tau which is very
30:21very small. Suppose tau is almost zero.
30:24Then after dividing by tau,
30:26like if you divide by tau, then what
30:28happens is that maybe this becomes 200.
30:30This becomes 100. I'm I'm giving you a a
30:32concrete example which is a
30:34a little bit unrealistic, but to
30:36demonstrate a point. And what's the
30:37difference between the softmax of this
30:39vector versus the softmax of the
30:41readjusted vector.
30:43The difference is that the softmax of
30:45this is much sharper.
30:46So, maybe let me just give you an
30:48example here. So,
30:50So, if you just have let's say maybe
30:52let's just have three dimensions. So,
30:53that it's easier to demonstrate the
30:55point.
31:00So, if you do softmax of this vector
31:05of 2 1 -1, what happens is that you get
31:11you sometimes get exponential 2
31:14over exponential 2
31:16plus exponential 1 plus exponential
31:20-1.
31:21That's the after softmax was the
31:23probability of the first
31:25token.
31:26Right? And the second token will be
31:27exponential 1
31:29exponential
31:312 plus exponential 1
31:34plus exponential -1.
31:36Right? The third one will be the the
31:38same denominator, the different
31:40numerator, right?
31:41So, that's a
31:42that's the softmax of this.
31:44But, if you take the softmax of the 200
31:46100 -100, suppose you do that.
31:55Then, this will give you
31:57exponential
31:58200
32:00versus exponential
32:02200 plus exponential
32:04100 plus exponential
32:07-100.
32:09Blah blah blah.
32:10So, what's the difference here? The
32:12difference is that in terms of the these
32:13three numbers, in terms of the order,
32:15it's always like this is bigger than
32:16this, this is bigger than this, right?
32:18Because 200 is big two is bigger than
32:19one, one is bigger than minus one. The
32:21order is always maintained.
32:23But, the
32:25the the relative place um um scale is
32:28different because here you can see
32:30exponential 200 is way way bigger than
32:32exponential 100. The difference is like
32:35so astro astronomical. Like it's too the
32:37the difference is too big so that these
32:39two numbers are just so small compared
32:41to exponential 200.
32:43So, they are even negligible.
32:44So, basically in this calculation, these
32:47two numbers don't even have to show up
32:48here. Like it don't even matter. So, you
32:50just get what?
32:51So, basically the first token will just
32:52have probably exactly almost exactly
32:54one. I mean, 0.9999 something. And then
32:56if you look at the second I entry,
32:59which is exponential 100
33:01divided by exponential
33:04200 plus exponential
33:06100 plus exponential
33:08-100.
33:11Again, everything is negligible compared
33:13to exponential 200.
33:14So, this is not very useful. This is not
33:16very useful. This is kind of like close
33:18to zero. So, basically this entry will
33:20be zero.
33:21So, that's why after you you you divide
33:23by tau to make this bigger, then this is
33:25just approximately
33:271 0 and 0. So, that's that's what I mean
33:30by it's much sharper. So, you you you
33:32you sharpen the the distribution to make
33:34it most of the distribution to be the
33:37the probability mass to be on the the
33:38most likely token.
33:40And however, here
33:42exponential two and exponential one are
33:43not that different. If you compute this,
33:45you'll get something like maybe the
33:47first one will be like maybe 0.7, the
33:49second one will be 0.3. I I don't know.
33:51Like we can use a calculator to see. But
33:53you know, it will still be like this one
33:54is bigger than this. And the second one
33:56is bigger than the third one, but it
33:57won't be like
33:58all the mass is on the first token.
34:00So,
34:01so so basically what I'm trying to say
34:02is that if tau is a small number so that
34:04you
34:05mhm
34:05you you amplify the logits, then you are
34:08sharpening the resulting distribution to
34:10make it more like a deterministic
34:11distribution.
34:12And then as a result, you are just
34:14generating
34:15um you are just a sampling from the the
34:17most likely token. You are taking the
34:19max as opposed to
34:20sampling from it.
34:22So, and um and and that's actually what
34:25people actually in
34:26um um do. Like they say basically like I
34:30think actually
34:31uh in API call, you have the choice to
34:33choose the temperature
34:34and if you choose a small temperature it
34:36means that you want more deterministic
34:37generation.
34:39If you choose a temperature to be zero
34:40it means that your generation will just
34:41be always deterministic every time you
34:43take the largest
34:46most likely token and then keep going on
34:48right and of course you can also change
34:50the temperature to be smaller than
34:53bigger than one which makes the the
34:55distribution more softer so that there's
34:56a you focus on more of the long tail
34:58like you can generate more tokens from
35:01the tail so you have more stochasticity
35:03and uncertainty in your generation.
35:05Sometimes that's useful because you want
35:07to encourage some diversity in your
35:09generation and you you do that
35:11especially in the RL sometimes you have
35:12to do that
35:14um
35:15Cool. Okay, sounds good.
35:18And sometimes there are also other ways
35:19of something which is
35:21probably I shouldn't go into details
35:23there are some notes
35:24in the lecture notes. So another way to
35:26do it is that you
35:28this vector right now I have three
35:30dimensions but actually it's like a
35:32the dimensionality is V right like which
35:34is like 250 or something. You can take
35:36top K and then drop all of the other
35:39dimensions so you only sample from the
35:41the most likely K tokens and you
35:43re-normalize the distribution. So
35:45basically the the first K tokens may not
35:48cover the all the probabilistic
35:49probability mass so
35:51but still that's probably easier to
35:53generate because you don't want to
35:54generate too rare tokens so you just
35:56drop everything below top K and then
35:58generate.
36:00Okay,
36:02sounds good. So I think I'm
36:04probably
36:10And then okay so once you have so this
36:13is the generation and then
36:15for the training I think I already kind
36:17of like describe it
36:18a little bit
36:20so when you do the training
36:24you just
36:25compute a log
36:26the negative
36:29log likelihood.
36:33Like this.
36:36The
36:37negative log likelihood. And you say
36:40this is a log, so if you have example X1
36:42up to XT
36:44So the minus log P theta X1 up to XT
36:50capital T is just the
36:53minus log the product of all of this,
36:56right?
36:58P theta X1
37:00P theta X2 given X1, so on and so forth.
37:11And this is the minus the sum of the log
37:14of the P of XT
37:16given X1 up to XT minus 1.
37:21And then you change this to
37:24log of the softmax
37:28of P1
37:29of this F theta.
37:37And here this XT is a particular choice
37:39of XT, right? Which is seen in the data.
37:42So that means you are looking at the
37:44XT's entry of the softmax. This softmax
37:47will give you the probability for every
37:48possible tokens. And then you are only
37:50interested in the probability to see
37:52that particular X token XT. So that's
37:54why you're looking at XT, the index of
37:57that vector. And that's a
37:59single scalar, you take the log and you
38:01take the sum over T.
38:02Um and that's your loss function.
38:06And then you can optimize this loss
38:07function by, for example, some
38:09optimizers,
38:10um like SGD or Adam. Uh we didn't
38:13discuss Adam in this lecture, which is
38:14an extension of the X SGD optimizer, but
38:17mostly you can just call the optimizer
38:20as a black box to optimize the loss
38:21function.
38:38It's not actually not the last one,
38:39right? Because this is sum from T to
38:41>> Oh.
38:42>> 1 to capital T.
38:44The little T and large T. Little T.
38:54Okay. Sounds good.
38:57Okay. So, then in the next uh
38:5930 minutes, I'm going to tell you what
39:01is this function F is, right? Or maybe
39:04what's hidden in this transformer.
39:06What's the operation we have to do to
39:08compute um the the the logits here.
39:12Um and it's pretty complicated. So, um
39:15it probably takes more than 30 minutes,
39:16but I'll see how much I can do.
39:20So, um
39:25Any questions before that?
39:27Okay.
39:28So, at a high level, this transformer
39:30has multiple layers.
39:32And uh so, you start with like a EX1 up
39:35to EXT.
39:37And you first apply a so-called
39:39attention layer,
39:42which I will define.
39:43And then you apply the so-called MLP.
39:46So,
39:47um first of all, the attention layer,
39:49the attention always takes in
39:51uh a sequence of vectors and output a
39:53sequence of vectors.
39:55So, you output a bunch of vectors. At
39:57each position, you output one vector.
39:59And then, each of these vector will go
40:01through a MLP. MLP means like a
40:04multi-layer perception, which is just a
40:05two or three-layer
40:07two or three-layer network.
40:09And uh each of these vector will go
40:11through this MLP.
40:15And then, you apply another attention.
40:17This attention will takes in
40:19many, many vectors and and then output a
40:22sequence of vectors.
40:24And then you do the MLP.
40:28So, that's the the high-level um
40:31structure of this attention. And
40:32eventually, you have to output
40:35Remember, eventually, you have to output
40:37a sequence of vectors. So, basically,
40:39you do enough of these alternations
40:41between these two and eventually you
40:42have this uh
40:43uh MLP and you output
40:46some vectors U1 up to UT.
40:49Right.
40:50And you alternate between attention and
40:52MLP.
40:53That's the high-level idea.
40:55And um and I think we'll focus most of
40:59the our energy on defining attention,
41:00which is kind of like the most uh uh new
41:03thing, right? Because MLP is just a
41:04two-layer network uh with some special
41:07kind of gating uh or kind of like some
41:09kind of like a special activation
41:10functions. Uh and another small thing is
41:13that the MLP are sharing the parameters
41:16and they are applying independently. So,
41:18each of these vectors is
41:20processed by the MLP in an independent
41:22fashion. There's no kind of dependencies
41:24between positions when you apply the
41:26MLP. They are just very separate
41:28operations. So, the only way that you
41:30are using um um you are kind of like
41:33fuse you fuse the input from different
41:35positions or different time steps is the
41:37attention. So, the attention will look
41:38at all those vectors and and process a
41:41little bit and then output a sequence of
41:42vectors. And MLP is is very parallelized
41:45and parallelizable and independent.
41:49Okay. So,
41:50now let's let's just define our task.
41:58This entire thing will be treated as a
42:00single system. So, the So, So,
42:01basically, once you define the
42:02architecture, you apply the auto
42:04differentiation.
42:06You you like a you treat this whole
42:08thing as a as a gigantic network. Yeah,
42:10end to end.
42:20>> [clears throat]
42:21>> Yeah, but the auto differentiation still
42:22works.
42:23So, so the auto differentiation, I think
42:25as we discussed, if you remember like
42:27it's kind of like fundamental is
42:28claiming that if you have a way to
42:30compute your forward pass,
42:32as long as they are compositions of many
42:33many different components, you know,
42:35like a basic components, you can compute
42:37a backward pass the same fast the same
42:39way. So, here
42:41is is indeed a composition of many many
42:42small arithmetic operations because as
42:45you see the attention layer has many
42:49small matrix multiplications and the MLP
42:51has many matrix multiplications and you
42:52can automatically differentiate.
42:56Um
42:57Okay, so
43:01Okay, so now let's let's talk about
43:02attention.
43:14So, for attention,
43:16one thing to remember first is that the
43:18type signature is that you your input a
43:20sequence of vectors to the attention and
43:22they output a sequence of vectors.
43:25And and
43:27we're going to start with the so-called
43:28single head attention.
43:35So, the single head attention is like
43:36this. So, you first have um
43:40uh input as a a sequence of vectors.
43:44Um let's say H1.
43:46Say
43:47uh in
43:48up to HT in.
43:51And and you want to output, you know, H.
43:53I'm trying to define the the type
43:55signature.
43:57So, your output
43:59H1 out up to HT out.
44:03And what you do is the following. So,
44:06the first thing is that you turn
44:09this input vectors into a set of query
44:11key
44:13and and the value vectors.
44:15So
44:17So you define so basically you say H
44:22one in
44:24you multiply it by a matrix WQ. This is
44:26a weight matrix. This is a parameter.
44:30And
44:31by the way, let's treat all of these as
44:32row vectors and then all the
44:33multiplications are on the right hand
44:35side as opposed to on the left hand
44:36side. So every vector is a row vector
44:38and you multiply on the right hand side
44:40you get another row vector.
44:41So so this H in let's say this is like a
44:44dimension D
44:46RD
44:47I say say R one by D and this WQ is in
44:50dimension
44:53So this is in R one by D and this WQ is
44:56in
44:57R D by
44:59um
45:00DH.
45:02DH is a parameter. It's a dimension you
45:04choose. um
45:06And and then the resulting thing is
45:09called the Q
45:11one which is in dimension
45:14DH. Maybe one by DH depending on how you
45:16think about it.
45:18um
45:19And you can do this for every index
45:20every time step. Here I'm only doing it
45:22for time step one. You can do it for
45:23time step T. So you have QT.
45:26So basically this give you a factor a
45:28sequence of vectors Q1 up to Q capital
45:30T. Each of them is in dimension
45:33uh
45:34one by DH.
45:36And this these are called queries.
45:40And you do the same thing for keys and
45:42values. So keys
45:45Let me define them first. So key
45:47is the same thing. You say the HT in
45:50multiplied by a so-called key projection
45:52matrix WK
45:54and that's your KT. And the
45:56dimensionality I think is the same. So
45:58this is also in the same dimension and
46:00the KT is in the same dimension. And the
46:03same thing for value function value
46:04vectors, not value functions. Uh HT
46:08in times W
46:11V.
46:12So,
46:13in some sense maybe
46:15maybe this is the right time to
46:18talk about some of the intuitions, you
46:19know, for the attentions, which is not
46:21that obvious because eventually it just
46:22works, you know, like nobody really know
46:24exactly how it works. Um um but the the
46:27rough intuition is that you have to have
46:28some interactions between these inputs
46:30to make it even meaningful, right?
46:32Because if everything is independent,
46:33then you just dealing with all the time
46:35steps independently.
46:36So, that's why you have to have some um
46:38um interactions. And the interaction
46:40sometimes is about how do you
46:42kind of like a retrieve or kind of pay
46:45attention to each other. So, that's
46:47actually how the word attention come
46:48from. So, basically, when you are
46:50processing the maybe this vector,
46:53you probably should attend to or pay
46:55attention to some other vectors which
46:57are have relevant information.
46:59And which one you you pay attention to
47:01is what we're going to decide here using
47:02with this
47:03query and key and vectors. So, in some
47:05sense, intuitively, the attention will
47:07what attention will do, which I didn't
47:09show you yet, is that
47:11you try to
47:12um for processing each of the vectors,
47:14you try to figure out which are the
47:16relevant vectors
47:18that has relevant information. And and
47:20then you pay actual attention to those
47:22vectors.
47:23And and then you compute the output
47:25depends on depending on those special
47:27vectors to some degree.
47:29So, that's what we are trying to do.
47:31Um and
47:32and to kind of like decide which one you
47:34pay attention to, you have to prepare
47:36these queries and key and value vectors.
47:39Um
47:40and how do you use them? So, you use the
47:41query and keys as a way to decide which
47:44one you pay attention to. So, basically,
47:46let's say um suppose you have um
47:49uh the teeth position, right?
47:52I want to decide
47:53what the teeth position should pay
47:55attention to. Um and and
47:58and the way to do it is you say you
47:59compute
48:00the query QT.
48:03This is uh the query at this position.
48:06And you multiply the query with the keys
48:09at all the other positions.
48:11So, maybe I I will just draw it.
48:14So, you have queries Q1 up to
48:16QT.
48:18And one of them is
48:19Q little T. And you have Q
48:22K1 up to KT.
48:25KT.
48:27And you you multiply Qs with all the Ks.
48:33So, you have Q1 times, you know, K1
48:36transpose,
48:37QT times K2 transpose,
48:40and
48:41QT times K capital T transpose.
48:45Um, and there is a um
48:47sometimes there's a scaling factor here
48:50to kind of like normalize this in a way
48:51that
48:52makes it a little more kind of like a
48:54well-behaved, but I think for the moment
48:56you probably don't have to pay too much
48:58attention to the scaling factor. So, the
49:00important thing is that you are taking
49:01the inner product between Qs and each of
49:03the other Ks. And this QT transpose K
49:06Q
49:07K transpose is actually inner product
49:09because Q is a
49:11is a row vector,
49:13and K transpose is a column vector.
49:17And you are taking the inner product
49:19between these row vectors and the column
49:21vectors.
49:22Um,
49:23and you get one scalar out of it.
49:25So,
49:26so this is a
49:28all of these numbers are real numbers,
49:29right? So, this is in RT. So, you have T
49:32real numbers.
49:34Um, and then you take a softmax over
49:36these T real numbers to turn it into a
49:37probability vector.
49:39So, then you say I'm going to do a
49:40softmax.
49:45Say softmax
49:48of this whole thing.
49:57>> And now you get what? You get a This is
49:59a vector.
50:00The dimensionality here is RT and each
50:03of this This is this the the outcome of
50:06this softmax is a sequence of numbers,
50:08right?
50:10And there are T numbers and all of these
50:13numbers are non-active and the sum of
50:14them is one. So it's a probability
50:16vector over T positions. And if you see
50:19a large number, it means that
50:22Let's say suppose this number is big, it
50:24means that this number is somewhat big
50:27compared to other ones. It means that
50:28your Qs and keys have like more higher
50:31inner product. Right? And uh
50:34it means that you should pay more
50:35attention to K K1. Right? So So
50:38basically if any of these entries big,
50:39it means that
50:42intuitively
50:44QT needs to pay more attention to K1.
50:46And uh that's will be reflected by the
50:48probability vectors because the the
50:50corresponding probability will also be
50:51high.
50:52And
50:54so I think let's call this I guess I
50:55have a notation here, so let's call this
50:58let's denote this to be PT1,
51:01PT2,
51:03up to PT capital T.
51:06So the So each of the PTI is
51:10each of the PTI is non-active and the
51:12sum of the PTI over I from one to T is
51:17equal to one. So this is a probability
51:18vector.
51:19Um and and if this if any of these
51:21entries is big, it means that we should
51:23pay up more attention to that particular
51:24position when processing the little T's
51:27query.
51:29And then how do you turn this How does
51:30this relate to the final
51:32uh outcome? So finally, once you have
51:35this, then
51:36uh you say the
51:38the output at T's
51:41position, which is a vector,
51:43is using these probability vectors to
51:45weight
51:46uh the values.
51:49The values is um
51:52V's, which are we are defining here,
51:53right? VT, sorry, VT.
51:55So, you say it's PT1 * V1 + PT2
52:00* V2 +
52:03PTT * VT.
52:06So, basically, if each of any of this
52:09probability is much higher, then you are
52:12multiplying this high probability
52:15with the the corresponding value at that
52:16position.
52:18So, basically, you are using the V
52:20Right? If, for example, if this number
52:21is very big, then you are using V2
52:24a lot in the output at time T.
52:32So, you determine them by multiplying
52:34the matrices.
52:35And the matrices will be trained.
52:37Uh uh uh
52:38like like yeah. How how do you attend to
52:40each other is also trained. Like So, so
52:43you are only giving a template in some
52:44sense. Like how how you know Like it may
52:46be that, you know,
52:48like
52:49it could be that all of these are zero,
52:51you know. Then you are not doing
52:52anything with it. And then your network
52:54the loss function will be not good, but
52:56it's still a valid network. It's just
52:57the loss is not good. Yeah.
52:59So, eventually, you are going to train
53:00the model to
53:02behave well so that So, basically,
53:06the training algorithms will choose the
53:07right W so that this mechanism is
53:10uh uh doing the right thing.
53:27So,
53:28um let's try to articulate you know, I
53:29think it's a good question, but let's
53:30try to articulate so that we can answer
53:32it. So, what's the
53:34uh
53:36You are say So,
53:43>> [clears throat]
53:43>> Yeah.
53:44>> So, why
54:00>> [clears throat]
54:00>> So, I think you are asking why you need
54:02such an architecture but not just only
54:04using MLP for example, right? That's one
54:06way to rephrase the question.
54:08I think
54:10um
54:11So, if you are only using MLP in this
54:13way
54:14suppose you remove all the tensions you
54:16are you only using MLP in this way, then
54:17there's no interactions between
54:19different positions whatsoever.
54:21So, there's no chance that's going to
54:23work, right? You're just applying
54:24separate networks for each of the
54:26tokens. Uh there's no chance you can
54:28understand the the the the the the
54:31dependencies the the meanings, right?
54:33Like there's no chance I I can I can
54:34just give you individual words and you
54:36can understand a sentence. There's no
54:37chance, right? So, so you do have to um
54:40have a model that can read uh uh
54:43um all the tokens you know and and use
54:46and study their kind kind of
54:47correlations and combinations and so
54:49forth, right?
54:50But um there are other possibility
54:53possible ways to deal with that. For
54:54example, what if you you use a gigantic
54:56MLP
54:57uh that takes in all of these maybe you
54:59concatenate all of them and you take in
55:01one MLP. Right? So, so um for example,
55:05you just concatenating all of these
55:06tokens
55:07vectors into one long vector and you go
55:09through a network. So, there are several
55:10issues with that. One is that
55:13um you will have way too many
55:15parameters. Because your sequence lines
55:18maybe sometimes even millions. Then if
55:20you have a million by million matrices
55:23in your MLP, it's going to be way too
55:24big. Right now, the MLP is actually very
55:26small. It only takes in a vector for one
55:29position. It doesn't even depend on T.
55:32And another thing is that you're not
55:34only the the size of the MLP is big, but
55:35also the size of the MLP is also changes
55:37over
55:39uh as you change the sequence length,
55:41which is also very undesirable because
55:42for example, if you tune with sequence
55:44length 1,000 and then you test with
55:45sequence length 1,001, then you have to
55:48change your parameter count, which is uh
55:50not good. And and here, if you really
55:53look into details, you'll find out that
55:55the number of parameters doesn't depend
55:58on the sequence length.
55:59>> So, what would be
56:01like you have this gigantic MLP
56:03>> Yep.
56:03>> that is too big to compute.
56:05>> Yep.
56:05>> But you kind of know that this there's
56:08there is some structure because there
56:09are
56:10like words that match other words. So,
56:12this is kind of trying to learn
56:14structure
56:15in a more efficient way.
56:17>> Exactly, right? So, so so you you know
56:19that, you know, if you have sufficient
56:21compute, you know, probably your
56:22gigantic MLP will work. If you have
56:24sufficient data so you don't overfit,
56:26and you have sufficient compute, then
56:28there is some MLP that can actually
56:30solve this problem.
56:31But that's too expensive. So, you need
56:33to prescribe some
56:36structure
56:37so that you can reduce the complexity.
56:39Right? And and this is the the the
56:41prescription we have. Um but you still
56:43have to leave enough flexibilities. You
56:45cannot just say, oh, for example, one
56:46way to do it is you just say, let's
56:47average. Then there's no parameters, you
56:49just uh prescribe too much, right? You
56:51you dictate too much, then that's not
56:53good. So, it's kind of like an you need
56:55to strike a balance between how much
56:57structure you introduce and how much
56:59flexibilities you give the models to
57:00figure out.
57:03Yeah. Um
57:06Yeah. Any other questions?
57:09By the way, there's actually in the
57:11history, there was actually other ways
57:12to deal with the sequence. For example,
57:14you can use a convolutional network on
57:16this, which is kind of like MLP, so you
57:18have use a
57:19use a MLP but with a specific receptive
57:22field so that you each time you only see
57:24a kind of like a window. I think there
57:27are still
57:28papers that so says that
57:31that's not
57:32a terrible idea. Actually, if you use a
57:34convolutional network, it actually can
57:35work to some degree. It's just not
57:37as good as this one, you know. Uh
57:40and uh and and also you can see like
57:42there are also other
57:44Probably we'll talk about that later.
57:45So, there are other variants of
57:46attentions too that does similar stuff
57:48but have better efficiency.
57:50Um
57:51Anyway, so let's continue.
57:53We have We are not done yet, actually.
57:55This is only a single so-called single
57:56head attention, you know. Um actually,
57:59the real thing is a little more
58:00complicated.
58:02Um
58:03but
58:04let's see. Before I go into that, let's
58:06see.
58:11So, I think
58:13you can see the notation here is very
58:14messy. So, let's try to simplify it a
58:17little bit so that we don't have to
58:19always work with this messy notation.
58:22So, one typical way that people simplify
58:24this is that you try to organize all the
58:26queries
58:28into
58:29a matrix.
58:30Recall that each of the queries is a row
58:32vector, and if you put them all in a
58:34matrix, then this is uh
58:36uh a matrix of dimension T by DH.
58:40And you can put all the keys
58:42into a matrix.
58:45And you can put all the values in in a
58:47matrix as well.
58:51Um
58:53All right. So, and then you can write
58:54this H out
58:56um
59:05the output signal factors in either kind
59:08of like a
59:10um
59:11also as a matrix is in some sense you do
59:14a softmax
59:16of the row of
59:18this QK transpose / C matrix * V.
59:22So, here the basically this QK matrix QK
59:25transpose matrix is the
59:28the family of Q I times KJ transpose
59:32divided by C.
59:34So, every entry is of this form,
59:37which is kind of like
59:38these guys.
59:39And then you if you do a row softmax,
59:41that corresponds to this softmax.
59:43And then if you multiply by V,
59:46it means it means this linear
59:47combination. I think you you we probably
59:49we don't have time enough time to
59:51um go into the details like um but you
59:54can double check, you know, it's just a
59:55very linear algebra. It's just you are
59:57rearranging like it's it's mostly just
59:59notations. Right, so this QK divided by
1:00:01C matrix is the inner product between
1:00:03the queries and keys. And then you take
1:00:05a row row wise softmax to get the
1:00:07probabilities. So, after the softmax,
1:00:09this matrix becomes some
1:00:11uh every row is a probability vector and
1:00:13then you multiply that with the V to you
1:00:15take the linear combination of these.
1:00:18Um and uh
1:00:20there's one thing that we kind of cheat
1:00:22a little bit here, which is that if you
1:00:24look at this
1:00:25architecture, then
1:00:28it's not true that
1:00:30the teeth output
1:00:32only depends on everything before it.
1:00:36There's no this kind of like a auto
1:00:38aggressive kind of like properties yet,
1:00:39right? Because
1:00:41the output
1:00:42at time T
1:00:44depends on and everything, right? It
1:00:47depends on V capital T even the last
1:00:49input. V capital T V capital T is a
1:00:52function of the last input.
1:00:54And so and the output at time T depends
1:00:57on the input um um
1:00:59depends on VT and it depends on HT uh
1:01:02in. So, um
1:01:04so basically there's no kind of like
1:01:05this dependency. So, then what you do is
1:01:07you say um
1:01:09you introduce concept of masking to
1:01:12um make sure that there's no
1:01:14uh
1:01:15this kind of like a wrong dependency you
1:01:17want. So, basically what you really want
1:01:18to do is that you just uh
1:01:20in some sense
1:01:21for each little t, you want to delete
1:01:24everything after
1:01:26uh time t. So, basically, um
1:01:29I say
1:01:31before you have this, you know,
1:01:33p t v
1:01:35t plus
1:01:39p t t v t p t capital t v capital t.
1:01:43So, basically, you say these
1:01:45dependencies are invalid for me, so I
1:01:46just
1:01:47don't have them. So, basically, you just
1:01:49delete all of this.
1:01:50And um
1:01:52and then you re-normalize these p's so
1:01:54that the sum of these p's
1:01:56uh got normalized to one. So, that is
1:01:58still a linear combination or convex
1:01:59combination of the v one up to v t.
1:02:02So, that's pretty much what you do. So,
1:02:03you you just say if the the dependency
1:02:05is um not good, you just
1:02:07sometimes mask them out.
1:02:09And mathematically, what you do, uh
1:02:10actually, also implementation-wise, uh
1:02:12what you do is through this masking,
1:02:14which means that
1:02:17you
1:02:19um
1:02:23which means that you add
1:02:25um
1:02:27a masking here.
1:02:29Maybe I should use a different color to
1:02:31so that they're
1:02:33So, you add a masking here.
1:02:35Let me use a different
1:02:36color
1:02:43Okay.
1:02:44So, you add a mask here
1:02:46so that um you can mask out the
1:02:51Let me see. You can mask out half of the
1:02:53this matrix.
1:02:54So,
1:02:56so you have like a q one k one transpose
1:02:59q one k two transpose so on and so
1:03:01forth.
1:03:03And q one k two transpose is not valid
1:03:05in some sense because you are looking
1:03:06you are tending to something you
1:03:08shouldn't attend, right? Q one shouldn't
1:03:10be able to see k two. So, basically, you
1:03:12add the this masking matrix. You what
1:03:14you add is that you add
1:03:16minus infinity.
1:03:19You add a minus infinity here.
1:03:23And for
1:03:25Q1 K3 transpose, you also add minus
1:03:27infinity.
1:03:29So and so forth.
1:03:31And then in the next row you have Q2 K1
1:03:34transpose, which is good. You have Q2 K2
1:03:36transpose, which is good. And you have
1:03:38Q2 K3 transpose, which is not good. So
1:03:42you add minus infinity here.
1:03:44So and so forth. So basically everything
1:03:46here
1:03:47will be
1:03:48adding additional minus infinity. So
1:03:50basically this
1:03:51upper triangle um part of this matrix
1:03:54will be all minus infinity. You know, I
1:03:55mean
1:03:57by adding minus infinity you just mean
1:03:58that you you turn it into minus
1:03:59infinity. Right? Any number
1:04:01added by minus infinity is the minus
1:04:03infinity. And then what happens is that
1:04:06or in other words, it's kind of like you
1:04:07are just uh
1:04:08trying to
1:04:11add all of minus infinity to all of this
1:04:12part.
1:04:14And then when you add when you turn all
1:04:15of this into minus infinity, we'll do
1:04:17the soft max,
1:04:19this part just doesn't help doesn't
1:04:21change doesn't um
1:04:22like uh doesn't um by this part I mean
1:04:24like everything after T, right?
1:04:27So everything after little T, you you
1:04:29turn them into minus infinity and after
1:04:31you do the soft max they become zero.
1:04:34So because exponential to the minus
1:04:36infinity becomes like so so so so become
1:04:38basically zero. So
1:04:40that's why
1:04:42in this P, so everything after PTT
1:04:48everything after here will become zero.
1:04:50And the the sum of the rest will be
1:04:52what? So that's exactly what you how you
1:04:54achieve this because basically the
1:04:55linear combination of the rest of the
1:04:57terms
1:04:58doesn't matter because their
1:04:59coefficients are zero.
1:05:07Yeah, we are masking this because you
1:05:09shouldn't in this auto regressive model
1:05:11you don't allow
1:05:13the output at time T to depend on
1:05:16anything that is after T. So, you have
1:05:19to respect this kind of like a
1:05:21sequential dependency. So, um, you know,
1:05:24in generation, you you are not able to
1:05:26see the future tokens to generate the
1:05:27current token. So,
1:05:29um, so every token can only depends on
1:05:32every things before it. So, that's why
1:05:34we have to mask it out.
1:05:40Okay, we have
1:05:42Yeah, we are almost done here. Okay, so
1:05:45um
1:05:49Okay, so this is the so-called single
1:05:51head attention.
1:05:52Um, and we have to make it a little more
1:05:55complicated to even finish the attention
1:05:57part. So, then what we do next is that
1:06:02maybe I should erase here.
1:06:04Let's just say
1:06:13So,
1:06:14in some sense, what we are doing here is
1:06:16that we have H1 in
1:06:19and HT in and uh, we
1:06:22go through this single head attention.
1:06:24I guess uh, in the note, I call it uh
1:06:27attention 1H and you get this H out.
1:06:33H1 out
1:06:35up to HT out. That's what we defined so
1:06:38far.
1:06:39And uh,
1:06:40in practice, you have multiple copies of
1:06:42this single head attention with
1:06:44different uh, weight matrices. So, you
1:06:47get another copy
1:06:49of the single head attention.
1:06:51Maybe let's call this
1:06:54single head attention one, this is
1:06:55called single attention head attention
1:06:56two.
1:06:57And they have the same
1:07:01input
1:07:02and then you you get the
1:07:04uh, another copy of the output.
1:07:08But here, by copy I don't really mean
1:07:10literally copy because then it's not
1:07:11useful. So, what's the
1:07:13the difference between this and this is
1:07:15that the the weight uh are the weight
1:07:18matrices are different. So, basically
1:07:20this WQ
1:07:21WK WV
1:07:23for these two
1:07:25copies of the attention are different.
1:07:27So, so So, that's why they're not really
1:07:29literally copies. They are just uh uh
1:07:31like you have another
1:07:32um uh another version of it.
1:07:35And uh
1:07:36um
1:07:42Yeah, so So, in some sense you should
1:07:43index this by something else because the
1:07:45output will be different, right? So,
1:07:46maybe you should index this by
1:07:48um
1:07:51Let's say this is
1:07:53because this is the first attention you
1:07:54say H11 and HT1 and this is H12 HT2.
1:07:59And you also have another one
1:08:01and you get H1
1:08:033 up to HT3.
1:08:06See? So, So, when you have so many
1:08:09uh copies you have a several sequences
1:08:11of vectors and but eventually you only
1:08:13want one sequence of vectors. So, you
1:08:14have to combine them in some way.
1:08:16And the way to combine them is that you
1:08:18um
1:08:20uh you
1:08:23concatenate this
1:08:25three things. So, you concatenate
1:08:30H11 out
1:08:32H12 out.
1:08:36Just
1:08:37one I think that how many Each of these
1:08:40is called one head. So, you have let's
1:08:42say NH NH head.
1:08:45So, this is a attention
1:08:48and sub H.
1:08:50So, you concatenate all of these vectors
1:08:53and you get uh and then you multiply
1:08:54that with a another projection matrix
1:08:57and then this will give you the final uh
1:09:00H uh
1:09:01uh the output
1:09:03at one.
1:09:04So, that's the final output at position
1:09:07one. And you do the same thing for
1:09:08position two, where you concatenate
1:09:11uh
1:09:12maybe these guys, these guys.
1:09:16And uh and then you do a projection, you
1:09:17get the output at position two.
1:09:20So, in some sense, this is just give you
1:09:22more power to compute different kind of
1:09:24like uh
1:09:25um
1:09:26uh um
1:09:27kind of attention. In some sense, you
1:09:29know, I think actually in practice, if
1:09:31you
1:09:32I mean, the intuition is that maybe each
1:09:34of these attention is doing
1:09:35single-headed attention is trying to pay
1:09:38special type of like uh
1:09:40uh is doing some special type of job.
1:09:42Like, for example, maybe here it's
1:09:43trying to find out which entity in the
1:09:46past passage matters to me.
1:09:49And here, this is trying to find out,
1:09:52you know, which
1:09:53uh um uh what are the kind of like the
1:09:55sentiment
1:09:56of the past passages uh I I and and
1:09:59which part of the sentiment matters to
1:10:00me, so and so forth, right? So, each of
1:10:02these attention probably pay attention
1:10:03to different type of information. And
1:10:05then eventually, you aggregate the
1:10:07outcome of all of this information into
1:10:09one
1:10:10um big vector, and you project it into
1:10:13um uh a single output.
1:10:16um
1:10:19Cool.
1:10:22>> The second attention is the output of
1:10:24the first one, and therefore
1:10:25>> No, no, they are they are all parallel,
1:10:27yes.
1:10:27>> Parallel how far
1:10:28>> They are they are parallel, and then you
1:10:30aggregate. So, and all of this is in one
1:10:33of this.
1:10:34And then, so basically, in one of this,
1:10:36you have multiple parallel heads, and
1:10:38then next round, you have another number
1:10:39of parallel heads.
1:10:46I think the number of heads is uh maybe
1:10:48on the order of like maybe
1:10:50at most a hundred, I guess.
1:10:52um
1:10:53um let me think about this. I think
1:10:57something on that order, I think. Yeah.
1:10:59Yeah.
1:11:00You know, it also depends on the model
1:11:01size, of course, right? I mean, the the
1:11:02biggest one
1:11:04I don't know. I don't know whether 100
1:11:05is is for which model size. I think
1:11:07maybe for like a
1:11:09100 billion parameter models, you have
1:11:10100 heads. Yeah, something like
1:11:16Yeah.
1:11:23>> [clears throat]
1:11:36>> Yeah, I think um I I don't know exactly
1:11:38what's your proposal, but um
1:11:40um
1:11:41maybe it's This sounds like a simple
1:11:42simplified answer, but like a like a
1:11:43like many of these proposals has been
1:11:45considered
1:11:46in various ways, and and some of them
1:11:48are
1:11:49for and most of them are for efficiency
1:11:51perspectives, because eventually you can
1:11:52just have a
1:11:54like as we discussed with like a if you
1:11:55really don't care about efficiency, you
1:11:56can just have a generic very gigantic
1:11:59networks, right? So, all of these kind
1:12:01of like
1:12:02special structure is to try to
1:12:05use your parameters in the right way, so
1:12:07that you can
1:12:08use it more efficiently in some sense.
1:12:13Um
1:12:15Okay, so
1:12:19And then
1:12:20Okay, so I guess uh let's try to Let's
1:12:22just give me two or three minutes to
1:12:24talk about to con-
1:12:26talk about some very quick thing. So,
1:12:29one thing is that actually, when I draw
1:12:32this, it's actually a little more
1:12:33complicated than
1:12:35just doing this. So, because there's
1:12:37also a residual part in the in this kind
1:12:40of like how to combine MLP with
1:12:42attention. So, um so actually, what
1:12:45actually you really do is that you say
1:12:47you apply
1:12:49um So,
1:12:51after you do the uh attention, you
1:12:54um
1:12:57you do some kind of like normalization
1:13:00here.
1:13:01So, there's a normalization layer.
1:13:03Often, this is using the RSM norm that
1:13:05we briefly discussed in one of the
1:13:07neural network section.
1:13:08And then you apply MLP, and then you add
1:13:12the residuals back to this. You you you
1:13:14kind of like in the residual network
1:13:16that we discussed.
1:13:17And then you go to the next layer. And
1:13:19there are different ways to kind of like
1:13:20add the residuals. You can add the
1:13:21residuals after you do the
1:13:23>> [laughter]
1:13:24>> attention or or before you do the
1:13:25attention. So, there's something called
1:13:28pre-norm and the and post-norm
1:13:30things. So, I guess the details you you
1:13:32can check out the lecture notes. Um, so
1:13:34there's some kind of like orders on how
1:13:36you when you apply your normalization,
1:13:37residuals, and attentions, so and so
1:13:38forth.
1:13:40And and finally, I think one important
1:13:42thing, especially for the next lecture
1:13:45about the system ML um
1:13:47perspective, is that the we should try
1:13:50to quickly think about the computational
1:13:53efficiency here
1:13:54and the memory footprint.
1:13:57So, how many operations we have to do,
1:13:59especially as a function of the of T.
1:14:02Um, as a function of other things is
1:14:04actually relatively easier because as a
1:14:05function of the number of parameters,
1:14:07you know, like all of those is kind of
1:14:08like a it's all multiple kind of like
1:14:10linear in some sense. The main important
1:14:12thing is what is the the dependency on
1:14:14T.
1:14:15And you can see that
1:14:16for each of the T's position, you have
1:14:18to compute all of these inner products.
1:14:20So, for each of the position, you have
1:14:22to compute capital T inner product.
1:14:24So, and and you have capital T of these
1:14:27positions.
1:14:28So, that means that you have to
1:14:30um basically do a T squared dependency,
1:14:33right? So, so or in other words, you
1:14:35know, even this just this matrix itself
1:14:39is a T T by T matrix, capital T by
1:14:41capital T matrix. And each of these
1:14:43entry requires some inner product. So,
1:14:45so I think the dimension, if you still
1:14:47remember the Q eyes of the Q T's are in
1:14:51R D H and you need to compute T square
1:14:54of this inner product. So, basically the
1:14:56the number of operations is T square
1:14:58times D H.
1:15:00The dependence on the H is less
1:15:01relevant. The the you know, less
1:15:03important to some degree. The dependence
1:15:04on T square is very important because
1:15:06this means that if you have
1:15:08capital T being a million
1:15:10it's going to take a million times
1:15:11square square operations which is
1:15:13prohibitive.
1:15:15So, and this is one of the issues with
1:15:17the long context issues probably have
1:15:19heard of like a like I think we are
1:15:21using chat GPT or cloud code you see the
1:15:23context and then and then after some
1:15:24point they say, "Let me compact my
1:15:26context." right? And the reason is that
1:15:28you have to shrink the context otherwise
1:15:29your computational efficiency is too
1:15:31bad. Um uh but of course you know, in
1:15:33the in the in the real production system
1:15:35it's not like all of this capital T is
1:15:37really like there are other ways to
1:15:39reduce the dependencies. I think in
1:15:41practice I don't think necessarily it's
1:15:42T square it's probably more closer to
1:15:44linear in T um because there are other
1:15:47um ways variants of transformers that
1:15:49can reduce the dependency on T. Um I
1:15:51think after in the next Wednesday we're
1:15:53going to discuss a few um things that
1:15:55can reduce the dependency on T a little
1:15:57bit. Um exactly how you do it we don't
1:16:00really know in some sense because the
1:16:03uh there are a lot of open source papers
1:16:05that uh propose different approach. Uh
1:16:07um with at least I don't know exactly
1:16:10which one is used uh
1:16:11uh in the in open and a topic. Um um And
1:16:15another thing is that what is the memory
1:16:17usage here? So, the memory usage you
1:16:20know,
1:16:20as as at at the first side you have to
1:16:22spend T square memory
1:16:24because this matrix itself requires T
1:16:26square memory.
1:16:28But that actually can be reduced because
1:16:31you don't have to actually compute the
1:16:32whole matrix one by one and then do the
1:16:35multiplication with V. So, in some sense
1:16:37to some degree you can do some part of
1:16:39this matrix and multiply with part of
1:16:40the V first and then you don't have to
1:16:43save that part of the matrix. You can
1:16:44compute a second part. So, and and the
1:16:46idea is called flash flash attention,
1:16:48probably you have heard of, which is a
1:16:50way to reduce the the the memory
1:16:54footprint in this systems.
1:16:57Um So, yeah, I think that's
1:17:00but but you know, like some some of
1:17:02these kind of like ideas do have a
1:17:04trade-off because when you reduce the
1:17:06um for example, if you change your
1:17:08architecture to make the dependency on T
1:17:10squared better, for example, we'll
1:17:12discuss some of this um on Wednesday.
1:17:14So, if you change the architecture, the
1:17:16dependency on T is better, but your
1:17:17architecture is less expressive, and
1:17:19then you may lose some performance to
1:17:21some degree.
1:17:23Okay, sounds good.
1:17:24>> Thank you.
1:17:24>> Yeah, thanks.

About this transcript

This page contains the full transcript of YouTube transcript (pwQ0l4hFCVI) , generated from the public captions YouTube serves with the video. The transcript has 12,541 words across 2,159 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.