YT YouTube2Text

YouTube2Text → Transcripts → YouTube transcript (114i2Kz-LZA)

YouTube transcript (114i2Kz-LZA) — Transcript

13,846 words · 2,190 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:06Hello everyone and welcome to CME 295
0:10transformers and large language models.
0:13My name is Afin and I'll be teaching
0:15this class with Shervin who's in the
0:18back. So before we start, I just want to
0:21say a few words about ourselves.
0:25So Shervin and I are twin brothers and
0:28we have very similar backgrounds.
0:31So we both went to a school in France
0:33called Central Pari and then we came to
0:35the US for grad school. So on my end I
0:39went to MIT and then Shervin came here
0:41at Stanford in the ICM department and
0:44then after that we both went to the tech
0:47industry. So we first both went to Uber
0:51and then we went to Google and now we're
0:54both at Netflix.
0:56So Shervin and I,
1:00we like LLMs. So we've been working in
1:04the field of LLM, I would say since the
1:06early 2020s. And so that's one of the
1:08reasons why we're so lucky to be
1:10teaching this class. We've been teaching
1:13it since 2021 under various formats. But
1:17as attention gained towards LLM, we have
1:21actually made it an actual Stanford
1:24class since last year. And this session
1:28is the third instance.
1:31And so if you're here wondering if this
1:34class is good for you. So the goal of
1:37this class is to tell you how LLM's work
1:40and in particular understand the
1:43underlying mechanism of its
1:47architecture. So we're going to see the
1:48transformer but also how LLM are
1:52trained, how we can use them, where what
1:55they're good at, what they're not good
1:56at.
1:58So hopefully you'll be able to get a
2:00sense of all of that in this class.
2:03And in terms of the target audience, I
2:06would say it's a pretty wide audience.
2:08So in case you want to go and be a
2:12research scientist, I would say it could
2:14be a great class to just open up the set
2:16of things that you are able to see in
2:19terms of you know what are the different
2:20techniques that are being used, what are
2:22the different uh applications and and so
2:26on. So I think that's one. The second
2:29one is if you want to do a personal
2:32project. So nowadays coding agents are
2:35everywhere and so if you want to just
2:38take one tool and just do something, you
2:40should probably know what the coding
2:42agent is good at and not good at. And
2:45third, I would say no matter your role,
2:49there's a growing need of being literate
2:52in AI. So there's this term called AI
2:55literacy.
2:56So no matter where you work and your
3:00role, I would say just knowing how these
3:02tools work are be is becoming
3:05increasingly important.
3:07So if you feel like you're in one of
3:09these categories, this class may be good
3:11for you. And in terms of the
3:14prerequisites, I would say if you have
3:18some foundations in linear algebra, if
3:20you know what a matrix is, how we do
3:22multiplications,
3:23then that would be great along with some
3:26basics in ML. So for instance, what is a
3:28loss function? What is a neural network?
3:31What is an embedding? Although we will
3:33be seeing that uh in the class, but it's
3:36good to have those I would say
3:38beforehand.
3:41Make sense?
3:43Okay, cool. Uh, so you may or may not
3:46know that uh, this class is being
3:50recorded. So you can see the camera in
3:51the back. It was actually recorded last
3:54year. So you may be wondering, well,
3:56what is the point in attending the
3:58lecture if it was already recorded? So
4:00the field is moving so quickly that
4:04there's been so many new things that
4:06have come up since last year. And
4:09actually one of the goals of this class
4:12is to incorporate all the advances that
4:15have happened. So I'll name a few. So
4:18there has been uh
4:21some progress on the post-raining side.
4:23So for instance on policy distillation
4:26if you've heard of it uh of course AI
4:28agents so coding agents and so on and uh
4:33new paradigms of LLM. So traditionally
4:36it was predicting tokens one at a time
4:39but nowadays there's been an increasing
4:43number of models that have approached
4:45things differently. So from the
4:46diffusion perspective
4:48uh among other things.
4:51So long story short if you're attending
4:53this class and you're preparing for
4:55exams please look at this year's
4:57lectures because they will be quite
4:59different
5:01uh in terms of the logistics. So I'm so
5:05happy to see you know so many of you on
5:07a Friday afternoon. So it will be uh
5:10every Friday from 3:30 to 5:20. It will
5:14be in this class
5:17and then uh it is a twounit class. Uh
5:21you can either choose it to take it as
5:24letter or credit nonredit.
5:27So again all the lectures are recorded.
5:30We'll make sure to post the recordings
5:32at most two days after the lecture has
5:35happened. So feel free to watch it
5:39remotely if you want or attend the class
5:42uh in person. Actually we'd love to see
5:44in person. So uh feel free to choose
5:46either way. And this class
5:50has no homeworks
5:53but it has two exams. So one is the
5:57midterm and one is the final. So the
6:00midterm will be about the first four
6:04lectures and it will be one month from
6:07now I guess. Yeah, October 23rd and the
6:10final exam will be on the last five
6:13lectures. So there will be nine lectures
6:15in total
6:17and it will be accounting for 50% of the
6:20total.
6:22So given that it's not the first time
6:24that we're giving this class, we have
6:27already posted online what a midterm
6:30looks like, what a final looks like. So
6:32if you go on the website, you can
6:34actually see in the tab 2025
6:38what we would expect from this exam. So
6:39it's a multi multi-choice questions. We
6:42also have some free form questions. So
6:45this is the kind of question we would
6:46ask.
6:49Any questions on the logistics so far?
6:54Yep.
6:56So, uh I'm going to always repeat by the
6:58way questions just for uh people who are
7:01watching remotely. So, question is are
7:03there office hours? So, we don't have
7:05official office hours. However, Shervin
7:08and I will be staying after each class
7:10in case you want to ask any questions
7:12and uh we're going to see it actually
7:15here in case you have any uh so we're
7:19going to see it later if you have any
7:20questions there is also ED so platform
7:23where we can ask questions um so yeah
7:25love to answer there
7:29cool
7:31okay so in terms of the material
7:36so I was talking about the the midterms
7:38and and so on. So they're all uh so the
7:40ones from last year they're on the
7:42website. Um the website al also contains
7:45the syllabus
7:48and uh the recordings will be posted
7:50there. One thing that we're doing is
7:53that we're posting the slides one day
7:57before the lecture because last year
7:59some people came to us and said okay
8:01we'd love to be able to annotate on the
8:03slides. So now we took the habit of
8:07posting them one day before the lecture.
8:09So if you go on the website you can
8:11retrieve them.
8:13So in terms of the material we do have a
8:17textbook which is this one the super
8:20study guide transformers and LLMs. Uh I
8:23believe the library has a few copies so
8:25you can uh borrow from there. Uh you can
8:28also get it online. We got the website
8:31as well.
8:33And along with that, we also have a
8:35cheat sheet which aims at being the most
8:40upto-date. So Sharon and I actually
8:43updated it just uh a couple of weeks ago
8:47and that one aims at covering the whole
8:52content in a concise way. So maybe it's
8:55a good companion for studying right
8:58before each exam.
9:01And also I know a lot of us are
9:03international people. In case you don't
9:05see your language being translated, feel
9:08free to come to us. We'd love to partner
9:10with you. Uh so right now it's available
9:12in I believe 15 languages including
9:14English. So yeah, love to see more
9:17languages as well.
9:20Cool. Um, in terms of how we
9:22communicate, so we will post
9:24announcements on Canvas and you can also
9:29access ED through Canvas, but if you
9:32have any questions that you want to ask
9:34us in private, we can also um, you know,
9:37answer through the mailing list that is
9:39on the slides along with our Stanford
9:42emails.
9:45I think I'm done for the logistics. Is
9:47there any questions?
9:54Is it scary? Because we have we had only
9:56one question. Uh just wondering if
9:59there's any [snorts]
10:00that you may worry. Is everything clear?
10:02Super clear. Yeah. So the question is uh
10:06will we see how this ties back to what
10:09is practiced in the industry? The answer
10:11is yes. So one of the things we'll be
10:13seeing especially in the second part of
10:15this class is how we can apply those
10:17LLMs in different settings and so just
10:20as an anecdote uh so last year I
10:22remember there were some things that
10:23came up that were released as we were
10:26teaching the class and so we updated the
10:30content as we went so I'm sure we'll do
10:32the same thing this cycle as well. So
10:34yeah the answer is yes.
10:37Yeah. So the question is what is the
10:40recommendation for preparation given
10:41that there's no homework and it's 50/50
10:44midterm exam uh final. I would say
10:46depending on how you usually study I
10:49think just studying regularly I think
10:51works best. Um I would say just paying
10:54attention to things that we're covering
10:57in the class during the class I think is
10:59most important. So there may be topics
11:02that are in the textbook that we're not
11:04covering and I would say those are not
11:06expected in the in either exams. So
11:10roughly speaking anything we say can be
11:12in the exam.
11:15So question is do you have any practice
11:17problems? So yes. So last year we said
11:20no but now yes you have the 2025 ones.
11:22So yeah but the questions will be
11:26different. So yeah.
11:29Okay, great.
11:32So, one thing is we only have two hours
11:36or 1 hour and 50 minutes per week. So,
11:39we will have nowhere near the time to
11:41cover everything. So one thing that
11:43we'll be seeing in uh each of the slides
11:46that we'll be talking over is always I
11:50mean most of the time a source at the
11:53very bottom of the slides that is meant
11:55for you to check in case you're
11:57interested to learn more details. So
12:00feel free to refer to the slides in case
12:02you're interested in a topic that maybe
12:04we've covered for a few minutes but you
12:06want to learn more.
12:09Cool. Um, another thing, I mean,
12:12something that I personally have felt
12:13when I first started in the NLP field
12:17like a few years ago, there's so many
12:20abbreviations all over the place and it
12:23can be intimidating. So our hope is that
12:27in case you're feeling overwhelmed today
12:29by these abbreviations that by the end
12:31of this class you'll have a mental
12:34mapping of which abbreviation correspond
12:37to what concept. So that is our goal
12:42and with that we're going to start
12:46by asking ourselves how we got here.
12:51So 10 15 years ago in the 2010s
12:55so I remember uh the field was
12:59completely different. We didn't have one
13:02model that did everything. We had
13:04typically one model per use case. So the
13:07kind of model that was dominating the
13:10field was was RNN's recurrent neural
13:13networks which we'll briefly see in this
13:16lecture. And you had these mo use case
13:19specific models such as uh for sentiment
13:23extraction. So you asked yourself if a
13:26sentence a review is positive, negative
13:29or neutral. If you had a text you wanted
13:32to translate, you had a dedicated model
13:34for that. And if you had, I don't know,
13:36text you wanted to identify names,
13:39locations, and so on, you would also
13:41have a dedicated model for that.
13:44So this was the 2010s
13:46performance was okay
13:50but the thing that really
13:53made a true difference that was a
13:54turning point
13:56was 2017.
13:58So 2017 there's a paper that came out
14:02that we're going to talk about at length
14:04today that's called attention is all you
14:08need that introduced a new architecture
14:13that relied on a paradigm that proved to
14:16be very scalable
14:18by scalable I mean if you give your
14:22model more data more compute and more
14:26parameters you're actually able to see
14:29performance gains that are far greater
14:32compared to what you used to see before
14:35and it's something that we're going to
14:36see today. So this architecture is
14:38called the transformer
14:41and on the right you have an
14:43illustration of how it's uh illustrated
14:46in the original paper. We're going to
14:48see how it works.
14:50And from there people have taken this
14:54architecture and just scaled it from a
14:58model size perspective but also from the
15:01number of tokens it's been trained on.
15:04And this is how modern day LLMs have
15:08come into the picture. LLMs that
15:11actually know I mean that actually
15:14generate text pretty well that generate
15:16code pretty well. And in particular in
15:192022,
15:21chat GPT got released. So I remember in
15:24November I believe. Yeah.
15:28And again it was a turning point from
15:30more of a product perspective because
15:32before that date people were not really
15:34used to interacting with the chatbot. So
15:38I believe that release was pretty the
15:41first time that people got introduced to
15:43that aspect and I think even the person
15:47or the company who launched it probably
15:49didn't even realize how big it would
15:51get. I mean I'm assuming because in
15:53their tweets that released the the model
15:57I believe they had like a typo which uh
15:59I find funny because nowadays whenever
16:01there is a model release
16:04you know there's 10 times review of what
16:07the wording is so I'm assuming that this
16:10was not the case for for the
16:12announcement here but yeah so that is
16:15also another turning point
16:17and then here we are 2026
16:22I'm assuming a lot of you or all of you
16:26are interacting with coding agents
16:28almost on a daily basis. So we're going
16:30to see how we got to that stage. How we
16:33went from a conversational
16:36LLM conversational assistant to actually
16:40agentic behavior.
16:43So we're going to see that especially in
16:45the second part of this class.
16:51So far so good.
16:54Cool.
16:56So with that, we're going to start with
17:01one foundational concept which is
17:04tokenization.
17:06So what is tokenization?
17:09So when we talk about text,
17:12text is re not really something that
17:14models can understand. models understand
17:18numbers. They don't understand text.
17:22So the first goal is to find a way to
17:27represent that text in forms of
17:29something that models can understand. So
17:31in forms of numbers.
17:34So the way that we do that is we first
17:37try to divide the text into entities
17:42that we can then represent.
17:46And so the process of dividing the text
17:48into indivisible entities is what is
17:51called tokenization.
17:53So in this example we have a cute teddy
17:56bear is reading
17:58and one possible tokenization is to say
18:02that a is one indivisible unit, cute is
18:08another unit. Teddy bear can be another
18:10unit and so on and so forth.
18:15So each of these elements
18:18is called a token
18:21and our goal
18:23is to represent these tokens with
18:26mathematical representations that are
18:29meaningful.
18:30So in this case we want each token to be
18:33represented by a vector, a vector of
18:36numbers.
18:40So,
18:42in order for you to tokenize an input
18:45text, you need to know what are the
18:48tokens that are allowed for you to be
18:51using. And so, we have something that's
18:53called a vocabulary,
18:55which includes all possible tokens that
18:58you can use.
19:00And depending on the vocabulary that you
19:03choose,
19:04you may end up with different sequence
19:08length. So here by sequence lengths I
19:11mean the number of tokens by which you
19:14divided your your text. So for instance
19:15in this instance we have 1 2 3 4 5 six
19:19six tokens.
19:21We're going to see in a second that
19:23there's a tight relationship between
19:27vocabulary size and sequence length. And
19:29that's something that will inform which
19:32tokenizer we should be choosing.
19:38So far so good. So now you may be
19:41wondering well all of that is great but
19:43how do I know how I tokenize my text?
19:47Like this is just a theoretical uh
19:50concept but how do I do that? So one
19:53natural way is for you to say okay so
19:56I'll just tokenize the text with respect
20:00to words. So I'm going to consider space
20:03as being a delimiter between
20:06tokens. So here I would say a cute teddy
20:09bear is reading. So every word in that
20:11text would actually represent
20:13a token.
20:16So what are the pros and cons of this
20:18approach? So pros
20:20very simple especially for the English
20:22language. You map a token to its words.
20:26Very simple. It is also interpretable as
20:29in you know when you get a
20:31representation of a word that it
20:33corresponds or representation of a token
20:36that it corresponds to a given word.
20:40However
20:42at least in English
20:45you have many variation of words. So for
20:48instance
20:50a noun like you can have it is singular
20:54in plural you can have the masculine
20:56form the feminine form you can have many
20:59many variations so your vocabulary size
21:03may get very large as a result of that.
21:05So that's one con.
21:07The second con is we're going to see in
21:10a second that when we learn the
21:13representation of each token,
21:16there is nothing that guarantees us that
21:19two tokens that represent the same thing
21:22would get the same representation. So
21:25for instance, if you consider the word
21:28bear and the word bears,
21:32these are the singular and plural of the
21:34same animal which is a bear. And so you
21:38wouldn't want the representation of
21:39these two tokens to be too far from one
21:42another because they represent the same
21:44thing.
21:46Well, with this way of tokenizing your
21:48text, you would have a hard time first
21:53of all leveraging the knowledge that you
21:56have from the similar token and second
21:59just guaranteeing that these
22:01representations are similar.
22:04And then the third one is
22:07well given that you need to learn the
22:10representation of each of these words.
22:13So one word is one token. You may run
22:16into a case where if you train your
22:21tokenizer on a given corpus of text that
22:25you actually do not have a given word
22:28that you then encounter
22:31at inference time.
22:33So you may actually be missing
22:35representations that you could not have
22:38had the chance to learn just because you
22:40did not encounter that word at training
22:42time. And therefore you can run into the
22:46risk of
22:49out of vocabulary meaning that
22:52a token
22:54is actually not something that you have
22:55seen at training time and you don't have
22:57the representation for that one.
23:00So long story short, word level
23:03tokenization is simple but it comes with
23:06a few cons
23:08and one of the cons is really the fact
23:12that you are not able to leverage words
23:16of similar roots and this leads to
23:19vocabulary being large and risk of out
23:23of vocabulary.
23:27So the next question is okay can we do
23:30better? Is there another way of
23:31tokenizing the text that could make more
23:34sense? So a natural thought could be
23:39well you don't divide by words you
23:42divide by character.
23:45You have a fixed number of allowed
23:47characters at least in in English or in
23:49any languages. And so
23:53you are not running the risk of out of
23:56vocabulary anymore because you will
23:58encounter the all the letters in the
24:01alphabet in your training set or you can
24:03try to do that
24:05and by doing so you will also have a
24:07small vocabulary
24:10and you would also be robust to casing
24:15and misspellings that people usually
24:17make in when I mean I'm not sure if you
24:20recognize yourself. I do. So when I type
24:22uh I don't know on uh my computer, I
24:25sometimes misspell things and I wouldn't
24:27want the representation of a word I
24:31meant to write to be different from the
24:33actual word. And so here we would not be
24:37going into the case of
24:41not recognizing a word just because it
24:42is misspelled.
24:46But we will see that
24:49if you do that the sequence length is
24:54going to get much longer
24:58and model complexity is a function of
25:02sequence length. So if the sequence is
25:05very long then your model will take a
25:08long time to process the sequence. So it
25:10will make computations much slower
25:14and the embeddings themselves they're
25:18much less interpretable as in you know
25:20if we tell you the letter A has this
25:23representation it it's maybe much more
25:26abstract than talking about word
25:28representation
25:31and we will see that in a second.
25:36So far does that make sense?
25:39Yeah.
25:41So I hope
25:44I am motivating a trade-off enough that
25:47the next natural thought in your mind is
25:49to actually try something in between. So
25:51not word, not character, but in between.
25:54So subwords. So, how about we tokenize
25:58the text
26:00by leveraging
26:03knowledge that we have about words that
26:06we can learn from
26:08some training corpus and we divide
26:13common occurrences of letters and we
26:17represent them with a token. So for
26:18instance for
26:20reading we would represent read with a
26:23token and ink with another token.
26:26So how about we would do that. So the
26:29good thing here is that we would
26:30leverage the root of words. So in my
26:32previous example bear and bears would
26:36have the for instance bear token in
26:39common.
26:41And then the second thing is it would be
26:43actually learned from the data as in the
26:46way you tokenize text is informed by how
26:50words are actually written.
26:53But then the con is well you need to do
26:55that extra work. So in the word level
26:57case you just need to produce tokens by
27:01just divi dividing based on space. So
27:03here you do you need to do some extra
27:05work and then
27:09the way you obtain these tokens will be
27:11highly dependent on
27:14the type of data that you have when you
27:17do your tokenizer training. So that's
27:21another thing another con. So we need to
27:23make sure that the data that you use to
27:26learn how to tokenize things is actually
27:29aligned with your end goal.
27:35So um I just want to uh say something
27:39here. So I talked about three ways of
27:42tokenizing text. So word level,
27:45character level, subword level. Among
27:49the three, subword level is by far the
27:52most popular nowadays for all the
27:54reasons that I mentioned. And you'll
27:57hear the word or the acronym BPE a lot.
28:02So BPE is a type of subword tokenizer
28:06that stands for bite pair encoding. So
28:09the paper is linked at the bottom of
28:11this slide in case you're interested to
28:13learn more, but I'll give the just the
28:15gist of how it works. So, BPE
28:19learns what tokens should be in your
28:22vocabulary by starting from an initial
28:26set of tokens and
28:32identifying the most frequent pairs
28:35in your in your training data and by
28:38coming up with dedicated tokens for
28:43merge tokens of these pairs. So for
28:45instance, I'll give an example. Let's
28:47imagine your initial vocabulary is all
28:50the letters in the alphabet and you have
28:54I don't know the word
28:56un like an idea and
29:01an an amazing course. So un that occurs
29:04very frequently. So here it will say
29:07okay a and n appear very frequently
29:10together. So let's actually add a token
29:12that merges the two.
29:15it will add that new token in the
29:17vocabulary and then repeat the process
29:20until reaching
29:23a desired final vocabulary size and that
29:27will be your final vocabulary.
29:31So the nice thing is that these tokens
29:33are learned from the data. So things
29:35that appear together are actually
29:37identified and added to your vocabulary.
29:40So this will lead to your sequence
29:42length
29:44being shorter. So you will not have to
29:45decompose your text into that many
29:48tokens. So your model will not take
29:52that much time to process your text. And
29:55so that's where the efficiency come from
29:57comes from. So if you have a training
29:58corpus that has let's say a lot of the
30:02words that you encounter in practice
30:04then the sequence length of the tokens
30:07you will get from a piece of text that
30:11uses the same words as what you had in
30:13your training corpus will be much more
30:16much shorter.
30:19So in case you're doing multilingual
30:23work, then make sure that your tokenizer
30:25is trained on a corpus that's represents
30:29all the languages that you're interested
30:30in. That's one example.
30:35Okay, so that was a lot. So I'll just
30:38pause for a second. Is there any
30:39questions so far on
30:43this part?
30:46Yes.
30:47So uh the question is what is the the
30:51relationship between BP and engram? So
30:53engram is actually something separate
30:55and this one yes to your point it will
30:58learn the tokens from your corpus by
31:02merging the most frequent pairs in an
31:04iterative way until reaching a final
31:07desired vocabulary size.
31:10So next natural question is okay you
31:13have your tokens then now how do you
31:15know how to represent them? Maybe you're
31:17asking yourself this and we will see
31:19that actually in about five minutes. So,
31:22more on that in a bit. Actually, I'm
31:25looking at the time. I don't think I'm
31:27on time. So, I'll probably move on. But
31:30if you have any questions, please feel
31:31free to interrupt me.
31:34Tokenization is really the building
31:36block of everything we'll see from here.
31:39So, yeah.
31:42Okay. So, so far we've seen tokens that
31:46represent text that we want to tokenize.
31:49But the truth is we also have some
31:51special tokens. And these special tokens
31:54are meant to represent things that we
31:57want to express in a sequence. So one
32:00such token is whenever we encounter a
32:03term or a piece that is actually not
32:06something that is part of our
32:08vocabulary. So the convention is to have
32:11a dedicated token that represents this
32:14out of vocabulary piece of text and it's
32:18called an unknown token.
32:21That's one special token. So give you an
32:24example teddy bear. Imagine if I write
32:26it teddy d bear and in the training
32:30corpus we've never seen such a spelling.
32:35Then
32:37if your vocabulary is only composed of
32:39these four tokens, so teddy, cute, bear,
32:41and unknown, then teddy bear will be
32:44tokenized as unknown token and then
32:48bear.
32:50One example,
32:52another set of special tokens. So a lot
32:57of the times you need to indicate when
33:01the sequence is starting, when it is
33:02ending. And we'll see how we do that. So
33:06we have two set of special tokens. One
33:08is beginning of se sequence bos and then
33:12end of sequence EOS. Those are mostly
33:16used when we want to generate text. So
33:19for instance when when you want to start
33:22the generation the convention is to feed
33:25your model with the beginning of
33:27sequence token just to indicate to your
33:29token to your model that you you want to
33:31generate text and your model will then
33:35at the end of its generation just output
33:38the end of sequence token to just tell
33:40you that this generation is finished.
33:44So this is how it's used. So what a cute
33:47teddy bear would be beginning of
33:49sequence with a cute teddy bear and end
33:51of sequence as an example. And then the
33:53last one that you may encounter is the
33:56padding token and that one is meant to
34:00make your sequences be of a certain
34:04size.
34:05So this is sometimes used for efficiency
34:09perspectives
34:11if you want all your sequences to be of
34:13the same length because we will see that
34:15we can represent that with vectors,
34:17matrices, tensors and hardware loves
34:21things that have the same dimension then
34:25padding can also be used.
34:28But I want to say that this list is by
34:31no means exhaustive
34:34and that you also have a lot of
34:37different conventions that people take.
34:40So this way of writing BOS, EOS, this
34:44way of writing padding, it's not
34:46universal.
34:48So just don't don't be surprised if you
34:50encounter different variations. And
34:53nowadays you have things like
34:56assistance. So you have a user,
34:59assistant and those are other things
35:01that people represent with special
35:03tokens. So again this is not an
35:06exhaustive list but just know that such
35:09tokens exist and they are meant to
35:12convey something about the text that the
35:14model should understand.
35:22Cool.
35:24That that's it for tokenization. Any
35:28questions?
35:30Yes. The question is what is the order
35:32of magnitude of the vocabulary size for
35:34lens right now? So it's a great
35:36question. It is on the order of hundreds
35:40of thousands. So you will see something
35:42like hundreds of thousand 200,000
35:45something like this. And these are
35:47tokenizers that are able to handle both
35:50English but also other languages.
35:54And as I mentioned, the most common type
35:58of tokenizer nowadays is the BP one. So
36:00yeah, so these are the things that are
36:02good to have in mind. Yeah.
36:06So the question is having multiple
36:09languages be handled by your tokenizer.
36:11Does that make your tokenizer a bit less
36:13of an expert in a given language? I
36:15think it is hard to say. I would say
36:18because there may be some synergies
36:21between languages. So I would say it's
36:24hard to say in general, but that being
36:27said, the vocabulary size that you fix
36:31when you when you run your BP algorithm
36:34will be filled with tokens from
36:37different languages. So you may get a
36:42vocabulary that may lead you to longer
36:47sequence length if you're only
36:50interested in one particular language.
36:52So from an efficiency perspective maybe
36:55you'll you'll see something but then
36:57from a representation perspective it's
37:00not super clear. So one example is
37:03um acronyms of organizations that may be
37:07something that people talk about in a
37:08bunch of languages. So you may benefit
37:11from learning that representation from
37:16different text of different languages.
37:18So pros and cons.
37:24Great.
37:26Cool. So now what we know what we have
37:29learned is given an input text we're
37:33able to generate tokens or represent
37:37that text with tokens. So the next
37:39natural question is how to represent
37:42those tokens. So this is what we're
37:45going to see now.
37:47So the most naive way to do that is to
37:51say that each token is different.
37:55So it's extremely obvious but you can
37:57represent your tokens by what we call
38:00one hot encodings. So here uh let's say
38:03I have the the token soft I can say okay
38:06it's the one 0 0 vector
38:11and if you do that for all your tokens
38:14well the bad thing is they're all going
38:17to be orthogonal to each other if you
38:20take the dot products.
38:23So, is it a good thing?
38:28Right? No, it's not a good thing because
38:30it doesn't tell you anything about what
38:33these tokens are and how they're related
38:36to each other. So, it's not what you
38:37want.
38:39What you want is to represent these
38:42tokens in a way that reflect how similar
38:47they are to one another.
38:50That is what you want. So for instance,
38:52you want teddy bear to be similar to
38:56soft because teddy bears are soft, but
38:58maybe book to be maybe independent. This
39:02is what you want to learn. You want to
39:03learn meaningful representations.
39:06And by the way, why do you want to
39:09learn meaningful representations? Well,
39:11it's because you want to do something
39:12with your text. You need to have it as
39:14input into your model in order to do
39:16something as output. So for instance
39:19generating the next words
39:21predicting some class. So you need to
39:24understand what is in your text
39:27which is the reason why what you want is
39:31meaningful token representations.
39:35So this
39:38is what an earlier method called v word
39:43tovec
39:44try tried to achieve. So here the idea
39:49is
39:52to learn these representations
39:54through what we call a proxy task. So a
39:57proxy task is a task that you're trying
40:00to learn but it's not the task that
40:02you're trying to learn as an end goal.
40:05It is is a task that you try to learn in
40:07order to achieve something else.
40:10And here the proxy task is something
40:12like predicting the word that is in
40:16between some context words. So this is
40:19the continuous bag of words version of
40:21vorttovec
40:23or predicting words that are surrounding
40:27a given word that is the skip gram
40:29version. So we'll go through an example
40:32so that it becomes clear.
40:34So does everyone know what a neural
40:36network is?
40:38Yeah.
40:40Okay, so I represented a very basic
40:42neural network. So you can think of it
40:45as an input matrix multiplication to get
40:50a hidden representation of size D and
40:53then again matrix multiplication to go
40:55back to size V your vocabulary size. So
40:58let's assume
41:00that you have the the sentence a cute
41:03teddy bear is reading. And let's assume
41:06for the sake of simplicity that your
41:08goal is to predict cute from the word a.
41:16So let's imagine that your proxy task is
41:18to predict the next word.
41:22So here a
41:25your very naive representation of a
41:29is the one hot encoding. So let's assume
41:31that you have a vocabulary size of six
41:35and a is your first token. So you would
41:38represent a with 1 0 0 0.
41:43You pass it through your network. So
41:45think of it as matrix multiplication
41:48with this vector. you would have a
41:50hidden state of size two. Let's assume
41:53two numbers 02 and 0.9
41:57and then you pass it to your network. So
41:59again another matrix multiplication and
42:01then you have predictions that sum to
42:04one. So you can use something like a
42:06softmax to normalize them. And your goal
42:11is
42:14to
42:16have these probabilities be as close
42:20to the label that you have which is
42:22predicting the word cute. And so cute
42:25let's suppose that it's the second word
42:26of your vocabulary. So it is represented
42:28by 0 1 0 0.
42:32So your goal is like on the slide for
42:35the 0.4 to be as close as one as
42:38possible.
42:41So the idea is to have a loss function
42:45that penalizes
42:48the fact of having your probability of a
42:51given word being too far from the word
42:53that you're trying to predict. So in
42:55case you know the cross entropy loss
42:58and then updates the weights of your of
43:00your model in order to reflect what you
43:04want.
43:06So just so that we're clear I will just
43:09walk through all the quantities that we
43:11mentioned. So here the one hand encoding
43:14that represents the token is of size the
43:17vocabulary size.
43:19The hidden state or also called latent
43:22representation
43:24is of size D and D is actually a
43:28parameter that you can choose typically
43:30much smaller than the vocabulary size.
43:33And then the predicted word
43:34probabilities well you have a
43:36probability per token. So it needs to
43:38match the vocabulary size. So it's of
43:41size V. And that also matches the size
43:44of the onehood encoding. And this is
43:46where you're able to
43:50try to penalize for deviations from the
43:52word that you're trying to predict. So
43:55let's assume you do that
43:57for a trying to gen to predict cute. You
44:01repeat the same thing for cute for teddy
44:04bear. So you go cute, you represent it
44:07with the 100 encoding. You pass it
44:09through your network. You have a latent
44:10representation and again probabilities.
44:13You compare that with the actual label
44:16that is representing teddy bear. Compute
44:18the loss. Update the weights given how
44:21far you are from the from the predict
44:24from the label. And you repeat that
44:28through your whole corpus.
44:30Well, at the end of this,
44:33what the authors realize
44:36is that the latent representation that
44:39you learn in the middle of the network
44:42actually is
44:45making sense.
44:47As in,
44:50you can compute similarity measures
44:54between the learned representations. And
44:56you see that representations of similar
45:00tokens are actually similar.
45:05So on this example, so teddy bear is
45:07soft. What is Persian poetry to arts?
45:09Another example is Paris is to France,
45:12what Berlin is to Germany. So you have
45:14these nice associations
45:16that are actually something that the
45:18model learns.
45:20So that gives you meaningful
45:24word representations.
45:27So that's a method
45:30that appeared I believe in the early
45:322010s. I believe it was 2013 if I
45:35remember correctly. So who can tell me
45:37what is a a limitation of this approach?
45:45So here the representation of the words
45:47that we learn
45:51we saw that they end up being meaningful
45:54but is there
45:56is there a limitation to this approach?
45:59Yeah. So one answer is that you would
46:02have trouble if you actually are not
46:05encountering something that you have
46:06seen during um your training process. So
46:10yes, that is right and I think that's
46:12right in actually most cases beyond that
46:14case in particular. Um so that's one. Is
46:17there another yes another limitation?
46:20So the answer here is uh the dimension
46:24scales up when the the vocabulary size
46:27gets bigger. So actually this is a
46:29limitation of your tokenizer because the
46:32tokenizer is the one that determines
46:34your vocabulary size. And but you're
46:38right that um that this is a limitation
46:41but it's not specific to that approach
46:43as in you will always need to find a
46:46representation of
46:48each element in your vocabulary. So I
46:51mean it's a limitation but limitation in
46:54methods in general.
46:57Yes. Oh yes. So answer is words can have
47:00multiple meanings. So exactly. So um one
47:04limitation here is if the word let's say
47:08cute
47:10is in this sentence
47:13versus in another sentence they will
47:16both get the same representation. So
47:19if you say a cute teddy bear in a nice
47:21way versus someone who sarcastically
47:25says oh how cute.
47:28They may mean different things between
47:29the word behind the word cute. So I
47:32think a very
47:36classic example is bank. So river bank
47:39versus going to the bank. These are the
47:41same words but different meaning. So the
47:44problem with this approach is that
47:47the representation of this word would
47:50actually be the same regardless of the
47:53sentence that it is in. So great point.
47:56Another limitation here is that the word
47:59order does not matter.
48:02So let's assume we're saying the child
48:05is hugging the teddy bear and the teddy
48:08bear is hugging the child. These are the
48:11exact same words, but they mean
48:14different things. We would get the same
48:16embeddings with this approach in both
48:18cases.
48:21So these are all limitations
48:24which is the reason why this is
48:27something that people have tried to
48:30resolve through different means.
48:33One such means is
48:37RNN
48:38which as we previously saw was the type
48:42of model that was dominating
48:45the models back in the 2010s.
48:49And so here RNN's what they do is that
48:54they actually have an extra quantity
48:56that they compute which here is
48:58represented with a
49:01that is also called the hidden state
49:05that tries to encapsulate
49:07the meaning of the sentence seen so far.
49:12So we're going to see that as an
49:14example. So let's assume we go with the
49:16same sentence. So a cute teddy bear is
49:18reading. So here you would actually
49:21put the word a into the RNN. It would
49:25try to predict the word cute. But then
49:28what it will do is keep
49:32what we call the hidden state that
49:34encodes the meaning of the sequence seen
49:37so far.
49:39That will be an input
49:42to the next prediction. So we'll have
49:45cute as an input and then these hidden
49:49states as an input as well in order to
49:52inform the next representation
49:56and so on and so forth. So here what
49:59you're doing is you're not just
50:01considering embeddings of tokens on
50:04their on their own. You're also
50:07considering
50:09the embedding of the sequence decoded so
50:11far.
50:13And that allows you to
50:16take into consideration
50:18the order of the words
50:21in the sentence.
50:24And RNN's are actually models that were
50:26used in a bunch of tasks back in the
50:28day. So for instance, for
50:30classifications, what you would do is
50:33process tokens one at a time and then
50:35keep a selfmutating state that encodes
50:38the meaning of the sequence seen so far
50:41and it would try to predict let's say a
50:44sentiment in the sentiment extraction
50:46case.
50:47In the multi-classification case, for
50:50instance, if you wanted to determine if
50:53a word was a noun or a verb, then you
50:55would actually leverage the
50:58representations corresponding to each
51:00input, which again would take into
51:03consideration the hidden states
51:06or just generation.
51:11But one
51:13limitation of RNN's is that okay it was
51:17great you would consider the order at
51:20which the words would appear in a
51:22sequence
51:24but it would have trouble
51:27remembering
51:29words that happened far in the past
51:33because what it used to do was to encode
51:36the meaning of the sequence
51:38in one single vector.
51:41that kept on being modified.
51:44So you had this issue of what we call
51:47long range dependency
51:50where if you end up two sentences after
51:53something that you have talked about
51:56prior to that, you would have trouble
52:00teasing that out just using the hidden
52:02state at that point.
52:06And that can be problematic. For
52:08instance, when you refer to something
52:10that happened in the past. So for
52:12instance, you can say, "Oh, my teddy
52:15bear is so cute. Uh, it is
52:193 ft tall." So it refers to something
52:22that is in the past. So now let's assume
52:24that the it is referring to something
52:27that is far in the past.
52:30then we have this problem of actually
52:33having the right set of information in
52:35the hidden state that we're considering.
52:40So this is the reason why there were
52:43some variations of RNN.
52:46One such variation was called the LSTM
52:50long short-term memory which tried to
52:53address this problem by not only keeping
52:57a hidden state but also another state
53:00called the cell state that was meant to
53:04carry information for longer.
53:10But the truth is
53:13even
53:14such variations had their limitations
53:18which again had the same issue.
53:22So if we were to just recap the state of
53:24the ways we could obtain our embedding
53:28so far
53:30we saw vortac which is a very simple
53:33approach to compute
53:36representations of words. But the
53:39problem is that
53:41if a word is let's say in a different
53:44sequence then the representation is the
53:47same. So that is a limitation and also
53:50as mentioned previously
53:53the same word can actually mean
53:55different things and so that is also not
53:58conveyed and in the case of RNN's we are
54:03actually addressing the word order order
54:06problem by having a hidden state that
54:09encapsulate the sequence that is seen so
54:12far.
54:13But the problem is that we have this
54:16long range dependency problem that is
54:18actually coming from a deeper problem
54:20come called the vanishing gradient
54:22problem which is a a problem when you
54:26want to back propagate
54:28that just introduces some limitation
54:31and another thing that we have not
54:33mentioned but RNN's the problem is every
54:37time you want to predict the next token
54:40you need to have
54:42the hidden state that encapsulates all
54:44the things that you have seen so far. So
54:47you always have to
54:50just go through tokens one at a time in
54:52order to make the next prediction. And
54:54when it comes to training such a model,
54:57it is just extremely slow
55:01and
55:03it had great results at the time but not
55:07amazing either.
55:10So for all these reasons in the 2010s
55:14people tried different techniques
55:17and there is one technique that really
55:21stood out now from hindsight but that is
55:25called attention
55:27and the idea is as follows.
55:31Instead of solely relying on the hidden
55:34state to encapsulate the meaning of the
55:37sequence that you have seen so far,
55:40the idea is to let your model have
55:45direct connections
55:47from tokens that you have seen in the
55:49past.
55:51So in other words, we're not saying only
55:54rely on the hidden state to know what
55:56has happened in the past. We're saying
55:58you can also have direct connections to
56:02tokens that you have seen in the past in
56:04order to predict the next token.
56:08So in the translation example, so let's
56:10assume you want to translate in French a
56:13cute teddy bear is reading and let's
56:15assume you say
56:18and you want to predict the next words.
56:21So intuitively in order to predict the
56:24next words,
56:26it would be amazing to know what the
56:29word that you want to translate is.
56:32So in the traditional RNN case, you
56:35would not have access to that direct
56:37access to that word. You would only have
56:39access to it via the hidden state. So
56:42here the idea is to have a direct
56:44connection to previous tokens and let
56:48the model learn which ones matter and
56:52here for the prediction to rely on the
56:54ones that matter in order to make the
56:57prediction.
57:00So this is the concept of attention.
57:04The concept of attention is letting your
57:06model draw direct connections from a
57:09given token to some tokens in the past.
57:16And
57:18this concept is very important because
57:21it is at a found at the foundation of
57:23what we're going to see now.
57:27So I mentioned previously that there was
57:31this paper called attention is all you
57:34need that came out in 2017 that was a
57:37turning point because people saw that it
57:40was it was very scalable. It was giving
57:43you amazing results.
57:45Well, this paper
57:48relies on the concept of the self
57:51attention, which is attention but on
57:55steroids.
57:57And we'll go through exactly what we
58:00mean by that. So let's assume you have
58:03the sequence a cute teddy bear is
58:06reading.
58:08So now the idea is to not say that we're
58:10going to compute the representation of a
58:12given word as a function of
58:16the hidden state that encapsulates the
58:19meaning of the sequence so far and the
58:22word itself. We're not saying that now.
58:24What we're saying is that in order to
58:27compute the representation of a given
58:28token, we we can actually do that by
58:32expressing that representation as a
58:34function of all the other tokens.
58:37in that sequence.
58:39Sorry. So I think you raised an
58:42important point which is how do you know
58:46the order of words if you just let
58:48everyone interact to everyone in case
58:50that was your question. Um so if you let
58:54a token interact with all other tokens
58:56you actually lose
58:59the concept of word order and it is
59:03actually something that is addressed
59:04later on. So we're going to exactly see
59:06that part. Uh but here you actually do
59:09not know where tokens are are because
59:12you have a direct connection in this
59:14very vanilla representation. You don't
59:16know where tokens are. So we're going to
59:18see exactly how you fix that issue.
59:22But does the idea make sense? The idea
59:25that in order to compute a
59:27representation of a token, you actually
59:30want to express that as a function of
59:32all the other tokens in the sequence.
59:34Does this idea make sense
59:40roughly?
59:43Yeah, I'll take that as a guess. So,
59:45we're going to go through examples and
59:47actually Shervin has a detailed example
59:49later on. So, hopefully that will make a
59:51bit more sense.
59:53There is a set of notations that people
59:56use
59:58that we're going to introduce now. So
1:00:00these notations are queries,
1:00:04keys and values Q, K and V.
1:00:11So
1:00:13the query
1:00:14is the entity that is trying to
1:00:17determine what a given token is about.
1:00:22And the idea is
1:00:25it wants to find tokens that are useful
1:00:29in order to construct a meaningful
1:00:31representation of the token that it is
1:00:33responsible for. So the query for teddy
1:00:35bear what it tries to do is to find
1:00:39other tokens that is relevant in order
1:00:41to compute a meaningful representation
1:00:43of that token.
1:00:45So what it will do is
1:00:48see how similar that quantity is to keys
1:00:54of the sequence that are associated with
1:00:56each token in the sequence.
1:01:00And once it determines the magnitude at
1:01:03which it is similar to each token,
1:01:06we will associate a value to each of
1:01:10these tokens.
1:01:12So in other words, we want to express
1:01:15the representation of teddy bear as a
1:01:17function of all the other tokens in the
1:01:19sequence. So what we will do is first
1:01:22determine the weights. So it's a
1:01:24weighted average. We want the weights.
1:01:25The weights are determined by how
1:01:28similar the query for teddy bear is to
1:01:30each key.
1:01:33And then the actual value will be just
1:01:36the value vector corresponding to each
1:01:41each token. So it will be something like
1:01:43if I were to write it. So it's like
1:01:47the sum of something that is
1:01:50representing the similarity. So Q Q and
1:01:54KI
1:01:58VI.
1:01:59So this is your query. You're
1:02:03trying to determine how similar your
1:02:05query is to all the other keys with some
1:02:08similarity operation. So here's the dot
1:02:10product as an example. And you associate
1:02:13that. So this is a scalar.
1:02:15This is a scalar
1:02:17and you associate that with vi which is
1:02:21also a vector.
1:02:25So
1:02:27this may seem a little bit abstract. So
1:02:29I'll give you an example. So let's
1:02:30assume we want to know what best
1:02:33describes the teddy bear. The query can
1:02:35be asking itself, okay, what best
1:02:37describes a teddy bear? And you look at
1:02:40all the tokens, you're like, oh, cute
1:02:44actually describes the teddy bear pretty
1:02:45well. So maybe the dot product between Q
1:02:50teddy bear and K cute is going to be
1:02:54high.
1:02:56So the representation that you compute
1:02:58for teddy bear is actually something
1:03:00that will depend heavily on cute let's
1:03:03assume.
1:03:06So these are the concepts of query key
1:03:09value. Now the next question is how do
1:03:11you obtain those? How do you obtain
1:03:13queries key values?
1:03:16So these are actually quantities
1:03:19that you can learn with projection
1:03:22matrices.
1:03:25So you initially have some input token
1:03:28that represents each
1:03:31tokens token and what you do is you take
1:03:35that representation
1:03:37and you project it on the space of
1:03:40queries of keys of values
1:03:45through some projection matrices.
1:03:49And then you use these quantities in
1:03:52order to compute meaningful
1:03:54representations
1:03:57of tokens.
1:04:00Yes.
1:04:02Amazing question. So question is what is
1:04:03the t? So t is actually just to be
1:04:06mathematically rigorous. So t is
1:04:08transpose. So it's just that uh this you
1:04:11can also write it as
1:04:15Q krpose
1:04:25any other question
1:04:28so we have a detailed example later so
1:04:30in case that roughly makes sense I think
1:04:33that's great
1:04:36we we'll see that in a
1:04:40So it just turns out that what I
1:04:41described is something that is very
1:04:45nicely something you can express in
1:04:47mathematical terms
1:04:50with matrices.
1:04:52So in particular the operation I just
1:04:55mentioned is something that can be
1:04:56expressed as the soft max of the query
1:05:00and the keys
1:05:02and you scale that by some factor times
1:05:05the values
1:05:08and actually I maybe have enough time
1:05:11for you to say why.
1:05:14So let's assume you have [snorts]
1:05:17so the softmax is just a way for you to
1:05:19get the sum of the coefficients being
1:05:23equal to one. So that's roughly what it
1:05:25does. But the question is well if you
1:05:28have query key transpose v I mean more
1:05:35or less the softmax and the scaling.
1:05:38Why does it give you the thing that we
1:05:40saw above?
1:05:42Well, let's assume that your query
1:05:45matrix matrix has
1:05:48like all your queries. So, q1
1:05:52qn.
1:05:54So, one is the first index of your
1:05:57sequence. So, let's assume in the
1:05:58previous slide that we say the first
1:06:00index is a. Let's say a is one and then
1:06:05the dot is n. Let's assume.
1:06:08So we have
1:06:10the query for A
1:06:13up until the query for the dots
1:06:16and you do the same for
1:06:19K and V.
1:06:22So if you multiply
1:06:25the square matrix
1:06:28times transpose
1:06:30what you get is
1:06:34K1 like this
1:06:38KN.
1:06:43And so this is where you're glad you
1:06:46know some linear algebra.
1:06:49So matrix multiplication. So you do this
1:06:52times this
1:06:54which is the dotproduct of Q1 K1
1:07:00uh and then second element is Q1 K2.
1:07:11So if you repeat this you will have the
1:07:14matrix of the Qi
1:07:17KJ
1:07:20going to maybe write that up up there.
1:07:32So I'm just going to write exactly
1:07:34what's in what's what is above. So you
1:07:38have
1:07:47Q1 K1
1:07:51up until Q1
1:07:54KN.
1:07:58And then you repeat that here. You put
1:08:00that here. You repeat that here. Now if
1:08:02you add a V in here
1:08:07then
1:08:12what you obtain is
1:08:16V1
1:08:19VN.
1:08:22So again you do your matrix
1:08:25multiplication and here what you get is
1:08:29this times this plus this times this
1:08:31plus this time this up until here. So
1:08:34you have the sum
1:08:37of Q1,
1:08:42KI,
1:08:45VI,
1:08:48and the same up until the sum of QN,
1:08:55KI,
1:08:58VI.
1:09:01So
1:09:03long story short, I just wanted to say
1:09:06that
1:09:08the fact of expressing
1:09:12something as a function of this
1:09:16similarity operation and the values is
1:09:18something that you can represent very
1:09:20well with matrices
1:09:24and you will see this formula quite
1:09:25often. So I believe there was a question
1:09:28on the midterm. So hint this formula can
1:09:31be important.
1:09:35And so now that we know how self
1:09:38attention is defined, how self attention
1:09:40works, now we're going to hopefully in
1:09:43the next 17 minutes
1:09:46see how self attention is used in the
1:09:50transformer architecture which is a
1:09:51trans which is the architecture that was
1:09:53introduced in the attention is all you
1:09:56need paper.
1:09:58So what why is it called attention is
1:10:01all you need?
1:10:03Well the authors at the time most models
1:10:09were recurrent. They were based on the
1:10:11RNN. So they used to compute things in a
1:10:14recurrent fashion.
1:10:16But here what we're saying is actually
1:10:18we don't care about keeping a hidden
1:10:23states by recurring through tokens.
1:10:27We're not doing that. What we're doing
1:10:29instead is letting tokens interact with
1:10:33all tokens
1:10:35through direct connections
1:10:38aka through self attention
1:10:42which is why attention is all you need.
1:10:46So this paper was released in the
1:10:48context of machine translation.
1:10:51So in this context, we're interested in
1:10:53translating a source text, let's say in
1:10:55English, over to another language, let's
1:10:58say in French.
1:11:00And what we do is we have two components
1:11:03in the transformer.
1:11:06On the left, we have what we call an
1:11:08encoder.
1:11:10And the role of the encoder is to
1:11:13compute meaningful representations for
1:11:16the source text that is fed as input.
1:11:21And we have a second part of the
1:11:23transformer which is called the decoder.
1:11:26And the role of the decoder is to use
1:11:29the meaningful representations from the
1:11:32encoder along with some internal
1:11:36embeddings that it is computing in order
1:11:39to generate
1:11:41the translation.
1:11:45So that is the context in which the
1:11:47transformer was released and it
1:11:50performed great and it is at the
1:11:53foundation of modern day LLMs which is
1:11:56the reason why we're focusing on this.
1:11:57So that that's the reason why we're
1:11:59here. The reason why I I wrote these
1:12:02formulas on the blackboard is because I
1:12:05think this is probably the most
1:12:07important formula when it comes to
1:12:09transformers. So just visualizing this
1:12:12in terms of matrix multiplies
1:12:16is very important.
1:12:19Okay. So what we will do now is walk
1:12:22through each component of the
1:12:24transformer and see how this all comes
1:12:27together. And I'm looking at you with
1:12:29respect to the position embeddings.
1:12:31We're go we're going to talk about this
1:12:33now. So
1:12:36for machine translation
1:12:39what you get as input is a text in a
1:12:42source language
1:12:44and your task is to translate that
1:12:47source language that source that text in
1:12:49the source language into a target
1:12:51language.
1:12:53So your first task is to understand what
1:12:58your source text is about.
1:13:00So here you have an input embedding that
1:13:04is tasked with just converting this
1:13:08input text into tokens and then
1:13:10associated associating each of this
1:13:13token with some embedding that you can
1:13:15learn. This embedding is only
1:13:18with respect to each token. It is not
1:13:21context aware in any way.
1:13:24And what you do is you add an embedding
1:13:28that is associated with the position
1:13:30that the token is in in the sequence.
1:13:33And this is where we learn we know where
1:13:36tokens are located.
1:13:39So we add the token embedding
1:13:41and the position embedding together.
1:13:46And so the question you may have is well
1:13:48how do you obtain the position
1:13:49embedding? So you have different ways.
1:13:52The ways people like the authors have
1:13:56tried is actually two ways. One way is
1:13:59to learn
1:14:02the embedding. The second way is to
1:14:04actually have something that is a
1:14:06function of uh just different
1:14:09frequencies. So there are some pros and
1:14:12cons but I just want to so I think the
1:14:15the learning way is very natural but a a
1:14:19clear con is if you have an input text
1:14:22that is longer than what you have
1:14:24learned at training time. So if you have
1:14:27let's say a thousand tokens as input but
1:14:31you already you only have learned
1:14:33positions like embeddings for positions
1:14:35up to let's say 500 then you're out of
1:14:38luck. You cannot represent positions
1:14:40greater than that. Which is why the
1:14:43arbitrary predefined representation can
1:14:46be useful.
1:14:48And I just want to illustrate why
1:14:51choosing
1:14:53different frequencies could be a good
1:14:55way to represent something like the
1:14:57position.
1:14:59So you may have a watch and you may
1:15:02notice that you represent the time with
1:15:05let's say an hour
1:15:08hour uh thingy with the minutes they all
1:15:12turn at different frequencies.
1:15:14So if you want to read the time so 449
1:15:17it's four which is the the hour piece
1:15:22and then you have the minutes and then
1:15:23you have the seconds and each of them
1:15:26they turn at different frequencies and
1:15:28if you put them together you can exactly
1:15:31know what time it is.
1:15:34So this is the idea behind having a
1:15:37vector of elements that vary with
1:15:40different frequencies. So you can obtain
1:15:42that with cosine and sine functions and
1:15:46this is what people use here. So the
1:15:49good thing with these representations is
1:15:51that positions that are
1:15:56close to one another
1:15:58are very similar and the ones that are
1:16:00far from one another are are dissimilar.
1:16:02So this is one nice property.
1:16:05So this is one way of representing the
1:16:07positions.
1:16:08Anyway, maybe I digress too much. We're
1:16:10going to talk more about position next
1:16:12time, but long story short, we can
1:16:16represent tokens along with their
1:16:18positions. So at this stage, what we did
1:16:21was add the token embedding and the
1:16:25position embedding.
1:16:27And here we come into the encoder.
1:16:30So the encoder is the component of the
1:16:33transformer which is tasked with
1:16:35computing meaningful representations of
1:16:37the tokens.
1:16:39So here we have one very important type
1:16:43of layer called the attention layer.
1:16:47It's actually called the self attention
1:16:48layer. So self attention layer, what
1:16:50does it do? What it does is compute
1:16:55representations of tokens as a function
1:16:58of all other tokens.
1:17:01And this is where this query key value
1:17:03thing happens.
1:17:06So you're going to see that in the
1:17:08example that Sherman will walk you
1:17:09through. But the idea is to learn
1:17:12projection matrices
1:17:15that will allow you to project your
1:17:18tokens into a space of queries, keys,
1:17:22and values that will then allow them to
1:17:25interact
1:17:27through the formula that we saw. So this
1:17:32so that allows you to represent your
1:17:35tokens as a function of others.
1:17:38And then you have another layer called
1:17:42the feed for neural network which is a
1:17:46way for your architecture
1:17:48to project representations into spaces
1:17:53of higher dimension
1:17:56along with some activation function
1:17:58which allows you to learn
1:17:59nonlinearities.
1:18:04So you go through the encoder at the end
1:18:06of the at the tip of the encoder what
1:18:08you get is a very meaningful
1:18:10representation per token
1:18:12that is a function of all of the tokens
1:18:14in the sequence.
1:18:18So what you do is now when you want to
1:18:20translate the text
1:18:23so you you take your special token that
1:18:25we saw in the very beginning you know
1:18:27end of sequence sorry beginning of
1:18:29sequence. So you tell your model well
1:18:31now I want to start the generation. So
1:18:34you input begin sequence as an input to
1:18:38the decoder. You add the position
1:18:40encoding that is associated with that
1:18:43token which is the first element in the
1:18:45sequence and what you do is
1:18:50you enter the decoder
1:18:53which does two things.
1:18:56The first thing is it also has an
1:19:01attention self attention mechanism
1:19:04with respect to the token itself and all
1:19:07the tokens that was generated so far.
1:19:12So it tries to express each token as a
1:19:16function of itself and all the tokens
1:19:18that were generated so far. So in the
1:19:21beginning of sequence case it's the only
1:19:23token that was generated so far. So that
1:19:24that's it. And then you have a second
1:19:30attention layer which is a different
1:19:32kind
1:19:34where you're actually letting the
1:19:37representations
1:19:38from tokens that were generated so far
1:19:43and let them interact with tokens from
1:19:46the source
1:19:48from your input source from the source
1:19:50language.
1:19:52So I'm not sure if you can see in the in
1:19:54the illustration you have one arrow
1:19:57which is from what was generated so far
1:19:59and then you have two arrows coming from
1:20:01the encoder.
1:20:03So the one error from the generated so
1:20:06far is the query
1:20:09and the key and the value are coming
1:20:11from the encoder
1:20:12because you're you're asking yourself
1:20:15how can I express this token as a
1:20:18function of all the tokens in the input
1:20:21input text
1:20:24by the way does that make sense? Yeah.
1:20:27Okay. Great. And then you do that and
1:20:29then you have another layer that's
1:20:30called the fit for neural network that
1:20:32again allows your model to learn
1:20:36ways to project your data
1:20:39and then at the end you get a very
1:20:43meaningful representation of the word
1:20:46that you the token that you passed as
1:20:48input. So here in this case in the very
1:20:50beginning you have beginning of sequence
1:20:52and at the end of the decoder you have a
1:20:56very meaningful representation of that
1:20:58token.
1:21:00So what you do is you then project that
1:21:02representation on the space of
1:21:04vocabularies
1:21:07and you get probabilities
1:21:10that correspond to what you think is the
1:21:13next token.
1:21:17And so here
1:21:20translation is just figuring out what
1:21:22the next token is. So you figure out
1:21:24what the next token is and then you
1:21:26choose what that next token is based on
1:21:28the probabilities and then you repeat
1:21:30that in an auto reggressive way. So you
1:21:33put that new decoded token as input of
1:21:36the dec decoder and you repeat exactly
1:21:38what I mentioned.
1:21:42So, we have five minutes. I have some
1:21:47tricks to tell you, but I want to make
1:21:49sure we're all on the same page. So, I'm
1:21:51just going to open the floor in case
1:21:52there is any questions so far. We we've
1:21:55seen a lot of things, by the way, and
1:21:58this first lecture is probably the one
1:22:00of the most difficult in terms of number
1:22:03of concepts we see. So, completely
1:22:05normal if you feel like we've seen a
1:22:07lot. Any questions on this?
1:22:12Yep.
1:22:13So the question is there are multiple
1:22:15arrows merging into one. Uh so it's
1:22:17because we have not seen it so far. So
1:22:19this is a trick that actually we can see
1:22:21now. So you may be referring to this
1:22:24one, the residual connections. So this
1:22:26is actually a trick to help your model
1:22:30learn better. So these residual
1:22:32connections, they're very useful in
1:22:36architectures that are deep because they
1:22:38allow your model to pass in features
1:22:41directly instead of having them be
1:22:46modified by some sub layer. So one way I
1:22:49like to think about this is instead of
1:22:52thinking about each layer as something
1:22:55that modifies the whole
1:22:58the whole input. So instead of saying
1:23:02my layer is just changing the input. So
1:23:05let's assume your input is
1:23:09let's assume your input is x. Instead of
1:23:14saying that the output of a layer is
1:23:16some function of x.
1:23:18What you're saying is the output of your
1:23:22layer is
1:23:24some function of x plus x.
1:23:27And the way I like to think about it is
1:23:29it's as if you were taking X and
1:23:32modifying it as opposed to changing
1:23:35completely X.
1:23:38So in practice doing that
1:23:43helps with back propagation
1:23:47and allows your model to learn better.
1:23:52So this is what this arrow is about.
1:23:54Yeah. So speaking of the tricks, I have
1:23:57some other tricks to tell you. So one
1:23:58other is the fact of normalizing
1:24:02the activations with what what we call
1:24:04layer normalization.
1:24:06So it also helps with convergence.
1:24:10Um, you may have seen the word masked
1:24:14self attention in the decoder. And the
1:24:16reason why it's masked is because you
1:24:18want your decoded tokens to only
1:24:21interact with tokens you have decoded so
1:24:24far
1:24:26because you cannot interact with tokens
1:24:28you have not decoded yet.
1:24:32Uh, multi-head attention. So what I
1:24:33described with QV, well, you can do it
1:24:36with
1:24:38in different ways. you can learn
1:24:39different projections.
1:24:42So for those of us who also are familiar
1:24:44with vision related models, who knows
1:24:47who who has a vision background here?
1:24:51Okay. One. Okay. Two. Okay. So in case
1:24:54you know uh when it comes to
1:24:56convolutional neuronet networks
1:24:59when you do the convolution operation
1:25:02you can use several filters which are
1:25:05different ways of convoluting on your
1:25:07input. So having different attention
1:25:10heads is also a way for your model to
1:25:12learn to have different ways to learn
1:25:15the same thing through through different
1:25:17uh projection matrices. And so you
1:25:19typically have h such
1:25:22heads.
1:25:25So another trick dropout. Who has who
1:25:27has heard of dropout? Okay, great. Yeah,
1:25:30this is a uh an idea that is not just
1:25:35applied in the transformer but widely
1:25:37used. So that one is a technique for
1:25:40which at training time you intentionally
1:25:42drop some units
1:25:45with some probability in order for you
1:25:49to have your model to not rely too much
1:25:52on some features and that allows your
1:25:56model to generalize better.
1:26:00And then what do I have? Yeah, label
1:26:02smoothing is another technique that was
1:26:03used here. So the idea behind label
1:26:05smoothing is when you tell your model to
1:26:09predict the next words instead of saying
1:26:15let's say what a nice day instead of
1:26:17saying okay I want you to predict the
1:26:19word day 100%.
1:26:21Your model is saying you can predict the
1:26:23word day actually not 100% you can have
1:26:26also other words because in language you
1:26:30can complete things in different ways
1:26:32and still be valid. So you can say what
1:26:34a nice day, what a nice class, what a
1:26:38nice
1:26:39evening. Anyways, so label smoozing
1:26:43takes
1:26:46these
1:26:48labels and it softens it by saying okay
1:26:52so instead of your model needing to
1:26:55predict that label
1:26:58as in you need to predict that model
1:27:00100%. you say okay actually
1:27:06the label that I want you to predict is
1:27:08maybe I don't know 90%. And then the
1:27:11remaining 10% is all the other words in
1:27:13the vocabulary.
1:27:16So it just allows your model to
1:27:19generalize a bit better and authors have
1:27:21found that this also improves metrics
1:27:25when it comes to machine translation. So
1:27:28uh for those of you who know blue is one
1:27:30such metric
1:27:34and with that I'll give it to Shervin.
1:27:38Thank you Afin. So in this last part our
1:27:42goal is going to piece all that Afin
1:27:44talked about together in one single
1:27:47example and see exactly how this
1:27:49computation works in a precise example.
1:27:53So as a mentioned the original
1:27:56transformer is in the business of
1:27:58machine translation. So what we're going
1:28:00to take is a sentence in English and
1:28:04we're going to try to translate it into
1:28:06some other language. So in the example
1:28:08of the original transformer paper they
1:28:10took French and German. So let's take
1:28:12that and let's start with our favorite
1:28:16toy example. So a cute teddy bear is
1:28:18reading. And as you recall, the first
1:28:21step in all of this is tokenization.
1:28:24And Afin mentioned that it's a way to
1:28:28arbitrarily split these words into
1:28:31atomic entities. So Afin mentioned
1:28:34subword is an optimal way to do it. But
1:28:38for the sake of this example, I'm going
1:28:39to take another split just to prove that
1:28:42it's arbitrary. So let's say we split it
1:28:45like this. As a mentioned, we have also
1:28:48special tokens. So, beginning of
1:28:50sentence and end of sentence sequence
1:28:54that you put to characterize where your
1:28:56sequence starts and ends.
1:29:00Now, let's look at how we encode them.
1:29:03So,
1:29:05we have in this architecture a lookup
1:29:09table that is made of learnable weights.
1:29:12So this is what we call embeddings and
1:29:15these will be learned as part of the
1:29:17learning process
1:29:19and as someone mentioned it here if you
1:29:23want to perform attention in [music] a
1:29:24way that's context aware and that you
1:29:26know where other tokens are positioned
1:29:29you need that position information and
1:29:31the original transformer paper puts this
1:29:35position information at the beginning of
1:29:38of the input. So you add it additively.
1:29:42So let's say you call your embedding of
1:29:45size D model. Then your position
1:29:48embedding that I've been described by a
1:29:50series of signs and cosiness of
1:29:52different frequencies will be a vector
1:29:55of the same length and it's additive
1:29:57operation.
1:30:00So at this stage you end up with an
1:30:02embedding that's at least half
1:30:06learnable. So the lookup table that I
1:30:08mentioned to you is at least learnable.
1:30:10Maybe the positions as well depends on
1:30:12your design choice and this is what you
1:30:15will put as an input to your encoder.
1:30:19So what happens next? Every token has
1:30:23this representation and you can
1:30:25represent it in a nice matrix form.
1:30:28Okay, let's go with it. And you're going
1:30:31to go into the encoder and as we have
1:30:34seen the first component of the encoder
1:30:36is going to be that self attention
1:30:38layer.
1:30:41So
1:30:42you project
1:30:44this input into the space of queries. So
1:30:47for that you have this WQ projection
1:30:51matrix. You get queries with the same
1:30:54initial embeddings. You project it on
1:30:57keys. So you have a WQ W K projection
1:31:02matrix that you will learn to project it
1:31:04into keys K and you do the same with
1:31:08values V.
1:31:12Okay, great. And then based on that you
1:31:16apply the formula that we mentioned that
1:31:18is so important that softmax formula and
1:31:22we do this matrix product and then we
1:31:25obtain uh another matrix that is of same
1:31:29dimension as the matrix V.
1:31:34So before we dive deeper into more
1:31:37specifics, I just want for us to get on
1:31:40the same page of on dimensions because
1:31:44we talked about a lot of embeddings
1:31:45here. We project things and I just want
1:31:48to make sure that we speak the same
1:31:49language as to what size are these
1:31:53objects.
1:31:55So your input sequence, so we mentioned
1:31:59it's of length n. And one notation that
1:32:03people use to characterize these
1:32:05embedding size is D model.
1:32:09And then what the projection matrices of
1:32:12Q, K and V do is respectively you start
1:32:16from the Dodel world and you turn it
1:32:20into the query world and then key world
1:32:24and then value world. So this is why
1:32:26it's of size D model and DQ and
1:32:30similarly D model DK D model DV for each
1:32:35projection matrix
1:32:37and you obtain these queries keys and
1:32:40values that are of size n rows but that
1:32:44are this time into this query key and
1:32:47value space. So the number of columns is
1:32:50DQ, DK and DV.
1:32:54And since you do this matrix multiply,
1:32:57you need this constraint that DQ is
1:33:00equal to DK. So here I put DQ and DK
1:33:04just for the sake of properly naming
1:33:06things. But DQ should be equal to DK by
1:33:09construction.
1:33:12And this is what you get.
1:33:15All good so far.
1:33:18Okay, great. So now let's pause a second
1:33:22and then go back to this example and see
1:33:26how the matrix matrix computation work.
1:33:29So as Ain was mentioning earlier each
1:33:33token can be represented as a row
1:33:37embedding
1:33:39and you have that for Q if you consider
1:33:42K transpose each column is going to be
1:33:46the embedding of a given vector. So when
1:33:49you do the matrix multiply, you're going
1:33:52to end up with a matrix of size n by n
1:33:57where each cell is going to be some
1:34:01similarity like here the exact dot
1:34:03product between a given query and a
1:34:08given key.
1:34:11And let's suppose here you multiply this
1:34:14by v. What you're going to end up with
1:34:17is for each row you have a given query.
1:34:21You project it on the space of keys that
1:34:24you have in your sentence. So you get
1:34:27some so you want to get some probability
1:34:29distribution over keys and then you're
1:34:32going to use these weights
1:34:35to
1:34:37to to weight the values and then you're
1:34:40going to do a sum of all of this. Now
1:34:42you're going to say okay Shvin and Ashin
1:34:44you talk about qkranspose v qkranspose v
1:34:48this is not what the formula is and then
1:34:49you are right the formula is
1:34:52approximately this we take the soft max
1:34:54in order to get a probability
1:34:56distribution that sums to one and has
1:34:59nice properties and one thing we haven't
1:35:02mentioned but that the authors chose is
1:35:05that you will normalize by some factor
1:35:08because the dimension dk K will
1:35:12influence the variance that you might
1:35:14get as part of the sum. So if you want
1:35:16everything to be properly normalized,
1:35:19you can divide by the standard
1:35:21deviation. And this is what happens
1:35:24here.
1:35:26And so as we mentioned this is a
1:35:30multi-headed process. So you do not do
1:35:32this process once, you do it eight
1:35:34times. So this process of learning
1:35:39queries, keys and values is done in
1:35:41parallel and then at the end you
1:35:43concatenate all these softmax of KQ uh Q
1:35:48krpose over square root of DK * V
1:35:52together and then you have a last
1:35:55projection matrix that we call WO that
1:35:58will turn these embeddings in a space H
1:36:02DV back into the model.
1:36:07So you have this nice property by the
1:36:09way in the transformer where a lot of
1:36:11the operations make the original
1:36:13dimension of embeddings invariant. So
1:36:16you start with D model, you do a bunch
1:36:18of attention calculations, you end up
1:36:22with D model and then you go into the
1:36:24next layer which is a fitforward neural
1:36:26network that is a one hidden layer
1:36:30network that projects this D model
1:36:33representation into a DFF1 and then
1:36:36typically you would have this dimension
1:36:38that is bigger than the model in order
1:36:41to allow for learning complex
1:36:43representations.
1:36:46And you do all of that big end time. So
1:36:50in the original transformer paper, big N
1:36:53is equal to six. So you do that a bunch
1:36:55of time. And then at the end
1:36:58of the encoder, you have for each token
1:37:01a representation that represents that
1:37:04token in a context aware and learned
1:37:07way.
1:37:10Is everything good so far?
1:37:14Okay, awesome. So, I'm asking because
1:37:16we're going to switch to the decoder.
1:37:18So, I just want to make sure all makes
1:37:19sense.
1:37:22So, now that we have that,
1:37:26as we mentioned, the original
1:37:28transformer is in the business of
1:37:30machine translation. So, you need to
1:37:31start the beginning of your translated
1:37:34sentence. And one way to force the start
1:37:37of it is to have this placeholder token.
1:37:41So you add this POS token, look at the
1:37:44lookup table of your embeddings, you
1:37:46have an initial embedding, you add your
1:37:49position embedding to it, and then you
1:37:51pass it to the decoder.
1:37:53And you have this self attention layer
1:37:57that Ashin mentioned is a masked self
1:37:59attention layer because what happens
1:38:01here is that it's causal. Every token
1:38:04looks at itself and the tokens that
1:38:07precede it.
1:38:10And
1:38:12once that happens, you have another
1:38:14attention layer that's called cross
1:38:16attention where the query is the decoded
1:38:21token and then the keys and the values
1:38:25are the tokens that come from the
1:38:27encoder. So you look at the stack of
1:38:29encoders, you look at the output
1:38:31embedding and then every layer of
1:38:34decoder like every decoder block will
1:38:38get that last embedding as keys and
1:38:40values.
1:38:42Okay, so you do that and then you have
1:38:44just like in the encoder a layer of fit
1:38:47forward neural network that allows for
1:38:49complex feature learning
1:38:52and all of that goes and then you have
1:38:54some linear layer and then a softmax
1:38:58layer that will turn these
1:39:00representations into a probability
1:39:03distribution over the space of your
1:39:05words of your of your tokens over the
1:39:07vocabulary space to predict the next
1:39:10word.
1:39:11And then what happens here is that we
1:39:13will see in the next lectures what could
1:39:16be heruristics to pick the next word.
1:39:18But let's say here we pick like the max.
1:39:21So you pick the max, you know what word
1:39:23we have predicted and then you put it
1:39:25back as an input to the decoder and you
1:39:28do the exact same process again. So here
1:39:33we put beginning of sequence, you get
1:39:36and then you you get the next token
1:39:40which is the equivalent of teddy bear
1:39:42[snorts] and so on
1:39:44and then you get at the end your
1:39:48your your sequence in the target
1:39:50language and then you do that until you
1:39:53hit an end of sequence token. This is
1:39:57where you know that you should stop the
1:39:58translation process.
1:40:01Okay, so this is how we start from a
1:40:05from a sentence in a given language and
1:40:06we end up in one in the target language.
1:40:10So
1:40:12any questions on this so far?
1:40:18Yep.
1:40:20The comment here is it look it all looks
1:40:22complicated. If you were to boil down
1:40:24exactly what makes it work, what would
1:40:26it be? So attention is all you need. The
1:40:29title of the paper translates a bit the
1:40:32secret sauce. So the attention layer
1:40:36here is key. You we're going to see in
1:40:38the next lectures that actually a lot of
1:40:40these things are dropped and we rely on
1:40:43a subset of these components that we
1:40:47repeat again and again. So uh this
1:40:49attention layer is is the key. And then
1:40:51the second thing is this feed forward
1:40:53neural network where most of the
1:40:55parameters of the network resides. So if
1:40:57you were to calculate exactly the
1:40:59parameters of your model, you realize
1:41:01that these contain most of the capacity
1:41:04and this is where the learning happens.
1:41:06So if you were to ask me these two and
1:41:08then the rest are just tricks to better
1:41:12learn to yeah improve it and then of
1:41:15course the quality of the data which is
1:41:17another layer of complexity.
1:41:20Yep.
1:41:22Yeah. So uh the question is can you talk
1:41:24more about the masked part? So it's a
1:41:29computational trick where you have a
1:41:31triangular matrix that will mask some
1:41:34connections of your self attention layer
1:41:38and at training time it prevents
1:41:40connections that you would you shouldn't
1:41:43know to be made. So yeah that that's the
1:41:47that's the gist of it.
1:41:49Yeah exactly. So yeah, how the question
1:41:51is how it relates to soft max. So you
1:41:53have minus infinities and then this will
1:41:54be zero. Like soft max of minus infinity
1:41:57would be zero. This is how it relates.
1:42:00Yeah.
1:42:01Yeah. But these are more computational
1:42:04tricks. I think the main thing to have
1:42:07in mind is to forget about all
1:42:10connections from a given token and all
1:42:12the ones that succeeded.
1:42:14I think that's uh you know good enough
1:42:16as a mental model.
1:42:19Yeah.
1:42:20Okay. Awesome. Yeah. The question here
1:42:23is on various uses of softmax. So the
1:42:27ones that are used in the the self
1:42:30attention layers and the cross attention
1:42:32layers have a different function than
1:42:34the one at the last layer. You should
1:42:37see the ones in the attention layer as a
1:42:39projection of the query over the space
1:42:41of keys. So what we do is projecting the
1:42:45representation of a given token across
1:42:48tokens whereas the last softmax layer
1:42:51has a different function. It's one that
1:42:53tries for you to predict the next token.
1:42:57So in some sense uh yeah so I I do get
1:43:00the I think it's a great point that
1:43:03we're using like several times this
1:43:05function but it's for different things.
1:43:09I think that was Yep. So the question is
1:43:12why is the transition not having
1:43:14beginning of sequence? It does. Yeah,
1:43:16it's the first token here. Oh yeah. So
1:43:19the first token here is a placeholder
1:43:22for you to predict the next token that
1:43:24comes after it. So it doesn't come at
1:43:26the output of the decoder. It's just the
1:43:28input of the decoder. And then what you
1:43:30get as the output of the decoder that
1:43:32you should watch out is end of sequence
1:43:35because this is where you should stop.
1:43:38Yeah.
1:43:40Okay. Awesome. And then with all that, I
1:43:43want to tell you that
1:43:46what we presented today is just the
1:43:48beginning. We're just in 2017 at this
1:43:51point and there has been, as I'm sure
1:43:54you know, many uses of this technology
1:43:57that now powers all these chatbots that
1:44:00we use. So we're going to cover how we
1:44:02train such models in the next few
1:44:05lectures. And then in the second part of
1:44:07the class, we will tackle how these can
1:44:11power the agents that we interact and
1:44:13use today and how it can be used in
1:44:16several systems. And with that, thank
1:44:19you for your attention and I hope you
1:44:21have a great weekend.

About this transcript

This page contains the full transcript of YouTube transcript (114i2Kz-LZA) , generated from the public captions YouTube serves with the video. The transcript has 13,846 words across 2,190 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.