YT YouTube2Text

YouTube2Text → Transcripts → Transformer Arch Decoder Inference [with Paper & Pen] -How Transformers ACTUALLY Generate Text Part3

Transformer Arch Decoder Inference [with Paper & Pen] -How Transformers ACTUALLY Generate Text Part3 — Transcript

by yogesh murumkar · 9,388 words · 1,766 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:01Hello everyone. This is Yogesh Murumkar.
0:04In this video, we'll discuss the
0:05Transformer architecture during
0:07inference or prediction phase.
0:10Before this, in the last two videos, we
0:13have already discussed about encoder and
0:15decoder architecture during training
0:17phase.
0:18And for your reference, the links for
0:20those videos are given in description.
0:24And to explain decoder architecture
0:26during inference phase,
0:28I'll use all these diagrams. You can
0:32have a look at all these diagrams
0:35one by one.
0:37Which will make the things uh
0:39easy and simple for understanding.
0:44So, you see everyone, these are the
0:45detailed diagrams which I'll be using.
0:48And the link for downloading all these
0:51diagrams is also given in the
0:54description.
0:56So, you can see,
0:57this is what I'll be
1:00using to explain you all the concepts.
1:05Fine?
1:08So,
1:10to start with everyone,
1:13this is our Transformer architecture
1:14diagram.
1:16Okay?
1:17Before starting with the explanation of
1:18Transformer decoder during inference
1:20phase,
1:21I'll recommend you,
1:23please refer previous two videos on
1:26Transformer encoder and Transformer
1:28decoder architecture, where you can get
1:30the complete understanding of how
1:32encoder
1:33this particular part and how decoder
1:35this particular part works during
1:38training. The link for both the videos
1:40is given in description.
1:43Now,
1:45we have not
1:47yet discussed how does the Transformer
1:50architecture behave during inference
1:53time.
1:56Or during prediction time. Inference
1:58means prediction. So, the topic of
2:01today's video is exactly this.
2:04Since we have read the entire
2:06architecture and got the idea of all the
2:08components, in this video, we will
2:10discuss that once your Transformer model
2:13is trained and after that, when you do
2:16prediction through it, how does the
2:18Transformer architecture behave during
2:20that prediction time?
2:22So, the topic of today's video is
2:25Transformer
2:27during inference.
2:29Okay? I really hope you understand the
2:31plan.
2:32Now, I'll click quickly tell you
2:35setup of this video. The setup is very
2:37simple.
2:38You can assume that
2:40uh let's say
2:41you had to solve a machine translation
2:43task.
2:44And the translation task is very simple,
2:46which is English to
2:48Hindi translation.
2:50So, for example,
2:52you have data set given.
2:55Okay? So, we have a machine translation
2:57task, which will be
2:59converting English sentence to Hindi
3:01sentence.
3:02Okay? So, you have the data set that
3:04looks something like this.
3:08Fine. You have English sentence in your
3:10input and Hindi sentence in your
3:13output.
3:15And you have many such rows.
3:19As you can see, let's say here the first
3:21English sentence is how are you? The
3:23corresponding Hindi sentence is aap
3:25kaise ho? Then thank you, dhanyawad.
3:28And so on.
3:30Okay? Like this,
3:32you have a lot of rows.
3:34Fine.
3:36Now,
3:37for now, I have shown only three rows,
3:40but you can assume let's say you have uh
3:4210,000 rows.
3:45Okay? Let's say you have 10,000 rows.
3:49And you have a Transformer architecture.
3:52Right? This particular transformer
3:54architecture. So,
3:56what have you done with this particular
3:59data?
4:00This particular data, you have trained
4:02this transformer architecture
4:05this particular transformer architecture
4:07on this data.
4:09Which is having 10,000 rows.
4:12Okay? Meaning, you have completed
4:15the entire training part.
4:18Okay? And all the weights and bias
4:21values have been obtained. So, your
4:23transformer architecture is ready to do
4:26predictions.
4:29Fine?
4:30And now, let's say you have asked it a
4:33question.
4:35Let's say I asked this English sentence.
4:38This one.
4:40After the training on this entire data
4:42input and the output that is English to
4:43Hindi sentence, let's say I ask it
4:46a question.
4:48Okay? I have I have this English
4:50sentence.
4:51We are ready.
4:53So, tell me tell me
4:56what will be its Hindi translation?
4:58We are ready. What will be its
5:00What will be its Hindi translation? This
5:02is what let's say I have asked.
5:04And now, we will discuss step by step
5:07what happens when you put this sentence
5:12in this particular transformer
5:14architecture.
5:16But during prediction time.
5:18Okay? So, this is the setup that we are
5:20going to start with this video.
5:24Fine?
5:26And uh
5:27although
5:30I have made a detailed diagram of how
5:32the transformer architecture behaves
5:35during inference.
5:37As you can see here, everyone.
5:39This is the detailed diagram to show you
5:41how transformer architecture behave
5:42during
5:43inference.
5:46Okay? That is nothing but uh
5:48prediction time.
5:50But before I deep dive a deep dive into
5:52this particular diagram,
5:54I want to simplify some things for you
5:57because
5:58if I do that, then when we discuss this
6:01particular diagram, you will have an
6:04easier understanding
6:06of all these concepts.
6:08Right? This is This is the diagram here
6:09you can see which we'll be using now.
6:12So,
6:13in simple terms, the Transformer
6:15architecture is divided into two parts
6:18as we are already aware about even,
6:20right?
6:21Encoder
6:22and decoder.
6:24Now, if you talk about training time and
6:26inference time,
6:28okay? So, if you talk about this
6:30training time and inference time,
6:33>> [clears throat]
6:35>> your encoder
6:37See this? Your encoder
6:42it behaves
6:43same.
6:45Okay? Your encoder behaves exactly the
6:48same way
6:49in both the stages, whether it is
6:51training or inference.
6:56Fine? The behavior of the encoder
6:59is what
7:00uh it is during
7:03training. It's exactly same during
7:05inference as well. That's why I have
7:06written same same for encoder.
7:09So, since the encoder's behavior is same
7:12in both the stages, that is training and
7:14inference, we do not need to discuss the
7:16encoder in today's video.
7:19Okay? And anyway, we have already
7:21discussed it in the previous videos.
7:23But the main difference comes
7:26on the decoder side.
7:28The decoder behaves differently during
7:30training
7:31and inference time.
7:34During training time,
7:36I have the output sentence.
7:38Okay? During this, we have output
7:40sentence, that is in our case Hindi
7:41sentence, All the words of which I
7:43already know beforehand.
7:45I can send all of them
7:47together to the decoder at once. So, our
7:50decoder behaves in a non
7:52auto-regressive manner.
7:55During training
7:57during training, decoder behaves in non
8:00auto-regressive manner.
8:03Okay? We discussed this in the masked
8:06self-attention during last video of
8:08decoder.
8:09Fine? Whereas
8:12during prediction time, that is
8:14inference time, we do not have the
8:16output sentence. We have to predict it,
8:18right? So, here we do
8:22one by one prediction in multiple time
8:24steps. So, here your decoder behaves in
8:27auto-regressive manner. This is the main
8:30summary that uh you should already know.
8:35Fine? So, to summarize, we can say the
8:38encoder behaves exactly the same way
8:41during training and inference time.
8:47Okay? There is nothing new
8:49to learn here.
8:51But, the decoder behaves in a non
8:53auto-regressive manner during training
8:55time, but behaves in an auto-regressive
8:58manner during inference time.
9:01Which means going forward, our focus
9:03will be more on decoder's working.
9:07Okay?
9:08Now, coming to this particular diagram
9:09again,
9:12as I've told you, this is a detailed
9:13diagram and which which will be
9:14following everyone.
9:16So, let us start with our example.
9:18So,
9:20as I already shown you, what was our
9:21example? Our example was
9:24"We are ready." Now, let's say we asked
9:27it, "What will be the Hindi translation
9:29of this English sentence?" So, let us
9:31start with our
9:33simple example, "We are ready." Please
9:35understand, okay? Training is already
9:37done on this data. Let's say we assume
9:39that there were 10,000 rows, English to
9:41Hindi. Now you are asking, "What will be
9:43the Hindi translation of we are ready?"
9:47Fine. So, let us start with our example.
9:49As you can see here, I use the same
9:51example, we are ready.
9:55Okay. So, we have this query sentence in
9:58English, we are ready, and
10:01we are being told that
10:04you have trained the transform
10:05architecture on a lot of data. So, using
10:08that, generate the Hindi translation of
10:11this English sentence.
10:14First, you send this sentence to the
10:17encoder. The encoder will behave exactly
10:19the way it did during training.
10:23And input embedding, as you can see.
10:28Okay. So, I think you know
10:30how how the encoder basically works. It
10:32will behave exactly the same way it did
10:34during training. So, what we can say is
10:37you tokenize this we are ready sentence.
10:40You tokenize this
10:42into
10:43separate separate tokens. Like, we will
10:45be separate token, are will be separate
10:47token, and ready will be separate token.
10:51And then you calculate the embeddings of
10:53all three tokens.
10:55So, see everyone, something like this.
11:00Okay. Tokenization
11:02uh
11:04as even we have already discussed. So,
11:06after that, we will be talking about uh
11:10all these steps. So, something like this
11:12we'll have to do.
11:13Right? After doing uh
11:15tokenization, right? Embedding, right?
11:18So, we'll we will have to check out this
11:21particular diagram, also.
11:23So,
11:24we will be separate token, are will be
11:26separate token, and ready will be
11:27separate token.
11:28And then what you do? Then you calculate
11:30the embeddings of all three tokens. So,
11:32you see through through
11:34input embedding
11:37the embeddings of all these separate
11:39tokens will be calculated. So, obviously
11:40you know before embedding tokenization
11:43will happen. And I think
11:46And I think everyone you know
11:50how these [clears throat] things are
11:51basically done. We already discussed
11:52this in the previous videos.
11:56Then you add positional encoding to
11:59these embeddings.
12:01Then you send it to
12:03multi-head attention
12:05to extract the context.
12:07Then you send it to feed forward neural
12:10network
12:12just to introduce non-linearity. And in
12:14between all this
12:16what we do is we do
12:18add and normalize here as you can see.
12:21Okay, encoder part.
12:22And how many times you do all this work?
12:26Six times because we know everyone the
12:28the value of this n is six.
12:32Right? Because we have
12:33six encoder blocks like this. And
12:35finally the output from the sixth
12:37encoder is what you get here.
12:44Okay? Or I can say this is the output
12:46this is the output driven from the sixth
12:48encoder block.
12:50So, contextual embedding for we
12:52for are and for ready. Like these will
12:55be vectors given, okay? These will be
12:56vectors. Just for understanding purpose
12:57I have written like this we are ready.
13:00See, all these steps in encoder we have
13:02already discussed in previous video. And
13:04the link for it is given in the
13:05description. If you want
13:07you can refer it for revision.
13:11Moving forward
13:13So, for every token that you have that
13:16is we are ready
13:18you get a corresponding context vector.
13:21For we you get a context vector, for are
13:24you get a context vector, and for ready
13:26you get a context vector.
13:28So, we have reached a stage where our
13:30transformer has understood our English
13:32sentence. Now, from here, the work of
13:35the decoder begins.
13:37Which has to print the output that is
13:41which has to print out the Hindi
13:42sentence one by one.
13:45So, now that encoder's work is done, we
13:49will talk about decoder. The behavior of
13:51the decoder during
13:54training stage, during inference stage,
13:57it is different.
14:00Okay? So, at this point,
14:02we have processed the output from the
14:03encoder.
14:05We have our context vectors.
14:08And now the decoder will start its
14:10processing. You will notice
14:13here that although
14:15we have drawn only one encoder,
14:19but here you may see multiple
14:21decoders. 1 2 3 4. One encoder, but
14:24multiple decoders.
14:27Okay?
14:28This is because the processing in the
14:30decoder during inference happens step by
14:33step. First step, second step, third
14:35step, fourth step.
14:37And at every step, you get one word
14:39output like this.
14:41Like hum tayar hai.
14:43And then this is
14:45end of sentence.
14:48Okay? At every step, you get
14:50one word
14:51as output.
14:53So, the first decoder you will see give
14:55you the first word.
14:57The second will give you the second and
14:58so on.
14:59So, let us now deep down deep dive into
15:02how the decoder is processing the
15:04output. To start the decoder processing,
15:07you send a special token into the
15:10decoder.
15:12So, you see, this SOS is what I'm
15:14talking about. SOS is a special token,
15:17start of sentence.
15:22Okay?
15:22>> [clears throat]
15:23>> This special token you call SOS, start
15:26of sentence, or simply start.
15:29As soon as the decoder receives this,
15:30the decoder understands that it now has
15:33to start its work.
15:36Fine?
15:37Now, to explain the process in detail,
15:39what I will do is I will use this
15:40particular diagram.
15:44Fine?
15:45I hope so it is clearly visible.
15:47So, here
15:50in our SOS token,
15:53okay, start of sentence or start,
15:56right? This is our SOS token. Since this
15:59is currently an English word, our
16:00machine cannot understand it. So, the
16:02first thing you do is vectorization.
16:06Or vectorize this word. Convert it to
16:08numbers.
16:10And this work,
16:11as you know, will be done by embedding
16:14layer.
16:15Okay? This will be done by embedding
16:17layer. As soon as this word goes into
16:19the embedding layer, you get a 512
16:22dimension vector.
16:25Okay? This is
16:26Basically, this is 512 dimension. This
16:28will give you 512 dimensional vector.
16:32Okay?
16:33>> [clears throat]
16:34>> Right after that comes the positional
16:36encoding layer.
16:37You see? This positional encoding layer.
16:39So, we got this vector, 512 dimension
16:41vector, and then after that, we apply
16:44positional encoding, or positional
16:46encoding is applied.
16:51Now, the job of positional encoding is
16:53to encode positions in the embeddings.
16:56Right?
16:57Encode the positions in your
17:00embeddings. Although, here it is not
17:02necessary because we have only one
17:04token, but since we already we always
17:06apply positional encoding, we will do it
17:08here, too.
17:10So, we add the positional encoding
17:13vector to this embedding vector.
17:15As a result, we finally get our input
17:17vector, which we call X1.
17:19And this X1 is finally
17:21what goes into the
17:23decoder.
17:26Okay? So,
17:29it means everyone, I want I want to say
17:31see this
17:33SOS we we have passed, embedding is
17:35done, positional encoding is applied,
17:37and then now we are here.
17:39Okay? We are here.
17:41In the first decoder.
17:43Now, as per the discussion in the video
17:45of second part of transformer decoder
17:47architecture, if you remember, the
17:49decoder has mainly three blocks.
17:53Which are those anyone? The first block,
17:55as you can see,
17:57is the masked attention block, this one.
18:00Second is the cross attention,
18:03this one, and third is the feed forward
18:06neural network, this one.
18:09Okay? We have already discussed about
18:11this one as well in the last video.
18:13And if you see in between, there are
18:16add and norm.
18:18Right?
18:19Add and normalization layer. So, first
18:21we are going to send our input X1.
18:24We were here, right? So, we are going to
18:25send our input X1 to the masked self
18:29attention block.
18:31Okay? That is this one.
18:35Now, [clears throat]
18:36inside this
18:38self attention block, okay? This is our
18:40X1 input. Now, we are going to send it
18:42to the masked self attention block.
18:46Okay? Inside the attention block, there
18:48are three matrices,
18:50WQ, WK, WV.
18:55You do dot product of X1
18:59with WQ to get a query vector Q.
19:03Then dot product of X1 with WK to get
19:07key vector,
19:09this one. And similarly, dot product of
19:11X1 with WV to get value vector, that is
19:14V.
19:16Then, you do a dot product between your
19:19query and key vector.
19:22Okay? So, dot product between this query
19:26and
19:27key vector.
19:30Which gives you a scalar.
19:33You do scaling on that particular scalar
19:36and then apply soft max, which which
19:38basically gives you attention score.
19:40Let's say this is 0.9.
19:42Okay? As you can see, let's say it is
19:440.9. So, this attention score represents
19:47how similar your start of sentence token
19:50is with itself. Since we have only one
19:52token, we are extracting the similarity
19:54of the token with itself.
19:58And then,
19:59we multiply that similarity with the
20:01value vector. You see this
20:03multiplication operation? We multiply
20:04that similarity with the
20:08uh value vector.
20:11Okay? Which gives you
20:14uh
20:16let's say uh
20:18another vector, but how similar?
20:23As I've told you this attention score
20:25represents how similar your start of
20:26sentence token is with itself.
20:29Right? Because we have only one token,
20:31we are extracting the similarity of the
20:32token with itself.
20:34Whatever the similarity we have got,
20:37right?
20:39We multiply that similarity with the
20:41value vector, that is this one.
20:43See, all vectors you see are of 512
20:47dimensions.
20:50When you multiply a scalar with a
20:52vector,
20:53you get back a vector.
20:56And we are calling this as a Z1. So, we
20:58we have done this multiplication and we
21:00got this particular vector Z1, which is
21:02also
21:03a 512 dimension vector.
21:06So, at this point we have completed the
21:08masked self-attention processing.
21:12Okay?
21:13That is nothing but this one given.
21:16This is done and now we are here.
21:19Next
21:20Next is the
21:22add and norm layer.
21:24Okay? So, what you do is
21:27you take this Z1, this is Z1, you take
21:29this Z1 and add it back with X1 through
21:32a residual connection. You see this
21:34residual connection?
21:35Right? This residual connection. And And
21:38why are we adding it again because you
21:39see
21:40because you see
21:41this residual connection. Right? This
21:44one is going here. So, exactly same
21:46thing I'm showing you with the help of
21:48this particular diagram.
21:50Okay?
21:51Through this residual connection we have
21:54X1 here.
21:56So, you take this Z1
21:59and add it back with X1
22:01through through residual connection.
22:04By adding both
22:05X1
22:07and Z1 we get Z1'.
22:10All these vectors are again of 512
22:14dimensions.
22:16Okay? Finally, you send this vector to
22:18layer normalization because there was
22:20add a norm, so we send this to layer
22:22normalization.
22:25And finally, you get Z1 norm or Z1
22:28normalized.
22:31So, everyone
22:32we are at this particular point now. So,
22:36we have done this and we are at this
22:37point now.
22:38Okay? We are at this point.
22:41Next, we will go to our cross-attention,
22:44that is this one. Cross-attention block.
22:46Okay? Cross-attention is a very
22:48important aspect of this architecture
22:50because its job is to calculate
22:51attention scores between two given
22:53sequences.
22:56Here we take sequence from the decoder
22:59which has only one token. You see this
23:01one.
23:03Right? Which has only one token.
23:05That is SOS.
23:07And
23:08we also take encoder sequence.
23:13You see this this arrow? This is coming
23:15from here. We also take encoder sequence
23:18even.
23:20Which is we are ready.
23:23Whose contextual vectors we have. We
23:25calculate the attention scores between
23:27these two sequences.
23:29One coming from here and one coming from
23:30the
23:31decoder part. In a way, we want to find
23:34out how our SOS is similar to we, how
23:38much to are, and how much to ready.
23:41We want to find out the similarity of
23:42SOS with all of them.
23:47This entire calculation happens inside
23:50the cross attention block. This one.
23:54Okay?
23:55Two points are coming.
23:57One and two.
23:59One from the Like two points means I
24:00mean to say, one is coming from here,
24:02one is coming from here, and one
24:04basically coming from the
24:06left side, that is this encoder. You see
24:08this this arrow? Right? Encoder.
24:10So, one and two. These two things are
24:13basically coming.
24:16Moving forward,
24:20let me explain you with the help of this
24:22particular diagram also.
24:24So,
24:25at this point,
24:27as I've told you earlier, you know, we
24:29got this Z1 normalized.
24:31Right? We got this particular
24:33uh
24:34Z1 norm from the decoder.
24:36The cross attention block is attention
24:38block. So, inside it, there are
24:41three matrices again. So, I will write
24:43down this Z1 norm here. Okay? This Z1
24:44norm is written here, as you can see.
24:49Okay? And inside this particular
24:51attention cross attention even, as it's
24:54attention block, so inside this, there
24:56are three matrices.
24:58W
24:59Q W K and WV.
25:02The query vector is extracted from the
25:06decoder sequence. This query vector is
25:07extracted from decoder sequence, that is
25:09Z1 now.
25:12And the keys and the values vectors are
25:13extracted from the encoder sequence.
25:15This is This is basically coming from
25:17encoder view, okay?
25:18This is coming from where? This is
25:20coming from encoder.
25:25So, remember this.
25:29Keys and values vector are extracted
25:31from the encoder sequence. The query
25:33vector is extracted from decoder
25:35sequence.
25:36So, we take the vector Z1 now
25:40and dot product with WQ
25:43to get a query vector of SOS. This is
25:45the query vector of SOS given Q.
25:48Then these
25:50>> [clears throat]
25:50>> three vectors, we are ready.
25:54This also go into the cross attention
25:56block, that is this one. This is cross
25:57attention block. See, I have shown you
25:58already.
26:00These three also go inside cross
26:01attention block. Each does dot product
26:04with WK and WV.
26:07When V goes
26:10when V goes
26:13Okay? All the vector of contextual
26:15vector of V goes
26:17it takes dot product with WK
26:21and WV
26:23giving us
26:24the
26:26key and value. This K and V means key
26:29and value vectors for V.
26:33Similarly, everyone
26:36R will go and there will be a dot
26:38product with
26:40WK and WV and similarly for ready, also
26:42will go and there will be dot product
26:44with WK and WV. So, we'll get
26:47key and value vectors for R and key and
26:48value vectors for
26:50ready as well.
26:52Okay?
26:53So, once you see here we have the key
26:57value vectors for
26:58we are ready. And we have query vector
27:01for start of sentence.
27:04Now, what you do is
27:07you do dot product between the query
27:09vector
27:10and all the key vectors. Key vectors of
27:14we are ready.
27:15So, you have one query vector, but three
27:18key vectors. So, we get three scalars,
27:21w1, w2, w3.
27:24We scale this by square root of d, then
27:27apply softmax, and finally
27:29what we do is we multiply w1
27:33as you can see, we multiply w1 with the
27:36value vector of v.
27:38So, this is value vector of this what I
27:39showed, w1 with value vector of v, plus
27:42w2 with value vector of r, and w3 with
27:45value vector of
27:46ready.
27:49Okay?
27:51From this, we get cross attention output
27:54vector, and let's say it is represented
27:57as zc1. C for cross attention.
28:02So,
28:04till now,
28:06you can say we have
28:10reached here. Okay? We have reached here
28:13again.
28:15Here.
28:18Now, after this, there is also add a
28:21norm, as you can see in this diagram.
28:24So, what we will do is we will add the
28:27zc1
28:30with z1 norm. And z1 norm is coming
28:34through a residual connection.
28:36As you can see, clearly see here, right?
28:39And which was there in the original
28:40transformer architecture, also.
28:42So, then we normalize through
28:45So, so after adding these two, z1 norm
28:47and ZC1, we got ZC1 dash. Okay, which is
28:50also 512 dimension vector.
28:52And then we normalize through layer
28:54normalization. And finally, everyone, we
28:56got ZC1 norm.
28:59Okay? So, now
29:02in the in our diagram, we are actually
29:06at this particular point. So,
29:09this add a norm is done, right? So, we
29:11are here.
29:13So, we are here, then.
29:14Okay? We are here.
29:17At this point.
29:18So, at this point, the next block is
29:20what? Feed forward neural network block.
29:23This is the feed forward neural network
29:25block that has neural network inside it.
29:28So, what we do is we send the vector to
29:30the feed forward neural network. It does
29:32a non-linear transformation, and then we
29:35apply add a norm layer again.
29:38Let me show you that.
29:41Okay, so we got ZC1 norm here.
29:43And after this feed forward neural
29:45network is what I will be showing you.
29:47So, you see, everyone, ZC1 norm is here.
29:50And as I've told you that we send the
29:52vector to the feed forward neural
29:53network block. It does a non-linear
29:56transformation, and then we apply add a
29:58norm layer again. So, this is our feed
30:01forward neural network architecture, you
30:03can say. Here you can see we have two
30:04layers, one and two.
30:06So, this particular layer is having 2048
30:09neurons.
30:10This particular layer is having 512
30:12neurons.
30:13Here we are we have a rectified linear
30:16unit activation function, and here we
30:18have linear activation function.
30:20So,
30:23we do matrix multiplications, and from
30:25this we get our output vectors.
30:29Then we add and normalize. We send the
30:32vectors, uh do addition, and apply layer
30:35normalization.
30:37Fine? So, I think uh you know, everyone,
30:38how basically this works. So, this can
30:41take 512 dimension vector,
30:43right so let's see if I have Z Z 7 norm
30:45so what will happen is it will go 1 1
30:48cross 512 dimension vector I will have
30:50and these weights will be what 512 cross
30:532048 so
30:56512 cross 2048
30:59that means what 512 512 will match you
31:01will get 1 cross 2048 right and these
31:04weights will be what these weights will
31:06be like 2048 512 so 2048 cross 512
31:10right so here we have got 1 cross 2048
31:142048 cross 512 weights are there this
31:16this will match and finally everyone
31:18will will get a vector of size 1 cross
31:20512 that means one vector of size 512
31:23dimension
31:25this is how it is going to happen which
31:26we have even already discussed
31:28okay so we completed one decoder block
31:31and you can say that we have reached
31:34this particular point again so after
31:36feed forward neural network after add
31:38and norm I can say that we have reached
31:41here
31:43okay
31:47now
31:48this was the work of single decoder
31:50block but we have six such blocks
31:53right
31:54we have six such blocks we already know
31:56everyone
31:57the value of n is 6 this is same process
31:59we will repeat five more times with the
32:01same architecture but with different
32:03different parameters and that is what
32:05everyone I showed you see
32:07uh
32:08we applied this we got Y1 right we we
32:11had an add and norm operation and then
32:13we got Y1 norm and now as I have told
32:16you we need to repeat the same process
32:18five more times
32:21right
32:22with the same architecture but with
32:23different different parameters so we
32:25will finally get our final vector after
32:29going through all the six
32:31blocks
32:33after all six blocks
32:35you get an output vector
32:37like this.
32:40Okay, this output vector again is of 512
32:43dimension.
32:45You can see here.
32:47And it is passed through a linear layer.
32:50This particular linear layer here.
32:52Right? As as you can see in our diagram
32:54as well. See, this whatever we have got
32:56here, we are going to pass it through
32:59linear layer.
33:00So, this particular linear layer, if I
33:03want to talk about even this linear
33:04layer is uh
33:06going to take again 512
33:09dimension vector as an input.
33:11And it is basically this weight having a
33:16size of 512 cross V dimensions. Here V
33:19is a vocabulary size all of you. V is
33:22the vocabulary size. That means the
33:24count of all unique Hindi words in all
33:27the sentences of your data.
33:30This V is nothing but what? Count of all
33:31unique Hindi words in all the sentences
33:34of your data. And there is
33:36one neuron
33:38corresponding to each unique Hindi word,
33:41which we call it as logit. So, one logit
33:43means
33:44it represents one unique Hindi word. So,
33:46as soon as the operation is done, each
33:49neuron throws an output. This neuron
33:51will throw output. This neuron will
33:52throw output. And so on.
33:55But the problem is that this output is
33:57unnormalized. And that's why softmax is
34:01applied to make them normalized. To make
34:05those numbers normalized. And an output
34:08number should take the form of
34:11probability distribution. And softmax
34:13gives you probability for each word in
34:16your vocabulary. For example, let's say
34:19for hum, we got the probability as 0.3.
34:23Okay? Then for let's say
34:26tayyar,
34:27we got the probability as 0.03.
34:30And for let's say hai, we got the
34:33probability as 0.2.
34:35So, in this way our our time step one,
34:39which is our first step from the
34:40decoder,
34:42that gets complete.
34:44Okay? And from this step, the output we
34:46got is let's say hum because the
34:47probability is highest.
34:50Okay? So, this one here. You see, we got
34:52hum.
34:53In the time step one, we got hum.
34:57Hum is our first word
34:59from our auto regressive decoder. And
35:02finally, we pick such a word for for
35:05which we get highest probability. Let's
35:06say in our case that word is hum. So,
35:08like this step one, decoder is complete
35:11and we got hum corresponding to this
35:13SOS. So, corresponding to SOS given, we
35:15got hum here.
35:18Now,
35:19we will move to second time step.
35:21So, I have already told you that decoder
35:24will work in an auto regressive manner,
35:26which means it will step by step do
35:29output generation. In the first time
35:31step, it extracted hum word as output.
35:35Now, we will move to the second time
35:37step. So, what will be the input for the
35:39decoder at the second time step?
35:43So, look here.
35:44Now, what your input will be is whatever
35:48tokens you have so far.
35:50The SOS token you had beforehand and hum
35:53token now
35:55also come to you now, right?
35:57Hum token is also there.
36:00So, now what input you will send
36:04is these two tokens, even SOS and hum.
36:09You will send SOS as well as hum.
36:12And now, this
36:15entire thing will repeat again. The only
36:18difference is that
36:19where the last time only one token was
36:22there, this time we will go forward with
36:25these two tokens, SOS and hum.
36:29Okay? So, again, I have made a detailed
36:31diagram.
36:32I'll
36:34use that particular diagram to explain
36:37you these concepts. So, see here. This
36:40is another diagram, but now with two
36:42tokens.
36:43So, look here.
36:46We have started from uh here.
36:50SOS
36:52and hum.
36:53Instead of only one token now, we have
36:54two.
36:56So, we have sent it to the decoder.
37:01First, we got embedding.
37:03The embedding layer, so what will be the
37:05embedding layer What will embedding
37:07layer do? It will have the embedding of
37:09SOS already.
37:11It will calculate the embedding of hum.
37:13So, this embedding it was already
37:14having, it will calculate the embedding
37:16of hum, that is this one.
37:18Right? This is the embedding of
37:20uh this is the embedding of hum. And
37:22this SOS embedding, it already got
37:25earlier.
37:28So, from this, I get two embedding
37:30vectors.
37:32Okay? And
37:34this is the embedding of SOS, and this
37:37is your embedding of hum.
37:39Now, in these two, I will add positional
37:42encodings for position zero and position
37:45one.
37:46As you can see, right? This is This is
37:47basically positional encoding.
37:52And by adding both, now finally I get my
37:55two input vectors. One is X1,
37:58which represents SOS, and another one is
38:00X2, which represents the vector of 512
38:04dimension vector for
38:06hum.
38:08Okay? One is X1, and another one is X2.
38:12In the step we just did the last time,
38:15it had only one vector, if you remember
38:18it. That was X1. You see, this last
38:20diagram is also there.
38:23Okay, you see it was only X1 but this
38:26time again you see we have X1 and X2
38:29both.
38:32Right, this X1 was corresponding to SOS
38:35but since now we have two tokens X1 and
38:38X2 came here.
38:42Okay, now we will send this X1 and X2 to
38:45our first block. Our first block is
38:47masked multi-head attention. That means
38:50if I go to our previous diagram again,
38:52now I am at time step two, okay, so I'll
38:54use this particular decoder.
38:57So, what we will do is we will send this
38:59X1 and X2 so so we are here now. We are
39:01here now, you can say.
39:03And now we are going to send that X1 and
39:05X2
39:07right? to masked
39:10multi-head attention.
39:13Okay, our first block is what? Masked
39:15multi-head attention. So, this is our
39:17masked multi-head attention block and
39:19since this is an attention block inside
39:21uh this there are three matrices
39:24WQ WK and WV. So, this this this is our
39:28masked multi-head attention block, okay?
39:31Masked multi-head
39:35masked multi-head attention block.
39:38Fine. So, inside this we have WQ, WK and
39:41WV.
39:43So, we will calculate three vectors for
39:46each token. We will calculate query, key
39:49and value vector for X1.
39:52And we'll also calculate query, key and
39:54value vector for X2.
39:57So, at this point I have both of my
40:00tokens
40:01with their corresponding
40:03with with their corresponding query.
40:06See.
40:10Not only query, we have query, key and
40:12value, okay?
40:14These three vectors.
40:16So, now you know how the work happens.
40:19What will you do? You will take the
40:21first tokens query.
40:24Okay?
40:25Whatever we have.
40:27See this.
40:29Query, key, and value vectors, right?
40:32So,
40:34with the help of this X1
40:37and WQ, let's say we got Q1.
40:41After doing dot product, X2 and WQ, we
40:43got Q2.
40:45Fine? This query vectors we got.
40:48Now,
40:51you will take the first tokens query,
40:53that is Q1.
40:56This Q1 vector and multiply it with the
40:58first
41:01key vector.
41:02Okay? So, you will take Q1 and multiply
41:05it with K1. You see this arrow?
41:09Or do a dot product, I can say. And then
41:11from that, you will get
41:13a number. Let's say this is 0.2.
41:17So, Q1 into K1,
41:19let's say you got
41:210.2.
41:22After that, you will take Q1 dot product
41:26with K2.
41:28And from that, you get a second number,
41:30let's say zero.
41:31Then, you will take Q2
41:34do a dot product with K1, let's say you
41:36got 0.1, and Q2 with K2, let's say you
41:38got 0.6.
41:40That is the fourth number. So,
41:41basically, you are calculating this
41:43particular matrix anyway.
41:45Okay?
41:48So, see,
41:51actually, you are you are calculating
41:54something like this one. So, I'll show
41:58you.
41:59I'll show you here on this particular
42:01page, something like this one here. You
42:02see, we have SOS token
42:05and we have hum token.
42:07So,
42:09we have SOS token and we have hum token,
42:12right? So, you are actually calculating
42:14a matrix like this.
42:16SOS hum SOS hum. So, you will calculate
42:20SOS with SOS, SOS with hum, hum with
42:23SOS, and hum with hum.
42:25So, you actually got these values.
42:28Now, as soon as you get these numbers,
42:30you will divide them, scale them by
42:33square root of D, and after that, you
42:35will apply softmax. But, before applying
42:37softmax, everyone, what will you have to
42:40What will you have to do? You will have
42:42to do masking.
42:44What you will have to do is masking.
42:47Which is very important.
42:48So, what is masking basically during
42:51inference?
42:52So, what will you do? You will make sure
42:55that this SOS token,
42:58right? This SOS token, this does not see
43:01its future token.
43:03That means hum token.
43:06Right? As in, hum should not be visible
43:08to SOS. Whereas, hum can see both,
43:12itself and SOS.
43:14This is how it was working so far in
43:17training.
43:19So, you will do masking in this way.
43:21Okay?
43:23That is why see again.
43:25Something like this.
43:290.2 That's why we got these numbers.
43:32Right? So,
43:35this number, 0.2,
43:390, 0.1 and 0.6. This is what we'll be
43:42using here.
43:45So, as we want to do masking, so as I
43:47have told you, this is SOS hum SOS hum,
43:50right? Now, SOS should not be able to
43:52see the next token, that is hum.
43:55So, everyone, this will become zero. The
43:56upper triangular matrix will become zero
43:59here.
44:01Now, what will be the effect of masking
44:02on calculation? So, see now.
44:05What will happen is that when you
44:06calculate the vector for SOS
44:09then this 0.2 will be multiplied with
44:12V1.
44:14And this zero will be multiplied with
44:15V2.
44:16Which means in SOS calculation
44:20hum component will not come. This is SOS
44:23hum. SOS hum. So in SOS calculation
44:26everyone, hum component will not come
44:28and that is what we wanted. Whereas when
44:30you when you will calculate the vector
44:33for hum
44:36then this 0.1 will be multiplied with V1
44:40and 0.6 with V2.
44:44So in hum SOS component is coming and
44:47hum component is also coming.
44:50So this masking you have to do even
44:53during inference time. Some people get
44:56the doubt that masking does not happen
44:58during inference time. But that is not
45:00the case. In transformer architecture
45:03masking happens even during inference
45:05time.
45:06So why I'm masking during inference by
45:08the way? See, ideally you can argue like
45:11what is the need of for masking? I have
45:13both the tokens.
45:15SOS and hum.
45:17Both these tokens. SOS beforehand and
45:20hum
45:21hum basically is predicted.
45:23Right? Hum is predicted here.
45:26As you can see.
45:27Hum is predicted.
45:30Then why masking is needed?
45:35Right? So see
45:37even though we have hum predicted, we
45:40have this SOS as well.
45:42So
45:44I can show
45:47hum to SOS, right?
45:49But you don't do that. You won't do it
45:52because during your training process
45:55you have used masking. Now if you do not
45:58use masking during prediction or
46:00inference time, then in a way a shift
46:02will come in the data. And because of
46:05that shift coming, the quality of your
46:07prediction will not be good. And this is
46:10why I'm masking is used during
46:11inference. So, in simple words, across
46:13all machine learning, this is
46:14applicable. Whatever work you do on
46:16training data, you also do the same work
46:19on query data during prediction time as
46:22well.
46:24Fine?
46:25Okay.
46:26Later, let's say if a third word comes.
46:31Let's say for example, something like
46:33this one.
46:34Let's Let's assume that even third word
46:36also comes into picture.
46:40Let's say there is uh
46:43SOS.
46:45There is hum,
46:46and then third word is let's say tayar.
46:51Here also,
46:52SOS.
46:53And then hum, and then let's say tayar.
46:57Okay? So, tayar is let's say third word.
47:03Then there'll be some numbers here also.
47:07There'll be some number.
47:10SOS SOS, there'll be some number here.
47:14This will become zero, this will become
47:15zero.
47:17SOS does not see
47:19hum
47:20and tayar.
47:22Now, in the hum one, this one,
47:24there's a number here, and there's a
47:26number here.
47:27But this will become zero.
47:31Right?
47:33So,
47:36here in case of hum,
47:39there is There's a number and there's a
47:41number here, right?
47:44Hum like SOS
47:46is shown to hum.
47:48Fine? And hum can see hum itself.
47:52But don't show tayar.
47:56Right? Because these are This is future
47:57token. And in case of the other way
47:59around
48:00these three numbers
48:03are going to be there.
48:07Because the other word came at last. So,
48:09the masking concept will be valid
48:12while inferencing as well.
48:14This is very important.
48:16Okay?
48:18Now,
48:19let's talk about the output from the
48:21mask attention and add and norm.
48:25So, whatever happens from these two
48:27calculations
48:28whatever we are discussing here
48:31whatever happens from these two
48:33calculations even
48:35I've told you, right?
48:37This multiplication and then this
48:39addition.
48:41You'll get two matrices again. Let's say
48:44one is uh
48:46So, here, right? Yeah, here here. Here
48:49here, actually. So, whatever happens
48:51with this this particular calculation
48:53even, we get two matrices. That is
48:54nothing but what? One is Z1 and another
48:57one is Z2. I hope you understood this. I
48:58already told you what happens here.
49:01Okay?
49:03Now, this is your output from the mask
49:06attention block.
49:08As you can see here.
49:12Okay?
49:13Mask attention block.
49:16Okay, this one.
49:19Right? It means uh
49:21we have reached this place in our
49:22diagram.
49:23Now, again, we will do add and norm
49:26as it is mentioned.
49:28As you can see here.
49:31But, listen carefully. This is my
49:35This is basically my X1 and this is X2.
49:40Right? These two.
49:41X1 and X2. We send to the block
49:44through residual connection.
49:46We look it from here through residual
49:48connection and we added it back to Z1
49:51and Z2.
49:52So, now our output is
49:55Z1' and Z2'. Basically, this is the
49:59output after addition, I can say. You
50:00see addition.
50:02Why addition? Because because
50:04because we were here and this add
50:07operation is here, that's why.
50:09So, now our new output is Z1' Z2'
50:13and now these both went to layer
50:16normalization.
50:17Right? As I showed you in the original
50:20diagram. So, layer normalization will
50:21happen.
50:23And from these, we got the normalized
50:25output Z1 norm, Z2 norm.
50:29As you can see in this particular
50:30diagram. So, at this point, we are
50:33basically at this place. After doing so,
50:37we are at this place.
50:41Now, the next thing is cross-attention.
50:43Now, the cross-attention
50:45layer
50:47this cross-attention layer or block
50:50uh
50:51we have
50:52like in it again, we will calculate the
50:54attention scores between two different
50:56sequences. As you can see, this is one
50:58sequence coming from the decoder part
51:00and this is another sequence, you see
51:01you see. This is basically coming coming
51:03from the encoder. I showed it like this,
51:05okay? This particular arrow. I hope so
51:07you are able to see that.
51:12Fine. Now,
51:13>> [clears throat]
51:15>> this as I've told you like what will
51:17happen what will happen again here in
51:19the
51:19encoder output
51:21this this encoder output
51:24this will be sent
51:26as you can see here.
51:30This is being sent this way. And from
51:32below, our
51:34two vectors are coming.
51:36Why I say two vectors? Because you see
51:38Z1 norm and Z Z2 norm I'm talking about.
51:42Fine?
51:43So, you can see this is being sent this
51:45way.
51:48And we have to calculate cross attention
51:50between these two sets of vectors.
51:53Basically, this is our cross attention
51:55block.
51:57Okay, cross attention.
51:59This is also similar to what we
52:00discussed earlier, but the only thing is
52:01that now we have we are actually getting
52:03out these two two SOS and hum. That's
52:06why we have got two vectors here.
52:08So, let me show you how this entire
52:10thing will happen.
52:13I meant to say setting up the cross
52:15attention input. So, look here. Here Z1
52:17norm, here Z Z2 norm we have, and
52:21uh this is our encoder output given,
52:24okay? Whatever you are able to see,
52:25whatever you are able to see this one,
52:27this is Z1 norm, Z2 norm, and this is
52:29our encoder output.
52:31Okay, I hope so it is it is visible now.
52:34Okay, this is our encoder output.
52:37So, what will happen again is that with
52:39the help of your encoder vectors, we
52:41will extract keys and values, and from
52:44the decoder
52:46and from the decoder, these two vectors
52:48we will get.
52:49Okay, these two vectors are that are
52:51coming from these uh
52:53uh
52:54from these two vectors basically, we'll
52:55calculate the queries, I meant to say.
52:57So, since
52:59here two vectors are coming, we will get
53:01two query vectors. As these two vectors
53:03are coming, so we'll be getting two
53:04query vectors.
53:07So, what will happen? This Z1 normalized
53:09will take dot product with uh WQ.
53:14And from that, we will get the query
53:15vector.
53:17Let's say this one, query vector for
53:19SOS.
53:22Okay, query of SOS. And then we will
53:24take dot product
53:26of this vector with the WQ. And then we
53:29will get the query vector of hum.
53:32Then we send
53:34V.
53:37Right?
53:38Its dot product will happen with
53:40WK.
53:43And uh we will get a vector of
53:46key of W
53:48key of V basically. We'll get this one.
53:51We take dot product with WV as well and
53:53we will get the value vector of
53:55V. Similarly,
53:58when R
54:00and this ready is also sent to you and
54:03same thing will happen. So, you see
54:05we'll get get the key value vector for
54:07ready and R.
54:09So, like this we'll get the key
54:10respective key value vectors for V, R,
54:13ready.
54:15Fine?
54:17Now, to compute the attention weights.
54:20So, now what do we have to do? We have
54:24as many query vectors as we have which
54:27is basically two. We have two query
54:28vectors.
54:30And we have
54:32these
54:33key and value vectors.
54:35Right? We actually have three key
54:37vectors.
54:40Two query vectors,
54:41three key vectors. So, we have to do
54:43multiplication between them.
54:45As as it is shown with the help of these
54:47arrows as well.
54:48So, we will do hums dot product that is
54:52this one
54:53with
54:55V's key,
54:56hums dot product with V's key,
55:00and with ready's key, ready's key, and
55:03with R's key.
55:06So, we will get what? We will get W11,
55:09W12, and W13.
55:14Okay, so in this matrix, as you can see
55:16we get W11, W12, W13.
55:21Right? These three vectors, if you pay
55:22attention everyone, these three vectors
55:24is what I'm talking about. This is what
55:26we will get after doing dot product of
55:27this query vector with key vectors of V,
55:30R, ready.
55:33Okay? And then we do the the thing for
55:36query vector of SOS. And here again
55:39we'll get W21, W22, W23. So, you see
55:42W21, W22, W23.
55:47Right?
55:49You can understand it this way. Here is
55:53we right? Like we have
55:57V R ready value vectors and V R ready
56:01key vectors.
56:04So,
56:06here is V, here is R, and here is ready.
56:09Or or or this matrix also you can
56:11understand. Here is V, here is R, here
56:13is ready, and here is SOS, and here is
56:14hum.
56:16So, you can easily understand. V SOS
56:18with V, SOS with R, SOS with ready.
56:21Hum with V, hum with R, hum with ready.
56:23Uh notations you can use anything.
56:27Okay?
56:28So, these are our similarity scores.
56:30Obviously, you will scale them and apply
56:32softmax on them, which will finally give
56:34your weights.
56:36Which will finally give your weights.
56:39And now what will you do? You will
56:41calculate these two vectors. You you
56:43will multiply W11
56:47with the value of value vector of V.
56:50You'll multiply W12 with the value
56:52vector of R, and W13 with the value
56:54vector of
56:55ready.
56:56So, from this we'll get a vector which
56:58is
56:59vector ZC1.
57:04Right? You can see ZC1. And then we will
57:07uh multiply this uh bottom row also,
57:11as you can see,
57:12with all the three value vectors. And
57:14from this
57:15uh let's say we will get a second
57:16vector, and that is named as ZC2.
57:20So, at this point
57:22in our original diagram video, earlier
57:24diagram,
57:26you [clears throat] can say that
57:29we have we are here.
57:31Okay? we are here.
57:34Now, we will do add a norm again as
57:37shown here. If you notice from here, I
57:39have sent a residual connection. You
57:40see, this residual connection is here.
57:45Which came here and we added
57:47like this. See?
57:49I'm talking about this one here. See?
57:51This is a residual connection, Z1 norm,
57:52Z2 norm. It is basically coming here.
57:54So, Z1 norm, Z2 norm is coming here and
57:57with these two things will be added to
57:59ZC1 and ZC2. And we will get a
58:03normalized
58:05So, addition addition basically ZC1' and
58:08ZC2'.
58:11We basically got ZC1', ZC2', ZC2'
58:15which we will send again to layer
58:17normalization.
58:19And we will get normalized ZC1,
58:21ZC1 norm, and normalized ZC2, that is a
58:24ZC2 norm.
58:26And now we are at this particular place.
58:30We are at this particular place.
58:34Okay? And now the next block is what?
58:36Feed forward neural network block. So,
58:39let's talk about that or let's try to
58:41understand feed forward neural network.
58:43Let's quickly see what happens in the
58:45feed forward neural network block. For
58:47that even
58:49I show you
58:51with the help of this particular
58:52diagram. See this?
58:54So, as I've told you
58:55we have ZC1 norm, ZC2 norm.
58:58Okay? After this this is how our feed
59:01forward neural network block looks like.
59:03Okay? There are two layers again.
59:072048 neurons, 512 neurons. This will
59:10take input of size 512. Right? Here you
59:13have ReLU,
59:14rectified linear unit. Here you have
59:16linear activation function. You already
59:18know this.
59:20So, it's exactly the same way matrix
59:21multiplication will happen from this and
59:24we will get two vectors,
59:26Y1, Y2.
59:29So,
59:31>> [clears throat]
59:31>> we have reached here.
59:34We have reached here.
59:38Now, again, you will do add a norm, as
59:40you can see, add a norm is there.
59:43Okay? Again, you will do add a norm. You
59:45can see, we send this
59:49uh
59:50this uh
59:52as we are going to do add a norm, so you
59:55can see, we send these ZC1 norm, ZC2
59:59norm
1:00:00here. Okay? Through the residual
1:00:02connection. We have sent it here. So, Y1
1:00:04we have got, Y2 we have got.
1:00:06And this ZC1 norm, ZC2 norm, also we
1:00:09have got here with through residual
1:00:10connection.
1:00:13And then again, perform the addition.
1:00:15This addition again, as you can see.
1:00:18And that addition uh we send inside
1:00:22layer norm or layer normalization. And
1:00:24from layer norm, we got normalized
1:00:27vectors.
1:00:28Y1 normalized, Y2 normalized.
1:00:31Two vectors have come to
1:00:34this place.
1:00:35And at this point, we are at this place
1:00:39in our diagram. After doing this, so we
1:00:41are at this place again.
1:00:44Okay?
1:00:45So, this was the work of a single
1:00:47decoder block. But we have six such
1:00:49blocks, which means the same process we
1:00:53will repeat five more times. And that is
1:00:55why five more decoder blocks it will go
1:00:57through.
1:00:59With the same architecture, but again
1:01:00with different different parameters. So,
1:01:02finally, we will get two vectors.
1:01:07YF1 normalized
1:01:09and YF2 normalized.
1:01:11F, let's say, for final.
1:01:13And now, on this entire journey, we have
1:01:17reached this place.
1:01:21So, after doing this,
1:01:23uh after doing this add and norm every
1:01:25time.
1:01:25Okay?
1:01:27As I've told you,
1:01:29sorry, this this normalization was
1:01:30actually done. So, we are actually here
1:01:32now. We are now because after addition
1:01:34this normalization should be done and
1:01:36now we are here.
1:01:38Okay? Now, we just have to print the
1:01:40output.
1:01:43So,
1:01:44what we will do is
1:01:46we use a linear
1:01:48plus
1:01:50softmax thing here. Okay? Softmax
1:01:52function.
1:01:54So, we use a combination of both to
1:01:56print the output. Now, there is an
1:01:58interesting thing here. Always at every
1:02:00time step of the decoder you need only
1:02:03one single token's output, but you are
1:02:05constantly carrying two vectors. So,
1:02:07like here we have YF1 norm, here we have
1:02:11YF2 norm. We are constantly carrying two
1:02:13vectors, but finally we need only one.
1:02:17So, now if you send both these vectors
1:02:19here,
1:02:22then two tokens will be printed, but we
1:02:25need only one token. The auto regressive
1:02:28property tells us this. Always at every
1:02:30time step one token output should
1:02:33happen.
1:02:33So, at this time step, what do you do?
1:02:36This vector that you have
1:02:39This vector is corresponding to SOS and
1:02:41this vector is corresponding to hum. So,
1:02:43now you won't use this thing.
1:02:48Again, you will not use this.
1:02:50You will only send this particular
1:02:53vector.
1:02:55Which corresponds to hum.
1:02:57You will only send this particular
1:02:58vector to this network.
1:03:01You won't send this one.
1:03:04Auto regressive property.
1:03:06And then this network that you have, it
1:03:09will work exactly the same way. A number
1:03:11will come out from every logit.
1:03:14This is a logit. A number will come out
1:03:15out of every logic, every neuron area.
1:03:18Right? We already discussed something
1:03:20like this before also.
1:03:24Which will be unnormalized. These
1:03:27numbers which will be coming everyone,
1:03:28they will be
1:03:30unnormalized.
1:03:32And now
1:03:34you will take those that logic whose
1:03:37probability is the
1:03:39And before that, before that, what you
1:03:41will do is you will send it to the
1:03:42softmax.
1:03:44As they are not normalized, so whatever
1:03:47the numbers which will be coming
1:03:48everyone, they will be sent to the
1:03:50softmax, which will normalize it and
1:03:52convert it to a probability
1:03:53distribution. And now you will take that
1:03:55logic whose probability is the highest
1:03:57from the probability distribution. And
1:03:59let's assume that
1:04:01the word that comes out with the highest
1:04:03probability is tayyar.
1:04:06So tayyar will be the output of time
1:04:09step two.
1:04:12So here, please don't get confused.
1:04:15This step is very crucial.
1:04:17Okay? This particular step is very
1:04:19crucial.
1:04:20All the way to the end you are carrying
1:04:22all the vectors with you.
1:04:24But at the very end moment, you send
1:04:26only the last one forward.
1:04:29You forget this one.
1:04:32And in this way, you got your second
1:04:34word that is nothing but tayyar.
1:04:37So as you can see here everyone, this is
1:04:39our second word, tayyar.
1:04:44Now think about what will happen in the
1:04:45next time step. In the next time step,
1:04:48you will send
1:04:50three tokens in the input.
1:04:52Like this one everyone.
1:04:55SOS hum
1:04:57and tayyar.
1:05:00Okay? Which means this whole process
1:05:03will repeat. But instead of two tokens,
1:05:06there will be three tokens. All three
1:05:08vectors everywhere. As tayyar is added
1:05:11here,
1:05:13as they are is added here, so I can show
1:05:15you
1:05:18>> [snorts]
1:05:18>> what will happen.
1:05:21See, as they are is added here, so there
1:05:23will be they are
1:05:25word.
1:05:28One more vector will be added here,
1:05:30everyone.
1:05:32One more vector will also be added here.
1:05:37Right?
1:05:38One more set of query key and value
1:05:41query
1:05:43key and value, whatever we have, so we
1:05:44will get one more set of query key and
1:05:46value here.
1:05:49One more vector here, one more vector
1:05:51here, one more vector here as well.
1:05:55And this entire calculation will adjust
1:05:58in that way. And everywhere we will get
1:06:00three vectors. Everywhere you'll get
1:06:01three vectors. Here also you'll get
1:06:03three vectors. Okay?
1:06:05One more query key value pair will be
1:06:07added. Here also one more query uh
1:06:10vector here also one more vector you
1:06:12will get. Here also one more vector you
1:06:14will get. And finally here as well.
1:06:19So, like this if you go,
1:06:23same thing will happen. One more vector
1:06:25here,
1:06:26one more vector here, one more vector
1:06:28here.
1:06:29And finally we will have here also. And
1:06:33this Y F3 norm also will be there.
1:06:38Exactly same, following same process.
1:06:41So, we will have Y F3 norm as well.
1:06:45And here we will have uh
1:06:48as as we have this third vector, which
1:06:50will be corresponding to they are,
1:06:51right? This this basically corresponds
1:06:53to they are.
1:06:55But now, what will you do? You won't uh
1:06:58you won't basically send this one.
1:07:02You won't send this one as well.
1:07:06Right? You'll only send
1:07:08this one.
1:07:11Right? Because you want only one output.
1:07:13Now, on the on this entire journey
1:07:17on this entire journey, everyone, here I
1:07:19will come.
1:07:23Here one more vector will come. Right?
1:07:25Like the output will be here. Now, once
1:07:27this is there, everyone,
1:07:31as I want to show you, see, this this
1:07:33hey we have got as a corresponding token
1:07:35of the year.
1:07:37And now, in the next step, what will
1:07:39happen is
1:07:41there'll be SOS as an input. SOS the
1:07:43year hey will be input.
1:07:46So, hey is also coming here. That means
1:07:49That means on this entire journey, I
1:07:51will get hey as well here. So, here you
1:07:53you will get four vectors, one more
1:07:55query key and value pair, one more
1:07:57vector here here here, and so on. Like
1:08:00this, everyone, finally you will get one
1:08:03more vector here.
1:08:07Okay? And like this, you will reach up
1:08:09again.
1:08:10And here, let's say I have YF4
1:08:14norm.
1:08:15YF4 normalize. And then again,
1:08:18you will ignore this one, you will
1:08:20ignore this one, you will ignore this
1:08:21one.
1:08:24You will only send this one.
1:08:28And hopefully from this, you will get
1:08:30end of sentence. That is nothing but
1:08:32EOS, end of sentence token.
1:08:36And as soon as you get end of sentence,
1:08:39you will understand that you have to
1:08:41stop your inference process.
1:08:43You stop it. And your output will be
1:08:45what? Your output will be hum the year
1:08:50hey. Okay? It will be like hum the year
1:08:52hey, like this area. Your output will be
1:08:54what? Hum the year hey. And finally, we
1:08:57got end of sentence.
1:08:59And this is is transformer inference
1:09:02works.
1:09:04So, I have tried to explain this entire
1:09:06process through a detailed diagram. The
1:09:08main difference is the encoder works
1:09:10absolutely in the same way, but the main
1:09:13difference comes in the decoder. The
1:09:14decoder behaves in a non-autoregressive
1:09:17manner during training time. It takes
1:09:19all tokens together, generates all
1:09:21outputs together. Whereas during
1:09:23inference time, it behaves in an
1:09:25autoregressive manner. It takes
1:09:28tokens one by one and gives only one
1:09:30token output at every time step.
1:09:33So, that's it
1:09:34about the decoder architecture during
1:09:36inference time.
1:09:38Thank you, everyone.

About this transcript

This page contains the full transcript of Transformer Arch Decoder Inference [with Paper & Pen] -How Transformers ACTUALLY Generate Text Part3 by yogesh murumkar, generated from the public captions YouTube serves with the video. The transcript has 9,388 words across 1,766 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.