YT YouTube2Text

YouTube2Text → Transcripts → Transformer Architecture Encoder Explained from Scratch (Pen & Paper Deep Dive) - Part1

Transformer Architecture Encoder Explained from Scratch (Pen & Paper Deep Dive) - Part1 — Transcript

by yogesh murumkar · 6,056 words · 1,212 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00Hello everyone.
0:01This is Yogesh Moramkar.
0:03In this video, we are going to discuss
0:05about transformer architecture in detail
0:08with very simple explanation using the
0:11concepts like self-attention,
0:14multi-head attention,
0:16positional encoding,
0:18and layer normalization.
0:21So, self-attention,
0:23multi-head attention,
0:27Okay?
0:29positional encoding,
0:34and
0:35layer normalization.
0:40Layer normalization.
0:44And by chance, if someone is not aware
0:46about any of
0:49these concepts, please don't worry.
0:51I'll try to give you basic understanding
0:53of these concepts whenever required. And
0:56if someone wants to understand all these
0:58concepts in detail, I'll create separate
1:01videos for it.
1:02Okay?
1:03So,
1:04also before starting the explanation,
1:06let me tell you that the link for notes
1:09and all the diagrams
1:12have been provided in the description of
1:14this video.
1:16Whenever I say notes and all the
1:17diagrams, means see everyone, I'll be
1:18using all these things.
1:21I'll be explaining you
1:22the concept of transformer architecture
1:26with a very simple explanation. Everyone
1:28will be able to understand.
1:30Okay? So, all these things are already
1:32shared.
1:34You can go, you can click on the link
1:36provided in the description, and you can
1:38download all this stuff.
1:40So, that if you want to go, if you want
1:41to revise these things,
1:43you can easily revise and try to
1:45understand the concept properly.
1:48Okay?
1:50So,
1:51I have broken down this particular
1:53transformer architecture into two parts.
1:57In this video, we will focus only on
2:00encoder part of the architecture.
2:04As it is mentioned here.
2:07In the next video, we will focus on the
2:10decoder architecture.
2:12Okay, so what is encoder? What is
2:13decoder? We will talk about it.
2:16The main goal of this video is to
2:17understand the encoder architecture
2:20well.
2:22And I'll be using this these handwritten
2:24diagrams for the entire explanation.
2:27Whatever I have shown you just now.
2:30So, let me explain the journey of
2:32understanding transformer encoder
2:34architecture from the very familiar
2:36diagram which you are able to see here.
2:40Okay, I hope so everyone is able to see
2:42this properly.
2:43Starting with the inputs, input
2:45embedding
2:46on this side. And here you can see
2:48output shifted right, output embedding
2:50and so on.
2:52Okay, this is the very familiar diagram.
2:54By chance, if someone has come across
2:56something like
2:57transformer architecture, definitely
2:59they must have come across this
3:00particular diagram.
3:03So, see everyone, this diagram
3:05represents the entire transformer
3:07architecture.
3:08Both encoder and decoder are inside
3:12this.
3:15But the problem is that this diagram is
3:17very complex.
3:18If someone sees this diagram for the
3:20first time, they will feel a little
3:22confused.
3:24And some things will not be understood
3:26by them.
3:27But if I try to simplify this diagram a
3:30little,
3:31then things become easy.
3:34Let me ask you, do you at least see or
3:36do you at least understand that there
3:38are two parts in this diagram?
3:42One part is here. You can see everyone,
3:45this one.
3:46This one.
3:48First part of the diagram.
3:51Okay? On the left side. and another part
3:54is here.
3:55This one.
3:57See, I'm highlighting it. Okay?
4:01Then we can say clearly this part is our
4:04encoder. On the left side you see this
4:06one everyone, this is our encoder part.
4:09And this part
4:11is our decode.
4:13Okay? So, I can show you
4:15this particular diagram
4:19in a very simplified way.
4:21Like this.
4:23You see everyone.
4:24So, whenever I say transformer
4:26architecture, transformer architecture
4:27basically will have two blocks.
4:30Encoder and decoder. So, in our diagram
4:33which I showed it to you here, you can
4:35see this is the encoder part and this is
4:37the
4:38decoder part.
4:40Okay?
4:41One more thing everyone you can notice.
4:44Next to encoder, here, you see this NX
4:48is written.
4:50Right?
4:51Next to the encoder block, as well as
4:53decoder block as well. You see NX here
4:56and NX here.
4:58Right? Or N written here.
5:00And here as well.
5:01This N means what? This N means that
5:03there is not just one encoder block, but
5:06there are multiple such blocks. Not only
5:09one encoder block, but there are
5:10multiple such blocks. Similarly, there
5:13is not just one decoder block, there are
5:15multiple such a decoder blocks.
5:20If you read the original transformer
5:23paper,
5:24you might be knowing
5:26the name of the original transformer
5:27paper is what? Attention is all you
5:31need.
5:34Okay? Maybe
5:36most of you might have come across
5:37definitely.
5:39This is a paper written by eight people
5:41from Google Brains.
5:42Okay?
5:43So, if you read the original transformer
5:45paper titled attention is All You Need",
5:47you will find that
5:50they basically used six blocks for the
5:53encoder. How many? Six blocks.
5:57And six blocks for the decoder.
6:01So, if I want to show the next thing,
6:04the architecture basically looks like
6:06this one.
6:08Six encoder blocks and six
6:11decoder blocks.
6:13See, six is not a magical number. They
6:17achieved it through experimentation.
6:19And the number six basically gave them
6:23best results.
6:24That's the only reason. Otherwise, there
6:26is no specific reason why six encoder
6:28blocks and why six decoder blocks.
6:31Okay?
6:32Obviously, the question might have arise
6:34that
6:36why six? So, I've told you, right? It
6:39It's
6:40It's all through the experimentation.
6:42Okay? Now, the next question
6:45might be like, are all these blocks
6:47identical?
6:49Whenever I say six encoder blocks, are
6:51all these blocks identical? The answer
6:54is yes. All these blocks are completely
6:57identical. Meaning, their internal
6:59architecture is exactly same.
7:01I can say copy-paste.
7:03This means if you understand one block,
7:06you automatically understand other five
7:08blocks as well.
7:10Okay?
7:11So, obviously, our focus is now to
7:13understand single encoder block.
7:16What goes
7:17inside one single encoder block, that is
7:20what we will try to understand.
7:22Okay?
7:23So, now moving forward, everyone.
7:27As I've told you, this is the
7:28Transformer architecture. Out of this,
7:30we will try to focus on the entire
7:32encoder part.
7:34In the next video, we will try to
7:35discuss about the decoder part.
7:37So,
7:38the architecture
7:40in a simplified way looks like this.
7:42Inside one encoder,
7:44there are six encoders. So, it looks
7:46like this. And now, we will go inside
7:49one encoder, everyone.
7:51Okay?
7:53We will go inside one encoder, and that
7:55one encoder basically looks like this.
8:01Okay?
8:02>> [clears throat]
8:03>> So, when we zoom into a single encoder
8:06block, we find it contains two smaller
8:10blocks.
8:11Okay? This basically contains two
8:13smaller blocks.
8:15First is
8:17a self-attention block.
8:19Second is a feed-forward neural network.
8:22As you can see clearly here in this
8:24particular diagram.
8:27Next, let me explain how
8:30data flows through complete encoder.
8:35Okay?
8:36Let's try to discuss
8:38about that with a very simple example.
8:43Your sentence enters the first encoder
8:46block.
8:49See, as I've told you, there are six
8:50encoders, six decoders. So, your
8:52sentence enters the first encoder
8:55encoder block. The first first encoder
8:57block
8:59looks something like this.
9:05See, as as I've showed you,
9:08okay? Something like this.
9:11And if I want to show you
9:16what exactly happens when I say
9:18feed-forward neural network and
9:20self-attention, so it looks like this.
9:23The first encoder block, everyone, looks
9:25like this.
9:27From where I got it? So, you see here,
9:29in this case,
9:31multi-head attention, add and norm,
9:33feed-forward, add and norm. Exactly same
9:36thing is shown here. Multi-retention add
9:39a norm, feed forward add a norm.
9:42That is what I'm trying to show you.
9:44And like this, how many are there? Six.
9:46Six encoder blocks are there.
9:49So, the diagram basically becomes
9:52something like this one here. See.
9:54As I've told you,
9:56your sentence enters the first encoder
9:58block. This is the first encoder block.
10:01And the first encoder blocks look like
10:03this.
10:04It looks like this.
10:06The output The output from the first
10:09encoder block goes to the second encoder
10:11block. Even though I did not draw it
10:13here,
10:14but you can imagine. It is exactly same.
10:16I've told you, right?
10:18So, the output from the first encoder
10:19block
10:22goes to second encoder block.
10:24The second encoder block is identically
10:26the same.
10:28Then, processing happens inside second
10:30encoder block as well. The output from
10:32this second encoder block goes to the
10:34third one. The output uh from the third
10:37goes to the fourth.
10:39And so on.
10:41Till the sixth encoder block.
10:43Finally, our sixth encoder block output
10:47goes towards decoder.
10:52Okay, as you can see here.
10:55So, what exactly is 3 cross 512? How it
10:58How it is achieved? We'll talk about it.
11:01Fine? But, I hope so you got an
11:03overview.
11:04Whenever I say some input is passed to
11:06the encoder, what exactly happens? So,
11:08this is like higher high-level overview,
11:11I can say.
11:13Fine?
11:14Finally, as I've told you, our sixth
11:16encoder block output goes to decoder.
11:20And
11:22one more thing, everyone. Before the
11:24input enters the first encoder, this
11:27input, before it enters
11:30first encoder or first encoder block.
11:35There are three important operations
11:37which are performed on the sentence.
11:41Let me take an example everyone and
11:44explain each and every step to you.
11:47Okay. So, see here.
11:50Let's say
11:53Before see, before input enters the
11:57encoder block,
11:58what exactly happens? Let's try to see
12:00this. Okay, and I've told you already
12:02that there are six encoder blocks.
12:04So, for example everyone, let's say I
12:06have a sentence. I have an example
12:10like this.
12:11I am Yogesh.
12:14Okay.
12:16The first operation
12:18which comes into picture
12:20is tokenization.
12:23You can see here, tokenizer.
12:26This means the tokenizer will break our
12:28sentence into words.
12:30Okay? Tokenization means you break down
12:32a given sentence into tokens. Tokens
12:35could be a single word, two words, three
12:37words. It depends on the tokenization
12:39approach. For now,
12:41we are doing word level tokenization.
12:43This means the tokenizer will break our
12:44sentence into words.
12:46Let's say
12:47first token is I. Another token is am.
12:51And next token is Yogesh.
12:54The second operation is very important.
12:57This one.
13:01This is also called as text
13:03vectorization.
13:05This means your encoder or transformer
13:08cannot process words. Obviously, no
13:10machine can process words, right? You
13:13must somehow convert them into numbers.
13:17The technique we use for text
13:18vectorization is called embedding.
13:23The technique we use for
13:25vectorization, for text vectorization is
13:28called embedding.
13:30What we do
13:32is that we take every word and put it in
13:36this embedding layer.
13:39And this embedding layer will generate a
13:40vector of every word.
13:45In our case, it will be 512 dimension
13:49vector. Okay, that's why I have written
13:50here 512 dimension vector.
13:54It's It's like this.
13:561 2 3 4 and so on. 512 dimension vector.
14:03Fine. This means for the word I, for the
14:07word I or the token I, we create a 5 112
14:11dimensional vector representation.
14:15Similarly, for words and and Yogesh, we
14:18get 512 dimensional vector
14:20representation.
14:22But the problem is
14:24after this, but the problem is
14:27we don't have any information about the
14:30order of words.
14:33Which word comes first, which word comes
14:36later.
14:37We don't know that.
14:39And guess what? This is very very
14:40important, obviously.
14:42You must always know which words in your
14:44sentence come first and which
14:47come later.
14:49So, to add this positional information,
14:55we use a technique called as positional
14:58encoding. Basically, this this icon you
15:01can see everyone, this is for positional
15:03encoding.
15:05Okay, if someone has come across
15:08the transformer architecture shown in
15:10the
15:11uh attention is all you need paper,
15:14you must have seen that. They also have
15:15used the same notation.
15:18So, everyone, to add positional
15:19information, we use a technique called
15:21as positional encoding. For each
15:23position,
15:24like position number one, it will
15:26generate a 512 dimensional vector.
15:30Similarly, for position number two, it
15:33will also generate 512 dimensional
15:34vector, and similarly for position
15:36number three.
15:38Now,
15:40those 512 dimensional vectors are P1,
15:42P2, P3. Okay? And E1, E2, E3 are nothing
15:45but what? These are the 512 dimensional
15:48embedding vectors for I and M and
15:51Yogesh, respectively.
15:54Now, we have to add
15:57our
15:58embedding vectors with their positional
16:00vectors. So, basically, we add these two
16:03vectors, E1 and P1. We add these two
16:06vectors, E2 and P2. And then E3 and P3.
16:10You can see this addition operation.
16:13So, after all these addition operations
16:15are even, we get three new vectors.
16:19As you can see, I mentioned here X1, X2,
16:22X3.
16:23X1 is what? It's embedding of
16:26I. It's embedding of I. Right? This is
16:29for I, this is for M, and this is for
16:33Yogesh. These are the embeddings.
16:35So, X1 is what?
16:38X1 equal to embedding of I plus
16:41positional encoding of position
16:44one.
16:46X2 is what? Embedding of M plus
16:48positional encoding of position two.
16:51And
16:51>> [clears throat]
16:52>> X3 is what?
16:53Embedding of Yogesh plus positional
16:56encoding of position three.
16:59This is how we got X1, X2, X3.
17:02Now, these three vectors, X1, X2, X3,
17:07enter the first encoder block.
17:10It means what, everyone?
17:12It means
17:14the previous diagram which I showed you,
17:17this [clears throat] one.
17:19This X1, X2, X3 will enter this block
17:22because now everything related to input
17:24operation is done. Now that X1, X2, X3
17:27will enter first encoder block.
17:31That is this one. I hope so you are able
17:33to see that.
17:34It means what? It means everyone in our
17:38architecture, where are we right now?
17:41If you want to understand that, I'll
17:43just point it out. So see everyone,
17:46we are here.
17:48Okay? We are here.
17:51Fine?
17:52>> [clears throat]
17:54>> So
17:55they encounter
17:57this X1, X2, X3, as I've told you, they
17:59enter the first encoder block.
18:02And
18:03first they encounter the multi-head
18:06attention block. You see?
18:08Multi-head attention.
18:10The first thing which they will
18:11encounter, X1, X2, X3 will encounter is
18:13multi-head attention.
18:16So whatever we got here,
18:18>> [clears throat]
18:19>> this X1, X2, X3, right? They will
18:22encounter multi-head attention.
18:25Or here also, in this diagram also I can
18:28show you. You see? Input will encounter
18:31multi-head attention. Please try to
18:32understand. I'm just trying to
18:35break this in small small parts so that
18:37it will be easy for you to understand.
18:40Fine?
18:44>> [clears throat]
18:44>> Now,
18:46self-attention,
18:49that is this multi-head attention
18:51everyone. Multi-head attention is like
18:53multiple self-attentions, okay? So
18:55that's why I'm saying self-attention is
18:57applied because your embedding vector
18:59has a problem.
19:02This multi-head attention is multiple
19:04self-attentions.
19:05Now why?
19:06Why this
19:08self-attention is applied everyone?
19:12Why it is applied?
19:14So, that's what I'm trying to tell you.
19:16Self-attention is applied because your
19:18embedding vector has a problem.
19:20This X1, X2, X3, it has a problem. The
19:23problem is that the same word has same
19:26embedding everywhere.
19:28Please try to understand.
19:30The same word has the same embedding
19:33everywhere. For example,
19:35for example, let's say
19:40a very common example given will which
19:42you will find out everywhere. Let's say
19:43I have a sentence like this.
19:45I
19:47went to
19:49bank. Okay, this is the first sentence.
19:54Another sentence is
19:56I sat
19:59on the
20:02river bank.
20:05This is another sentence.
20:07Now, here everyone,
20:10in this case,
20:11both these sentences, sentence one and
20:14sentence two,
20:16they use the word bank.
20:19But, with completely different meanings.
20:23Right? Yet, the embedding will be
20:26exactly same for both. Embedding is
20:28nothing but what? Embedding is nothing
20:29but representing these words in terms of
20:31vector numbers.
20:33So, embedding will be same for them.
20:35Because bank and bank word is same. So,
20:37self-attention solve this problem.
20:39Because, if you see here, in this case,
20:42I went to the bank, bank is basically
20:45used in the context of financial
20:47institution.
20:49And here in this case, I sat on the
20:51river bank,
20:52bank word is used in some different
20:54context, like river bank they are
20:55talking about.
20:58Right? So, obviously everyone, their
21:00embedding should not be same.
21:02So, self-attention solve this problem.
21:04As soon as you apply self-attention, you
21:06notice that the embedding of bank in the
21:08first sentence and the embedding of bank
21:12in the second sentence changes. It
21:15starts considering surrounding words
21:18contextually.
21:20It becomes contextually aware.
21:25Okay? And what is even better
21:28than self-attention?
21:30It is multi-head attention.
21:33Because in multi-head attention,
21:34everyone, in multi-head attention, you
21:37bring multiple self-attention modules
21:40and get opinions from all of them. It's
21:43like one opinion is good, two opinions
21:46are better.
21:48And more opinions means you have more
21:50diversity.
21:53So, without going into the details of
21:55what is happening inside this block, I
21:57will directly give you the output. All
21:59right, I'll directly show you the
22:01output.
22:03Right? This block I'm talking about.
22:06So,
22:07when you send the first vector through
22:10multi-head attention, that is this X1
22:13through multi-head attention, you get
22:15another vector Z1,
22:18which is again a 512 dimensions.
22:21Okay? This is again 512 dimensions.
22:28But this Z1 will consider surrounding
22:31words contextually, that is Z2, Z3.
22:34Similarly, when you send X2,
22:38this is basically vector for M,
22:41embedding vector for M,
22:43you get
22:45uh
22:46when you send X2, you get Z2.
22:49Right? And when you send X3, you get Z3.
22:52So, all of them, they are basically 512
22:55dimension vectors.
22:58Now, here is something very good that is
23:00happening.
23:01We use residual connections. Everyone,
23:04this is called as residual connection or
23:06skip connection.
23:10Okay, residual connection or
23:13skip connection.
23:15In the original diagram as well, you can
23:17see here. See, this is what I'm talking
23:18about. This this line this line is
23:21called as residual connection or
23:23skip connection.
23:24Okay, this one.
23:29>> [clears throat]
23:30>> Notice here where it says add and norm.
23:35You see, add.
23:37That is actually for this residual
23:39connection. Everyone, this add is
23:40actually for residual connection. What
23:42happens is when you send your X vectors,
23:45these X vectors
23:47X1, X2, X3.
23:50When you send these
23:51uh
23:52X vectors
23:56that is X1, X2, X3, one path goes into
23:59multi-head attention.
24:02You see?
24:03Okay, you see here?
24:05This goes into multi-head This goes into
24:07multi-head attention.
24:08X1, X2, X3 will come from here and it
24:10will go into multi-head attention.
24:13Right? That is nothing but
24:15this one.
24:16Okay, as we have three words, that's why
24:18it is like this.
24:20So, one path goes inside multi-head
24:22attention, but another path does what?
24:25It bypasses the multi-head attention and
24:27goes ahead. See, you see here? It
24:30bypasses this multi-head attention and
24:33and you can see
24:35it goes here.
24:38Right? It goes ahead or here you can
24:40see. Again, same thing as I've told you.
24:42This one. It bypasses this multi-head
24:44attention and it goes ahead.
24:49So, we have two sets of vectors at this
24:51point.
24:53You can see?
24:56Because of this residual connection
24:58it has bypassed the multi-headed
25:00attention and now we have two sets of
25:02vectors.
25:05Okay?
25:06One from multi-headed attention, that is
25:08Z1, Z2, Z3. How we got it? We already
25:10seen that.
25:12And one original input, that is X1, X2,
25:15X3, which we have got
25:17through
25:18this residual connection or skip
25:20connection.
25:22The good thing is all these are 512
25:25dimensions. These are already 512 and
25:27these also are These also are 512
25:29dimensions.
25:30Which means you can perform any
25:32mathematical operation.
25:34And what operation
25:36we perform here? We perform addition.
25:38You see?
25:40>> [snorts]
25:41>> That is nothing but this add.
25:44Okay? So, we add
25:47Z1
25:48+ X1, Z2 + X2, Z3 + X3.
25:53This gives us a new set of vectors
25:56and that is nothing but
25:57Z1', Z2', and Z3'. Now, let me show you
26:02that particular part.
26:05You see everyone.
26:07After this, you understand after this
26:09X1, X2, X3
26:11All right? After addition of X1, X2, X3
26:13and Z1, Z2, Z3
26:15All right? After addition of this, we
26:16got Z1', Z2', and Z3'. Now, I'm going to
26:20continue with this.
26:22Okay? I'm going to continue with this
26:24one. So, we got Z1', Z2', Z3'.
26:29Which are again 512 dimensions. All of
26:32them again. They are 512 dimensions. 512
26:36dimensional vector, 512 dimensional
26:37vector.
26:39Okay?
26:40And finally, everyone, we have to
26:42perform
26:44normalization.
26:45Something called as layer normalization
26:48is applied.
26:49You can You this.
26:51You can see this here in the original
26:53architecture.
26:54After doing addition,
26:57we are doing what? Normalization. So,
26:59this norm means layer normalization
27:01even.
27:02Okay? That is what is basically shown
27:04here.
27:08In layer normalization, you take each
27:10vector
27:12like Z1 dash, Z2 dash, and Z3 dash.
27:17Okay? So, Z1 dash, which has 512
27:21numbers,
27:24we calculate the mean of all these 512
27:26numbers, calculate the standard
27:28deviation, and normalize or standardize
27:31all 512 numbers within some given range.
27:34Okay? So, we calculate the mean, that is
27:36mu. We calculate the standard deviation,
27:38that is sigma.
27:41And normalize or standardize all 512
27:43numbers within given range. I hope if
27:45someone has
27:47basic idea about at least machine
27:48learning, they know how to normalize the
27:51data, how to standardize the data.
27:53Okay? So, that is basically done here
27:56in layer normalization.
28:02It means what, everyone? It means
28:03whenever I say normalize or standardize
28:05all 512 numbers, we try to bring them
28:07within some given range.
28:10You do the same work for
28:12Z2 dash and Z3 dash.
28:15After this operation, Z1 dash becomes Z1
28:19normalized, Z2 dash becomes Z2
28:21normalized, and Z3 dash becomes Z3
28:23normalized.
28:25You can see, Z1 norm, Z2 norm, Z3 norm.
28:30By the way, why are we normalizing? The
28:32answer is simple.
28:34Just to stabilize the training process.
28:38I've told you, right? You must have come
28:39across normalization concept and
28:42standardization concept in machine
28:44learning as well.
28:47Okay?
28:48Just to stabilize the training process,
28:51we are normalizing.
28:53When you compute
28:55Z1, Z2, Z3 through multi-head attention,
28:59right? As we have seen in the previous
29:01step, this Z1, Z2, Z3.
29:04Z1, Z2, Z3.
29:07When you compute
29:09all these through multi-head
29:11attention, this can be in any range.
29:14It means there is no upper or lower
29:17limit which is defined for this.
29:20And and because you're multiplying two
29:22numbers
29:24and then doing another multiplication,
29:26there is no defined range for the
29:28self-attention output.
29:31This is not a good thing.
29:33Right? Neural network training is
29:34generally stable when numbers are in
29:37small given range. So, we need some
29:40mechanism to normalize these numbers
29:43into a small given range.
29:45That is what layer normalization is
29:48doing.
29:50Okay? This layer normalization is doing
29:53the same thing.
29:55It's trying to bring those values in
29:56some given range.
29:58Fine?
29:59I hope till this point
30:01I hope so you are able to understand.
30:04Now,
30:06next is feedforward neural network. As
30:09you can see in the original architecture
30:11given,
30:13now we are here.
30:14Okay? We are here.
30:16We have seen
30:18multi-head attention,
30:20add norm, and this residual connection.
30:23Now we are here.
30:25Okay? At this point.
30:28So, next is what? You can see
30:30feedforward neural network.
30:33The feedforward neural network inside an
30:35encoder block is basically a two-layer
30:38neural network.
30:42I'll show you that anyway. It looks
30:43something like this.
30:46Okay? It's a two-layer neural network.
30:53>> [clears throat]
30:53>> You can input 512 dimensional input to
30:55this
30:57neural network, okay?
30:59512 dimension input you can input to
31:01this particular neural network.
31:04The first layer has
31:072048 neurons.
31:09This layer has 2048 neurons.
31:14Each neuron has a ReLU activation
31:16function.
31:18Rectified linear unit activation
31:20function.
31:21The second layer has 512 neurons. This
31:24one here.
31:26And
31:28it is having
31:30activation function which is linear.
31:35Now, let us imagine that Z1 norm, Z2
31:39norm, Z1 norm, Z2 norm, and Z3 norm
31:43let's say they form
31:45a matrix like this, okay? A matrix like
31:48this.
31:50Which is of 3 cross 512.
31:53It means what? It means three words.
31:55Each word
31:57of 512 dimensions.
31:59You can think of this as sending a batch
32:02of data where the batch has three rows.
32:05First row, you see here.
32:08This is first row. This is second row.
32:12And third row.
32:15But all three
32:16go simultaneously.
32:19This goes and multiplies with the
32:22weights. See here. All this goes
32:26All this go here and multiplies with
32:27these weights. How many weights are
32:29there? So, as I've told you,
32:30this is 512 2048. So, it will be like
32:33512 cross
32:352048
32:36weights.
32:38Right? That's why I've told you this
32:39goes and multiplies with the weights of
32:41the first layer, that is 512 to cross
32:442048.
32:45Since both dimensions match,
32:48this is 512 cross 2048.
32:51And here everyone you see,
32:55right? This is This is basically three
32:57cross
32:59This is basically three cross 512. This
33:01dimension and this dimension they match.
33:03You see?
33:05Right?
33:06So, since both the dimensions
33:09match,
33:10you can perform this multiplication.
33:13And what you get is a three cross 2048
33:16dimensional matrix.
33:18You will get three cross 2048
33:21dimensional matrix.
33:23That means three vectors, each of size
33:252048.
33:29Fine?
33:30Then, you add the bias. Obviously, 2048
33:34neurons, that means there will be 2048
33:36biases. Let's say I'll just represent
33:38them with
33:39uh only one value, let's say B1. So, B1
33:42will be added.
33:43Okay?
33:45One for each neuron, one bias for each
33:47neuron, obviously.
33:48After that, everyone, then you apply
33:50ReLU activation function.
33:53Why As I told you, right? ReLU here. So,
33:55why ReLU here? ReLU is for
33:57non-linearity.
33:59Okay? And once that is done, after this
34:01layer, the dimensionality increase from
34:04512 to 2048.
34:09Now, in the second layer here,
34:12you take this three three cross 2048
34:15matrix and multiply it with weights,
34:18that is these weights.
34:21These weights will be how many everyone?
34:23As you can see, they are 2048 and 512.
34:26So, everyone,
34:28these weights in the second layer as as
34:31as you can see as you can see here in
34:33the second layer
34:34uh you take this 3 cross 2048 and
34:37multiplied with weights. How many
34:40weights? 2048 cross this 512.
34:44So, 2048 cross 512.
34:49Since these dimensions also match,
34:52you can do the multiplication. See, they
34:54match. So, once you do the
34:55multiplication everyone, what you get
34:57back is you get back 3 cross
35:00512
35:02matrix. That is three vectors, each of
35:04size 512.
35:07That is what you got here.
35:09Okay? And then you add bias. Let's say I
35:12I'll call that bias. Obviously, there'll
35:14be biases. 512 neurons are there, so 512
35:17bias values biases will be there. So, I
35:19will represent them as B2. So, we
35:21we add bias B2, let's say.
35:25Fine?
35:26So, finally we got 3 cross 512 now.
35:28Obviously, the question in your mind is
35:30what is the benefit here? First, we
35:32increase dimensionality to 2 2048.
35:35Then we reduce it back to 512. So, what
35:38is the benefit of all these things?
35:40We end up with 3 cross 512 matrix again.
35:45So, what was gained? The benefit is that
35:48we were able to introduce non-linearity.
35:54This is the
35:56motive everyone. We were able to reduce
35:57non-linearity through rectified linear
36:00unit ReLU activation function. The
36:02entire
36:03self-attention process is linear.
36:06Remember this. All operations are
36:08linear, but sometimes if you have data
36:10where there is some form of non-linear
36:13complexity, then you need a feed-forward
36:16neural network with non-linear
36:18activation to capture those
36:20non-linearities.
36:22And this feed-forward network is what
36:26introduces non-linearity into the model.
36:30I hope till this point
36:32you got the idea. Okay?
36:35After this, now again, interesting thing
36:39is
36:40we do the same thing again.
36:42We use
36:43residual connection. You see again, this
36:45residual connection?
36:46Right?
36:47In the original architecture, if I want
36:49to show it to you,
36:52so I'm talking about this residual
36:53connection. You see? It is bypassing
36:55feed forward. You see?
36:57This residual connection.
36:59So we use a residual connection
37:01bypassing the feed forward neural
37:02network and add output of feed forward
37:06neural network, whatever the output. So
37:08actually we are here now.
37:10Okay?
37:11Here. So
37:14what happens? As I said, we do the same
37:17thing again. We use residual connection
37:19bypassing the
37:20feed forward neural network.
37:23Okay? Like this.
37:25And add the output of feed forward
37:28neural network with our input. And what
37:30was What was our input again? What was
37:34our input?
37:35As you [clears throat] can see,
37:38our input was
37:41Z1 norm,
37:42Z2 norm, Z3 norm. That's why you see
37:44this this residual connection or cell
37:48skip connection. You see? Y1, Y2, Y3 is
37:50what we have got as output of feed
37:52forward neural network and we are adding
37:54it to the original input. The input
37:57which we got basically from layer
37:59normalization.
38:00Through this residual connection. Z1
38:01norm, Z2 norm, Z3 norm.
38:04Okay?
38:06So our input was Z1 norm, Z2 norm, Z3
38:08norm. We bypass this and bring it where
38:11the feed forward output is sitting. And
38:14feed forward output is sitting here, Y1,
38:15Y2, Y3.
38:18Okay? The feed forward output is Y1, Y2,
38:21Y3. These are also 512 dimensions each.
38:25This one also, this one also.
38:30We add them. That is Y1 plus Z1 now, Y2
38:33plus Z2 now, Y3 plus
38:36Z3 now.
38:38And this gives us, let's say Y1 dash, Y2
38:41dash, Y3 dash.
38:44Again, all are of 512
38:48dimensions.
38:51And now everyone, we apply layer
38:53normalization again. You see here?
38:56Layer normalization. That means in our
38:58original diagram,
39:01we are here, normalization. I'm talking
39:03about this one everyone. After adding,
39:05I'm talking about normalization. And
39:07this normalization is nothing but what?
39:08Layer normalization.
39:11Okay, I hope you understand that.
39:15So, we apply layer normalization,
39:18getting Y1 normalized, Y2 normalized,
39:21and Y3 normalized.
39:25Okay? This is interesting. Y1
39:27normalized, Y2 normalized, Y3 normalized
39:31will go to the next encoder block. You
39:33can see I written here.
39:35To the next encoder block. At the start
39:37only I have told you, right?
39:39How many encoder blocks are there? There
39:41are six encoder blocks. So, the output,
39:44this output which we got here from the
39:46first encoder block, now will go to the
39:49next encoder block. That means everyone,
39:52as I have told you,
39:54>> [clears throat]
39:55>> we are done with this add and norm in
39:58the first encoder block. And whatever
39:59the output which we got here, it will
40:01now go to the second encoder block. That
40:03means in our original architecture, in
40:06our original architecture, we are here.
40:10But,
40:11as I have told you, N N value is six, so
40:13like this six encoder blocks are there,
40:15right? So, [snorts]
40:16that means the output of first encoder
40:19block we got here. As I told you we are
40:20here. This output will go to the
40:23go as a input to the second encoder
40:25block.
40:26That is to the next encoder block.
40:28Just like X1, X2, X3.
40:31Right? Whatever X1, X2, X3 was there
40:35initially.
40:36X1, X2, X3.
40:38Right? Just like X1, X2, X3 were the
40:40input to the first encoder block.
40:42Similarly, this Y1 norm, Y2 norm
40:46and Y3 norm.
40:48Okay? Y1 norm, Y2 norm, Y3 norm.
40:53>> [clears throat]
40:53>> Uh this will
40:56uh become the input to the next encoder
40:59block. Okay? And then
41:02the same things repeat. Whatever we have
41:05seen till now, the same things repeat.
41:08There is also a multi-head attention
41:10block
41:12in the second encoder, everyone.
41:14Obviously, as I have told you they are
41:15same.
41:16Right? So, there is also a multi-head
41:19attention block in the second encoder.
41:21Even though I have not drawn it, but I
41:23think you understand, right? It is
41:24exactly same.
41:26Right? It is exactly same.
41:28I I don't have enough space. That's why
41:31I I did not draw every one of them, but
41:33you you can imagine, right? Second
41:35encoder, whenever I say, it will be
41:36exactly same.
41:38So, as I have told you, there is also a
41:40multi-head attention for second encoder
41:42block.
41:43Right?
41:45We will do
41:47add
41:49and norm after multi-head attention in
41:51the second encoder block as well, like
41:52this. Then there is also a feed forward
41:54neural network
41:56in the second encoder block. And we will
41:58do add and norm in the second encoder
42:00block again after feed forward neural
42:02network.
42:03And we go like this to the
42:05next, that is third encoder block.
42:08And then to the next, fourth encoder
42:10block, fifth encoder block, and then
42:13finally everyone to the sixth encoder
42:16block.
42:19Okay? One important thing you must
42:22remember
42:23and don't get confused about this, okay?
42:25Although each encoder block has the same
42:28architecture,
42:29as I've told you, copy-pasting terms of
42:31structure, the parameters are different
42:33for each block. This block, this block,
42:35this block, parameters are different.
42:38Whatever the weights and biases you have
42:40in multi-head attention
42:43of this block,
42:46that is first block, the weights and
42:47biases in the multi-head attention of
42:49another block, that is second block or
42:50third block, they are different. That is
42:53the meaning.
42:54So, whatever the weights and biases
42:56which are in the feed forward neural
42:59network of one block, the weights and
43:01biases in the feed forward neural
43:03network of another block are different.
43:06Each encoder block has its own parameter
43:09values which keep changing during back
43:11propagation. Please remember this.
43:14Okay?
43:15Fine?
43:16So,
43:20whatever the output everyone which we
43:22got in the sixth encoder, this will go
43:25to the decoder.
43:27Okay? And that output obviously will be
43:29of 3 cross 512 dimension.
43:33Just to summarize all these things all
43:34of you,
43:35just to summarize,
43:37let me show you.
43:40We start from our input example, right?
43:43We started from our input example. What
43:45was our input? As I've given you example
43:47here everyone, the input was this, I am
43:50Yogesh.
43:51In the input block we do
43:53three operations,
43:55tokenization,
43:56embedding, and positional encoding.
44:02And from there we get vectors. Which
44:04one? X1, X2, X3.
44:08First encoder block
44:10in that what we did, we applied
44:12multi-head attention
44:14in the first encoder block, right?
44:17As I've told you, six encoders are
44:18there. So, in the first encoder block
44:19you can see after the input
44:22will come across the multi-head
44:25attention as soon as it enters the first
44:27encoder block.
44:28That means what is given here.
44:31Right? X1, X2, X3 will go through
44:33multi-head attention. That is multiple
44:35self-attentions.
44:38Then
44:40you know, we have done add
44:43residual
44:45right? Add residual and then layer
44:48normalization.
44:51These operations we have done.
44:53Adding through residual and layer
44:55normalization. That is nothing but this
44:57one.
44:58Add and norm. And you see residual
45:00connection, skip connection. What are
45:02the meaning of this? This also we have
45:03discussed.
45:05Then
45:07what we did, we applied feed forward
45:09neural network.
45:11You see?
45:13We were here. Now, feed forward neural
45:15network.
45:17And then add residual, add residual.
45:22And layer normalization again.
45:26That is nothing but
45:29you see it here.
45:31Okay?
45:33Feed forward neural network.
45:36Right? And then
45:38what we have done is we have done
45:39addition and layer normalization, layer
45:42normalization again.
45:45Output becomes
45:46the output of the first encoder block
45:48becomes the input to the second encoder
45:50block. And this repeats how many times?
45:53Six times for all six encoder blocks.
45:58Because we already
46:01discussed that there are six encoder
46:02blocks like this.
46:04And finally, everyone, the output from
46:06the last encoder last encoder block goes
46:10to the decoder.
46:13That means in the original architecture,
46:16if I want to show you everyone,
46:18after six encoder, the output will come
46:20here, and this output will go here in
46:24the decoder. You see? In the decoder.
46:30Okay? A very important observation. See,
46:32this is all summary whatever we are
46:33discussing till now, okay? But a very
46:35important observation, whenever you look
46:37in this entire pipeline,
46:40if you take a sample at any point after
46:43input processing,
46:45you'll always get
46:46a 3 cross 512
46:50vector. This never changes. Everywhere.
46:53Right? From here from here input
46:56processing, everyone, you'll always get
46:573 cross 512.
47:00When you leave this point, I've told
47:02you, right? As a final output as well,
47:04you got what? 3 cross 512 vector.
47:07So, when
47:09when you leave this particular point,
47:10you get 3 cross 512. Like this, when you
47:13pass through last encoder block here
47:15like this, the output you get is 3 cross
47:17512. And this is what you send to the
47:19decoder, as I've told you.
47:21So, this is how
47:23this is how
47:26the encoder part of transformer
47:29architecture works.
47:32We are focused on the encoder part of
47:35the transformer architecture.
47:38I hope so.
47:40You are able to understand the working.
47:42If not, please go and please revise.
47:46Okay? In the next video, everyone, we
47:48will discuss about the decoder part.
47:50I'll provide the link for the next video
47:53in the description itself. Okay? So,
47:55that's it from my side.
47:56Thank you everyone.
47:58Thanks a lot.

About this transcript

This page contains the full transcript of Transformer Architecture Encoder Explained from Scratch (Pen & Paper Deep Dive) - Part1 by yogesh murumkar, generated from the public captions YouTube serves with the video. The transcript has 6,056 words across 1,212 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.