YT YouTube2Text

YouTube2Text → Transcripts → Building makemore Part 3: Activations & Gradients, BatchNorm

Building makemore Part 3: Activations & Gradients, BatchNorm — Transcript

by Andrej Karpathy · 21,386 words · 3,051 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00hi everyone today we are continuing our
0:02implementation of make more now in the
0:04last lecture we implemented the multier
0:06perceptron along the lines of benj 2003
0:08for character level language modeling so
0:10we followed this paper took in a few
0:12characters in the past and used an MLP
0:14to predict the next character in a
0:16sequence so what we'd like to do now is
0:18we'd like to move on to more complex and
0:20larger neural networks like recurrent
0:22neural networks and their variations
0:23like the grw lstm and so on now before
0:27we do that though we have to stick
0:28around the level of malalia perception
0:30on for a bit longer and I'd like to do
0:32this because I would like us to have a
0:33very good intuitive understanding of the
0:35activations in the neural net during
0:37training and especially the gradients
0:39that are flowing backwards and how they
0:41behave and what they look like and this
0:43is going to be very important to
0:45understand the history of the
0:46development of these architectures
0:48because we'll see that recurr neural
0:49networks while they are very expressive
0:51in that they are a universal
0:53approximator and can in principle
0:54Implement uh all the algorithms uh we'll
0:58see that they are not very easily
0:59optimizable with the first order
1:01gradient based techniques that we have
1:02available to us and that we use all the
1:03time and the key to understanding why
1:06they are not optimizable easily is to
1:09understand the the activations and the
1:10gradients and how they behave during
1:11training and we'll see that a lot of the
1:13variants since recur neural networks
1:16have tried to improve that situation and
1:19so that's the path that we have to take
1:21and uh let's get started so the starting
1:23code for this lecture is largely the
1:25code from before but I've cleaned it up
1:27a little bit so you'll see that we are
1:29importing
1:30all the torch and math plb utilities
1:33we're reading in the words just like
1:34before these are eight example words
1:37there's a total of 32,000 of them here's
1:39a vocabulary of all the lowercase
1:41letters and the special dot token here
1:44we are reading the data set and
1:46processing it and um creating three
1:49splits the train Dev and the test split
1:53now in MLP this is the identical same
1:55MLP except you see that I removed a
1:57bunch of magic numbers that we had here
1:59and instead we have the dimensionality
2:01of the embedding space of the characters
2:03and the number of hidden units in the
2:05hidden layer and so I've pulled them
2:06outside here uh so that we don't have to
2:09go and change all these magic numbers
2:10all the time we have the same neural net
2:12with 11,000 parameters that we optimize
2:14now over 200,000 steps with a batch size
2:17of 32 and you'll see that I refactor I
2:20refactored the code here a little bit
2:22but there are no functional changes I
2:23just created a few extra variables a few
2:26more comments and I removed all the
2:28magic numbers and otherwise is the exact
2:30same thing then when we optimize we saw
2:33that our loss looked something like this
2:36we saw that the train and Val loss were
2:38about
2:392.16 and so on here I refactored the uh
2:43code a little bit for the evaluation of
2:45arbitary splits so you pass in a string
2:48of which split you'd like to evaluate
2:50and then here depending on train Val or
2:52test I index in and I get the correct
2:54split and then this is the forward pass
2:56of the network and evaluation of the
2:58loss and printing it so just making that
3:01nicer uh one thing that you'll notice
3:04here is I'm using a decorator torch.
3:06nograd which you can also um look up and
3:09read the documentation of basically what
3:11this decorator does on top of a function
3:14is that whatever happens in this
3:16function is assumed by uh torch to never
3:20require any gradients so it will not do
3:22any of the bookkeeping that it does to
3:25keep track of all the gradients in
3:26anticipation of an eventual backward
3:28pass it's it's almost as if all the
3:30tensors that get created here have a
3:32required grad of false and so it just
3:35makes everything much more efficient
3:36because you're telling torch that I will
3:37not call that backward on any of this
3:39computation and you don't need to
3:41maintain the graph under the hood so
3:44that's what this does and you can also
3:46use a context manager uh with torch du
3:49nograd and you can look those
3:52up then here we have the sampling from a
3:54model um just as before just a for
3:57Passive neural nut getting the
3:58distribution sent from it adjusting the
4:01context window and repeating until we
4:03get the special end token and we see
4:05that we are starting to get much nicer
4:07looking words simple from the model it's
4:10still not amazing and they're still not
4:11fully name like uh but it's much better
4:14than what we had with the BAM
4:16model so that's our starting point now
4:19the first thing I would like to
4:20scrutinize is the
4:21initialization I can tell that our
4:24network is very improperly configured at
4:26initialization and there's multiple
4:28things wrong with it but let's just
4:29start with the first one look here on
4:31the zeroth iteration the very first
4:33iteration we are recording a loss of 27
4:37and this rapidly comes down to roughly
4:38one or two or so so I can tell that the
4:40initialization is all messed up because
4:42this is way too high in training of
4:44neural Nets it is almost always the case
4:46that you will have a rough idea for what
4:48loss to expect at initialization and
4:51that just depends on the loss function
4:52and the problem setup in this case I do
4:55not expect 27 I expect a much lower
4:57number and we can calculate it together
5:00basically at initialization what we like
5:03is that um there's 27 characters that
5:06could come next for any one training
5:08example at initialization we have no
5:10reason to believe any characters to be
5:11much more likely than others and so we'd
5:14expect that the propy distribution that
5:15comes out initially is a uniform
5:18distribution assigning about equal
5:20probability to all the 27
5:22characters so basically what we' like is
5:25the probability for any character would
5:27be roughly 1 over 20
5:307 that is the probability we should
5:32record and then the loss is the negative
5:35log probability so let's wrap this in a
5:38tensor and then then we can take the log
5:41of it and then the negative log
5:43probability is the loss we would expect
5:46which is 3.29 much much lower than 27
5:49and so what's happening right now is
5:51that at initialization the neural nut is
5:53creating probity distributions that are
5:55all messed up some characters are very
5:57confident and some characters are very
5:59not confident confident and then
6:01basically what's happening is that the
6:02network is very confidently wrong and uh
6:06that that's what makes it um record very
6:09high loss so here's a smaller
6:11four-dimensional example of the issue
6:13let's say we only have four characters
6:16and then we have logits that come out of
6:17the neural net and they are very very
6:19close to zero then when we take the
6:21softmax of all zeros we get
6:24probabilities there are a diffused
6:26distribution so sums to one and is
6:29exactly
6:30uniform and then in this case if the
6:32label is say two it doesn't actually
6:34matter if this if the label is two or
6:36three or one or zero because it's a
6:38uniform distribution we're recording the
6:40exact same loss in this case 1.38 so
6:43this is the loss we would expect for a
6:44four-dimensional example and now you can
6:46see of course that as we start to
6:48manipulate these logits uh we're going
6:50to be changing the law here so it could
6:53be that we lock out and by chance uh
6:55this could be a very high number like
6:57you know five or something like that
6:59then case we'll record a very low loss
7:01because we're assigning the correct
7:02probability at initialization by chance
7:04to the correct label much more likely it
7:07is that some other dimension will have a
7:11high uh logit and then what will happen
7:14is we start to record much higher loss
7:17and what can come what can happen is
7:18basically the logits come out like
7:20something like this you know and they
7:22take on Extreme values and we record
7:25really high loss
7:27um for example if we have to 4. random
7:30of four so these are uniform um sorry
7:34these are normally distributed um
7:37numbers uh four of
7:39them then here we can also print the
7:42logits probabilities that come out of it
7:45and the loss and so because these logits
7:48are near zero for the most part the loss
7:51that comes out is is okay uh but suppose
7:54this is like times 10
7:57now you see how because these are more
8:00extreme values it's very unlikely that
8:02you're going to be guessing the correct
8:05bucket and then you're confidently wrong
8:07and recording very high loss if your
8:09loes are coming out even more
8:11extreme you might get extremely insane
8:15losses like infinity even at
8:17initialization
8:19um so basically this is not good and we
8:21want the loges to be roughly zero um
8:25when the network is initialized in fact
8:28the lits can don't have to be just zero
8:30they just have to be equal so for
8:31example if all the logits are one then
8:34because of the normalization inside the
8:35softmax this will actually come out okay
8:38but by symmetry we don't want it to be
8:40any arbitrary positive or negative
8:41number we just want it to be all zeros
8:43and record the loss that we expect at
8:45initialization so let's now concretely
8:47see where things go wrong in our example
8:49here we have the initialization let me
8:51reinitialize the neuronet and here let
8:54me break after the very first iteration
8:56so we only see the initial loss which is
8:5827
9:00so that's way too high and intuitively
9:02now we can expect the variables involved
9:04and we see that the logits here if we
9:06just print some of
9:08these if we just print the first row we
9:11see that the Lo just take on quite
9:12extreme values and that's what's
9:14creating the fake confidence in
9:16incorrect answers and makes the loss um
9:20get very very high so these loes should
9:22be much much closer to zero so now let's
9:25think through how we can achieve logits
9:28coming out of this neur not to be more
9:30closer to zero you see here that loes
9:33are calculated as the hidden states
9:35multip by W2 plus B2 so first of all
9:38currently we're initializing B2 as
9:40random values uh of the right size but
9:45because we want roughly zero we don't
9:46actually want to be adding a bias of
9:48random numbers so in fact I'm going to
9:50add a times zero here to make sure that
9:52B2 is just um basically zero at
9:56initialization and second this is H
9:58multip by W2 so if we want logits to be
10:01very very small then we would be
10:03multiplying W2 and making that
10:06smaller so for example if we scale down
10:08W2 by 0.1 all the elements then if I do
10:12again just a very first iteration you
10:14see that we are getting much closer to
10:16what we expect so rough roughly what we
10:18want is about
10:193.29 this is
10:214.2 I can make this maybe even
10:24smaller 3.32 okay so we're getting
10:27closer and closer now you're probably
10:30wondering can we just set this to zero
10:33then we get of course exactly what we're
10:34looking for um at
10:37initialization and the reason I don't
10:39usually do this is because I'm I'm very
10:41nervous and I'll show you in a second
10:43why you don't want to be setting W's or
10:46weights of a neural nut exactly to zero
10:48um you you usually want it to be small
10:50numbers instead of exactly zero um for
10:53this output layer in this specific case
10:55I think it would be fine but I'll show
10:57you in a second where things go wrong
10:58very quick quickly if you do that so
11:00let's just go with
11:010.01 in that case our loss is close
11:04enough but has some entropy it's not
11:07exactly zero it's got some little
11:09entropy and that's used for symmetry
11:11breaking as we'll see in a second the
11:13logits are now coming out much closer to
11:15zero and everything is well and good so
11:18if I just erase these and I now take
11:22away the break
11:23statement we can run the optimization
11:26with this new initialization and let's
11:28just see
11:30what losses we record okay so I let it
11:33run and you see that we started off good
11:35and then we came down a
11:37bit the plot of the loss uh now doesn't
11:40have this hockey shape appearance um
11:43because basically what's happening in
11:44the hockey stick the very first few
11:46iterations of the loss what's happening
11:48during the optimization is the
11:50optimization is just squashing down the
11:52logits and then it's rearranging the
11:54logits so basically we took away this
11:56easy part of the loss function where
11:58just the the weights were just being
12:00shrunk down and so therefore we're we
12:03don't we don't get these easy gains in
12:04the beginning and we're just getting
12:06some of the hard gains of training the
12:07actual neural nut and so there's no
12:09hockey stick appearance so good things
12:12are happening in that both number one
12:14losset initialization is what we expect
12:17and the the loss doesn't look like a
12:19hockey stick and this is true for any
12:21neuron that you might train um and
12:23something to look out for and second the
12:26loss that came out is actually quite a
12:28bit improved unfortunately I erased what
12:30we had here before I believe this was 2.
12:33um2 and this was this was 2.16 so we get
12:37a slightly improved result and the
12:40reason for that is uh because we're
12:41spending more Cycles more time
12:44optimizing the neuronet actually instead
12:46of just uh spending the first several
12:49thousand iterations probably just
12:50squashing down the
12:52weights because they are so way too high
12:54in the beginning in the initialization
12:56so something to look out for and uh
12:58that's number one now let's look at the
13:00second problem let me reinitialize our
13:02neural net and let me reintroduce The
13:04Brak statement so we have a reasonable
13:07initial loss so even though everything
13:09is looking good on the level of the loss
13:10and we get something that we expect
13:12there's still a deeper problem looking
13:14inside this neural net and its
13:16initialization so the logits are now
13:19okay the problem now is with the values
13:21of H the activations of the Hidden
13:24States now if we just visualize this
13:26Vector sorry this tensor h it's kind of
13:29hard to see but the problem here roughly
13:31speaking is you see how many of the
13:32elements are one or negative 1 now
13:36recall that torch. 10 the 10 function is
13:39a squashing function it takes arbitrary
13:41numbers and it squashes them into a
13:42range of negative 1 and one and it does
13:44so smoothly so let's look at the
13:46histogram of H to get a better idea of
13:49the distribution of the values inside
13:51this tensor we can do this
13:54first well we can see that H is 32
13:57examples and 200 activations in each
14:00example we can view it as1 to stretch it
14:03out into one large
14:05vector and we can then call two list to
14:08convert this into one large python list
14:12of floats and then we can pass this into
14:15PLT doist for histogram and we say we
14:18want 50 bins and a semicolon to suppress
14:21a bunch of output we don't
14:23want so we see this histogram and we see
14:25that most the values by far take on
14:28value of netive one and one so this 10 H
14:31is very very active and we can also look
14:34at basically why that is we can look at
14:38the pre activations that feed into the
14:4110 and we can see that the distribution
14:44of the pre activations are is very very
14:46broad these take numbers between -5 and
14:4915 and that's why in a torure 10
14:52everything is being squashed and capped
14:53to be in the range of negative 1 and one
14:55and lots of numbers here take on very
14:57extreme values now if you are new to
15:00neural networks you might not actually
15:01see this as an issue but if you're well
15:04vered in the dark arts of back
15:05propagation and then having an intuitive
15:07sense of how these gradients flow
15:09through a neural net you are looking at
15:11your distribution of 10h activations
15:13here and you are sweating so let me show
15:15you why we have to keep in mind that
15:17during back propagation just like we saw
15:18in microad we are doing backward passs
15:21starting at the loss and flowing through
15:22the network backwards in particular
15:25we're going to back propagate through
15:26this torch.
15:2710h and this layer here is made up of
15:30200 neurons for each one of these
15:32examples and uh it implements an
15:35elementwise 10 so let's look at what
15:37happens in 10h in the backward pass we
15:39can actually go back to our previous uh
15:41microgr code in the very first lecture
15:44and see how we implemented 10 AG we saw
15:47that the input here was X and then we
15:49calculate T which is the 10 age of X so
15:52that's T and T is between 1 and 1 it's
15:54the output of the 10 H and then in the
15:56backward pass how do we back propagate
15:58through a 10 H we take out that grad um
16:02and then we multiply it this is the
16:04chain rule with the local gradient which
16:06took the form of 1 - t ^2 so what
16:09happens if the outputs of your t h are
16:11very close to1 or 1 if you plug in t one
16:15here you're going to get a zero
16:17multiplying out. grad no matter what
16:20out. grad is we are killing the gradient
16:22and we're stopping effectively the back
16:25propagation through this 10 unit
16:27similarly when t is1 this will again
16:29become zero and out that grad just stops
16:33and intuitively this makes sense because
16:35this is a 10h
16:36neuron and what's happening is if its
16:39output is very close to one then we are
16:41in the tail of this
16:4310 and so changing basically the
16:48input is not going to impact the output
16:50of the 10 too much because it's it's so
16:53it's in a flat region of the 10 H and so
16:56therefore there's no impact on the loss
16:58and so so indeed the the weights and the
17:02biases along with the 10h neuron do not
17:04impact the loss because the output of
17:06the 10 unit is in the flat region of the
17:0810 and there's no influence we can we
17:10can be changing them whatever we want
17:12however we want and the loss is not
17:13impacted that's so that's another way to
17:15justify that indeed the gradient would
17:17be basically zero it
17:19vanishes indeed uh when T equals zero we
17:24get one times out that grad so when the
17:2810 h takes on exactly value of zero then
17:31out grad is just passed through so
17:35basically what this is doing right is if
17:37T is equal to zero then this the 10 unit
17:40is uh sort of inactive and uh gradient
17:43just passes through but the more you are
17:45in the flat tails the more the gradient
17:48is squashed so in fact you'll see that
17:50the the gradient flowing through 10 can
17:53only ever decrease and the amount that
17:55it decreases is um proportional through
17:59a square here um depending on how far
18:02you are in the flat tail so this 10 H
18:05and so that's kind of what's Happening
18:06Here and through this the concern here
18:09is that if all of these um outputs H are
18:12in the flat regions of negative 1 and
18:14one then the gradients that are flowing
18:16through the network will just get
18:18destroyed at this
18:20layer now there is some redeeming
18:23quality here and that we can actually
18:25get a sense of the problem here as
18:26follows I wrote some code here and
18:29basically what we want to do here is we
18:30want to take a look at H take the the
18:33absolute value and see how often it is
18:36in the in a flat uh region so say
18:40greater than
18:41099 and what you get is the following
18:44and this is a Boolean tensor so uh in
18:47the Boolean tensor you get a white if
18:49this is true and a black if this is
18:51false and so basically what we have here
18:53is the 32 examples and 200 hidden
18:56neurons and we see that a lot of this is
18:59white and what that's telling us is that
19:01all these 10h neurons were very very
19:05active and uh they're in a flat tail and
19:09so in all these cases uh the back the
19:12backward gradient would get uh
19:15destroyed now we would be in a lot of
19:18trouble if for for any one of these 200
19:21neurons if it was the case that the
19:24entire column is white because in that
19:26case we have what's called a dead neuron
19:28and this is could be a 10 neuron where
19:30the initialization of the weights and
19:31the biases could be such that no single
19:33example ever activates uh this 10h in
19:37the um sort of active part of the 10age
19:39if all the examples land in the tail
19:42then this neuron will never learn it is
19:44a dead neuron and so just scrutinizing
19:48this and looking for Columns of
19:50completely white uh we see that this is
19:53not the case so uh I don't see a single
19:56neuron that is all of uh you know white
19:59and so therefore it is the case that for
20:01every one of these 10h neurons uh we do
20:04have some examples that activate them in
20:07the uh active part of the 10 and so some
20:09gradients will flow through and this
20:10neuron will learn and the neuron will
20:13change and it will move and it will do
20:15something but you can sometimes get get
20:17yourself in cases where you have dead
20:19neurons and the way this manifests is
20:21that um for 10h neuron this would be
20:24when no matter what inputs you plug in
20:26from your data set this 10h neuron
20:28always fir
20:29completely one or completely negative
20:30one and then it will just not learn
20:33because all the gradients will be just
20:34zeroed out uh this is true not just for
20:3710 but for a lot of other nonlinearities
20:39that people use in neural networks so we
20:41certainly used 10 a lot but sigmoid will
20:43have the exact same issue because it is
20:45a squashing neuron and so the same will
20:48be true for sigmoid uh but um but um you
20:53know um basically the same will actually
20:55apply to sigmoid the same will also
20:57apply to reu
20:59so reu has a completely flat region here
21:02below zero so if you have a reu neuron
21:05then it is a pass through um if it is
21:07positive and if it's if the
21:09preactivation is negative it will just
21:11shut it off since the region here is
21:13completely flat then during back
21:16propagation uh this would be exactly
21:18zeroing out the gradient um like all of
21:21the gradient would be set exactly to
21:22zero instead of just like a very very
21:24small number depending on how positive
21:26or negative T is and so you can get for
21:29example a dead reu neuron and a dead reu
21:32neuron would basically look like
21:35basically what it is is if a neuron with
21:37a reu nonlinearity never activates so
21:41for any examples that you plug in in the
21:43data set it never turns on it's always
21:45in this flat region then this re neuron
21:48is a dead neuron its weights and bias
21:51will never learn they will never get a
21:52gradient because the neuron never
21:54activated and this can sometimes happen
21:56at initialization uh because the way and
21:58a biases just make it so that by chance
22:00some neurons are just forever dead but
22:02it can also happen during optimization
22:04if you have like a too high of learning
22:06rate for example sometimes you have
22:08these neurons that get too much of a
22:09gradient and they get knocked out off
22:11the data
22:12manifold and what happens is that from
22:14then on no example ever activates this
22:17neuron so this neuron remains dead
22:18forever so it's kind of like a permanent
22:20brain damage in a in a mind of a network
22:23and so sometimes what can happen is if
22:25your learning rate is very high for
22:26example and you have a neural net with
22:28neurons you train the neuron net and you
22:30get some last loss but then actually
22:33what you do is you go through the entire
22:35training set and you forward um your
22:38examples and you can find neurons that
22:40never activate they are dead neurons in
22:43your network and so those neurons will
22:45will never turn on and usually what
22:47happens is that during training these
22:48Rel neurons are changing moving Etc and
22:50then because of a high gradient
22:52somewhere by chance they get knocked off
22:54and then nothing ever activates them and
22:56from then on they are just dead uh so
22:59that's kind of like a permanent brain
23:00damage that can happen to some of these
23:02neurons these other nonlinearities like
23:04leyu will not suffer from this issue as
23:06much because you can see that it doesn't
23:08have flat Tails you'll almost always get
23:12gradients and uh elu is also fairly uh
23:14frequently used um it also might suffer
23:17from this issue because it has flat
23:19parts so that's just something to be
23:21aware of and something to be concerned
23:23about and in this case we have way too
23:26many um activations AG that take on
23:29Extreme values and because there's no
23:31column of white I think we will be okay
23:34and indeed the network optimizes and
23:35gives us a pretty decent loss but it's
23:38just not optimal and this is not
23:39something you want especially during
23:41initialization and so basically what's
23:43happening is that uh this H
23:45preactivation that's floating to 10 H
23:48it's it's too extreme it's too large
23:51it's creating very um it's creating a
23:54distribution that is too saturated in
23:55both sides of the 10 H and it's not
23:57something you want because it means that
23:59there's less training uh for these
24:01neurons because they update um less
24:04frequently so how do we fix this well H
24:07preactivation is MCAT which comes from C
24:12so these are uniform gsan but then it's
24:15multiply by W1 plus B1 and H preact is
24:18too far off from zero and that's causing
24:20the issue so we want this reactivation
24:23to be closer to zero very similar to
24:25what we had with
24:26logits so here
24:28we want actually something very very
24:30similar now it's okay to set the biases
24:33to very small number we can either
24:35multiply by 0 01 to get like a little
24:37bit of entropy um I sometimes like to do
24:40that um just so that there's like a
24:43little bit of variation and diversity in
24:45the original initialization of these 10
24:48H neurons and I find in practice that
24:50that can help optimization a little bit
24:53and then the weights we can also just
24:55like squash so let's multiply everything
24:57by 0.1
24:59let's rerun the first batch and now
25:01let's look at this and well first let's
25:04look
25:06here you see now because we multiply dou
25:08by 0.1 we have a much better histogram
25:11and that's because the pre activations
25:12are now between 1.5 and 1.5 and this we
25:15expect much much less white okay there's
25:19no white so basically that's because
25:22there are no neurons that saturated
25:24above 99 in either direction so this
25:28actually a pretty decent place to be um
25:31maybe we can go up a little
25:35bit sorry am I am I changing W1 here so
25:39maybe we can go to 0
25:412 okay so maybe something like this is
25:44is a nice distribution so maybe this is
25:47what our initialization should be so let
25:49me now
25:50erase
25:52these and let me starting with
25:55initialization let me run the full
25:57optimization
25:58without the break and uh let's see what
26:02we get okay so the optimization finished
26:04and I re the loss and this is the result
26:06that we get and then just as a reminder
26:09I put down all the losses that we saw
26:10previously in this lecture so we see
26:13that we actually do get an improvement
26:14here and just as a reminder we started
26:16off with a validation loss of 2.17 when
26:19we started by fixing the softmax being
26:21confidently wrong we came down to 2.13
26:24and by fixing the 10h layer being way
26:25too saturated we came down to 2.10
26:28and the reason this is happening of
26:30course is because our initialization is
26:31better and so we're spending more time
26:33doing productive training instead of um
26:36not very productive training because our
26:38gradients are set to zero and uh we have
26:41to learn very simple things like uh the
26:43overconfidence of the softmax in the
26:44beginning and we're spending Cycles just
26:46like squashing down the weight Matrix so
26:50this is illustrating um basically
26:52initialization and its impacts on
26:54performance uh just by being aware of
26:57the internals of these neural net and
26:58their activations their gradients now
27:01we're working with a very small Network
27:02this is just one layer multi-layer
27:04perception so because the network is so
27:07shallow the optimization problem is
27:08actually quite easy and very forgiving
27:11so even though our initialization was
27:12terrible the network still learned
27:14eventually it just got a bit worse
27:16result this is not the case in general
27:19though once we actually start um working
27:21with much deeper networks that have say
27:2350 layers uh things can get uh much more
27:26complicated and uh these problems stack
27:29up and so you can actually get into a
27:32place where the network is basically not
27:34training at all if your initialization
27:35is bad enough and the deeper your
27:38network is and the more complex it is
27:39the less forgiving it is to some of
27:41these errors and so um something to
27:45definitely be aware of and uh something
27:47to scrutinize something to plot and
27:49something to be careful with and um yeah
27:53okay so that's great that that worked
27:54for us but what we have here now is all
27:57these magic numbers like0 2 like where
27:59do I come up with this and how am I
28:01supposed to set these if I have a large
28:02neural net with lots and lots of layers
28:05and so obviously no one does this by
28:07hand there's actually some relatively
28:08principled ways of setting these scales
28:11um that I would like to introduce to you
28:13now so let me paste some code here that
28:15I prepared just to motivate the
28:17discussion of
28:18this so what I'm doing here is we have
28:21some random input here x that is drawn
28:23from a gan and there's 1,000 examples
28:27that are 10 dimensional
28:28and then we have a waiting layer here
28:30that is also initialized using caution
28:33just like we did here and we these
28:36neurons in the hidden layer look at 10
28:37inputs and there are 200 neurons in this
28:40hidden layer and then we have here just
28:43like here um in this case the
28:45multiplication X multip by W to get the
28:47pre activations of these
28:49neurons and basically the analysis here
28:52looks at okay suppose these are uniform
28:54gion and these weights are uniform gion
28:57if I do X W and we forget for now the
29:00bias and the
29:02nonlinearity then what is the mean and
29:04the standard deviation of these gions so
29:07in the beginning here the input is uh
29:09just a normal Gan distribution mean zero
29:11and the standard deviation is one and
29:13the standard deviation again is just the
29:15measure of a spread of the
29:17gion but then once we multiply here and
29:19we look at the um histogram of Y we see
29:23that the mean of course stays the same
29:25it's about zero because this is a
29:27symmetric operation but we see here that
29:29the standard deviation has expanded to
29:31three so the input standard deviation
29:33was one but now we've grown to three and
29:36so what you're seeing in the histogram
29:37is that this Gan is
29:40expanding and so um we're expanding this
29:43Gan um from the input and we don't want
29:46that we want most of the neural net to
29:48have relatively similar activations uh
29:50so unit gion roughly throughout the
29:52neural net and so the question is how do
29:54we scale these W's to preserve the uh um
29:58to preserve this distribution to uh
30:01remain
30:02aan and so intuitively if I multiply
30:05here uh these elements of w by a larger
30:09number let's say by
30:11five then this gsan gross and gross in
30:14standard deviation so now we're at 15 so
30:17basically these numbers here in the
30:19output y take on more and more extreme
30:21values but if we scale it down like .2
30:25then conversely this Gan is getting
30:28smaller and smaller and it's shrinking
30:31and you can see that the standard
30:32deviation is 6 and so the question is
30:34what do I multiply by here to exactly
30:37preserve the standard deviation to be
30:39one and it turns out that the correct
30:42answer mathematically when you work out
30:43through the variance of uh this
30:45multiplication here is that you are
30:48supposed to divide by the square root of
30:51the fan in the fan in is the basically
30:54the uh number of input elements here 10
30:58so we are supposed to divide by 10
30:59square root and this is one way to do
31:01the square root you raise it to a power
31:03of 0. five that's the same as doing a
31:05square root so when you divide by the um
31:09square root of 10 then we see that the
31:12output caution it has exactly standard
31:15deviation of one now unsurprisingly a
31:18number of papers have looked into how
31:21but to best initialized neural networks
31:23and in the case of multilayer
31:24perceptrons we can have fairly deep
31:26networks that have these nonlinearity in
31:28between and we want to make sure that
31:30the activations are well behaved and
31:32they don't expand to infinity or Shrink
31:34all the way to zero and the question is
31:35how do we initialize the weights so that
31:37these activations take on reasonable
31:38values throughout the network now one
31:41paper that has studied this in quite a
31:43bit of detail that is often referenced
31:45is this paper by King hatal called
31:47delving deep into rectifiers now in this
31:49case they actually study convolution
31:51neur neurals and they study especially
31:54the reu nonlinearity and the p
31:56nonlinearity instead of a 10h
31:58nonlinearity but the analysis is very
32:00similar and um basically what happens
32:04here is for them the the relu
32:06nonlinearity that they care about quite
32:08a bit here is a squashing function where
32:11all the negative numbers are simply
32:14clamped to zero so the positive numbers
32:16are pass through but everything negative
32:18is just set to zero and because uh you
32:21are basically throwing away half of the
32:23distribution they find in their analysis
32:25of the forward activations in the neural
32:28that you have to compensate for that
32:29with a
32:31gain and so here they find that
32:34basically when they initialize their
32:36weights they have to do it with a zero
32:38mean Gan whose standard deviation is
32:40square < TK of 2 over the Fanon what we
32:43have here is we are initializing gashin
32:46with the square root of Fanon this NL
32:49here is the Fanon so what we have is
32:52sare root of one over the Fanon because
32:55we have the division here
32:58now they have to add this factor of two
33:00because of the reu which basically
33:02discards half of the distribution and
33:04clamps it at zero and so that's where
33:06you get an additional Factor now in
33:08addition to that this paper also studies
33:10not just the uh sort of behavior of the
33:12activations in the forward pass of the
33:14neural net but it also studies the back
33:16propagation and we have to make sure
33:18that the gradients also are well behaved
33:21and so um because ultimately they end up
33:23updating our parameters and what they
33:26find here through a lot of analysis that
33:28I invite you to read through but it's
33:29not exactly approachable what they find
33:32is basically if you properly initialize
33:35the forward pass the backward pass is
33:36also approximately initialized up to a
33:40constant factor that has to do with the
33:42size of the number of um hidden neurons
33:45in an early and a late
33:48layer and uh but basically they find
33:50empirically that this is not a choice
33:52that matters too much now this timing
33:54initialization is also implemented in
33:57pytorch so if you go to torch. and then.
33:59init documentation you'll find climing
34:01normal and in my opinion this is
34:03probably the most common way of
34:05initializing neural networks now and it
34:07takes a few keyword arguments here so
34:09number one it wants to know the mode
34:12would you like to normalize the
34:14activations or would you like to
34:15normalize the gradients to to be always
34:18uh gsh in with zero mean and a unit or
34:20one standard deviation and because they
34:23find in the paper that this doesn't
34:24matter too much most of the people just
34:25leave it as the default which is Fan in
34:28and then second passing the nonlinearity
34:30that you are using because depending on
34:32the nonlinearity we need to calculate a
34:34slightly different gain and so if your
34:36nonlinearity is just um linear so
34:39there's no nonlinearity then the gain
34:41here will be one and we have the exact
34:43same uh kind of formula that we've come
34:45up here but if the nonlinearity is
34:47something else we're going to get a
34:48slightly different gain and so if we
34:50come up here to the top we see that for
34:52example in the case of reu this gain is
34:55a square root of two and the reason it's
34:56a square root because in this
35:02paper you see how the two is inside of
35:05the square root so the gain is a square
35:07root of two in the case of linear or
35:11identity we just get a gain of one in a
35:14case of 10 H which is what we're using
35:15here the advised gain is a 5 over3 and
35:19intuitively why do we need a gain on top
35:21of the initialization is because 10 just
35:23like reu is a contractive uh
35:26transformation so that means is you're
35:28taking the output distribution from this
35:30matrix multiplication and then you are
35:32squashing it in some way now reu
35:34squashes it by taking everything below
35:35zero and clamping it to zero 10 also
35:38squashes it because it's a contractive
35:40operation it will take the Tails and it
35:42will squeeze them in and so in order to
35:45fight the squeezing in we need to boost
35:47the weights a little bit so that we
35:49renormalize everything back to standard
35:51unit standard deviation so that's why
35:54there's a little bit of a gain that
35:55comes out now I'm skipping through this
35:57section A little bit quickly and I'm
35:59doing that actually intentionally and
36:01the reason for that is because about 7
36:03years ago when this paper was written
36:06you had to actually be extremely careful
36:07with the activations and ingredients and
36:09their ranges and their histograms and
36:11you had to be very careful with the
36:13precise setting of gains and the
36:14scrutinizing of the nonlinearities used
36:16and so on and everything was very
36:18finicky and very fragile and to be very
36:20properly arranged for the neural nut to
36:22train especially if your neural nut was
36:23very deep but there are a number of
36:25modern innovations that have made
36:26everything significantly more stable and
36:28more well behaved and it's become less
36:30important to initialize these networks
36:32exactly right and some of those modern
36:34Innovations for example are residual
36:36connections which we will cover in the
36:38future the use of a number of uh
36:41normalization uh layers like for example
36:43batch normalization layer normalization
36:45group normalization we're going to go
36:47into a lot of these as well and number
36:49three much better optimizers not just
36:51stochastic gradient descent the simple
36:53Optimizer we're basically using here but
36:55a slightly more complex optimizers like
36:57ARS prop and especially Adam and so all
37:00of these modern Innovations make it less
37:02important for you to precisely calibrate
37:03the neutralization of the neural net all
37:06that being said in practice uh what
37:08should we do in practice when I
37:10initialize these neurals I basically
37:12just uh normalize my weights by the
37:14square root of the Fanon uh so basically
37:17uh roughly what we did here is what I do
37:20now if we want to be exactly accurate
37:22here we and go by um in it of uh timing
37:26normal this is how it would implemented
37:29we want to set the standard deviation to
37:31be gain over the square root of fan in
37:34right so to set the standard deviation
37:37of our weights we will proceed as
37:40follows basically when we have a torch.
37:42Ranon and let's say I just create a th
37:45numbers we can look at the standard
37:46deviation of this and of course that's
37:48one that's the amount of spread let's
37:50make this a bit bigger so it's closer to
37:51one so that's the spread of the Gan of
37:55zero mean and unit standard deviation
37:58now basically when you take these and
37:59you multiply by
38:01say2 that basically scales down the Gan
38:04and that makes it standard deviation 02
38:07so basically the number that you
38:08multiply by here ends up being the
38:09standard deviation of this caution so
38:12here this is a um standard deviation
38:15point2 caution here when we sample our
38:18W1 but we want to set the standard
38:20deviation to gain over square root of
38:23fan mode which is Fanon so in other
38:26words we want to mul mly by uh gain
38:29which for 10 H is 5
38:33over3 5 over3 is the gain and then
38:38times
38:43um or I guess sorry
38:46divide uh square root of the fan in and
38:51in this example here the fan in was 10
38:53and I just noticed that actually here
38:55the fan in for W1 is is actually an
38:58embed times block size which as you all
39:00recall is actually 30 and that's because
39:02each character is 10 dimensional but
39:04then we have three of them and we can
39:05catenate them so actually the fan in
39:06here was 30 and I should have used 30
39:09here probably but basically we want 30
39:11uh square root so this is the number
39:14this is what our standard deviation we
39:16want to be and this number turns out to
39:18be3 whereas here just by fiddling with
39:21it and looking at the distribution and
39:22making sure it looks okay uh we came up
39:24with 02 and so instead what we want to
39:27do here is we want to make the standard
39:29deviation b
39:32um 5 over3 which is our gain
39:36divide this
39:38amount times2 square root and these
39:41brackets here are not that uh necessary
39:44but I'll just put them here for clarity
39:46this is basically what we want this is
39:47the timing in it in our case for a 10h
39:51nonlinearity and this is how we would
39:53initialize the neural net and so we're
39:55multiplying by .3 instead of multiplying
39:58by
40:00.2 and so we can we can initialize this
40:04way and then we can train the neural net
40:06and see what we get okay so I trained
40:08the neural net and we end up in roughly
40:11the same spot so looking at the
40:12validation loss we now get 2.10 and
40:15previously we also had 2.10 there's a
40:17little bit of a difference but that's
40:19just the randomness of the process I
40:20suspect but the big deal of course is we
40:22get to the same spot but we did not have
40:25to introduce any um magic numbers that
40:29we got from just looking at histograms
40:31and guessing checking we have something
40:32that is semi- principled and will scale
40:34us to uh much bigger networks and uh
40:37something that we can sort of use as a
40:39guide so I mentioned that the precise
40:41setting of these initializations is not
40:43as important today due to some Modern
40:45Innovations and I think now is a pretty
40:46good time to introduce one of those
40:48modern Innovations and that is batch
40:50normalization so bat normalization came
40:52out in uh 2015 from a team at Google and
40:56it was an extremely impact paper because
40:58it made it possible to train very deep
41:00neuron Nets quite reliably and uh it
41:03basically just worked so here's what
41:05bash rization does and let's implement
41:07it
41:08um basically we have these uh hidden
41:11States H preact right and we were
41:14talking about how we don't want these uh
41:16these um preactivation states to be way
41:19too small because the then the 10 H is
41:21not um doing anything but we don't want
41:24them to be too large because then the 10
41:25H is saturated in fact we want them to
41:28be roughly roughly Gan so zero mean and
41:31a unit or one standard deviation at
41:34least at
41:35initialization so the Insight from the
41:37bachor liation paper is okay you have
41:39these hidden States and you'd like them
41:42to be roughly Gan then why not take the
41:44hidden States and uh just normalize them
41:47to be
41:47Gan and it sounds kind of crazy but you
41:50can just do that because uh
41:53standardizing hidden States so that
41:55their unit caution is a perfect ly
41:57differentiable operation as we'll soon
41:58see and so that was kind of like the big
42:00Insight in this paper and when I first
42:02read it my mind was blown because you
42:04can just normalize these hidden States
42:06and if you'd like unit Gan States in
42:08your network uh at least initialization
42:11you can just normalize them to be unit
42:13gion so uh let's see how that works so
42:16we're going to scroll to our
42:17preactivation here just before they
42:19enter into the 10h now the idea again is
42:22remember we're trying to make these
42:23roughly Gan and that's because if these
42:25are way too small numbers then the 10 H
42:28here is kind of inactive but if these
42:30are very large numbers then the 10 H is
42:33way too saturated and gr is no flow so
42:36we'd like this to be roughly goshan so
42:39the Insight in Bat normalization again
42:41is that we can just standardize these
42:43activations so they are exactly Gan so
42:47here H
42:48preact has a shapee of 32 by 200 32
42:52examples by 200 neurons in the hidden
42:54layer so basically what we can do is we
42:57can take H pract and we can just
42:59calculate the mean um and the mean we
43:02want to calculate across the zero
43:04Dimension and we want to also keep them
43:06as true so that we can easily broadcast
43:10this so the shape of this is 1 by 200 in
43:14other words we are doing the mean over
43:17all the uh elements in the
43:19batch and similarly we can calculate the
43:22standard deviation of these
43:25activations and that will also be 1 by
43:28200 now in this paper they have
43:31the uh sort of prescription here and see
43:35here we are calculating the mean which
43:37is just taking uh the average
43:40value of any neurons activation and then
43:44the standard deviation is basically kind
43:45of like um this the measure of the
43:48spread that we've been using which is
43:50the distance of every one of these
43:53values away from the mean and that
43:56squared and
43:57averaged that's the that's the variance
44:01and then if you want to take the
44:02standard deviation you would square root
44:04the variance to get the standard
44:06deviation so these are the two that
44:09we're calculating and now we're going to
44:10normalize or standardize these X's by
44:13subtracting the mean and um dividing by
44:16the standard deviation so basically
44:18we're taking in pract and we
44:22subtract the mean
44:29and then we divide by the standard
44:33deviation this is exactly what these two
44:35STD and mean are calculating
44:39oops sorry this is the mean and this is
44:42the variance you see how the sigma is a
44:44standard deviation usually so this is
44:45Sigma Square which the variance is the
44:47square of the standard
44:49deviation so this is how you standardize
44:52these values and what this will do is
44:54that every single neuron now and its
44:56firing rate will be exactly unit Gan on
44:59these 32 examples at least of this batch
45:01that's why it's called batch
45:02normalization we are normalizing these
45:05batches and then we could in principle
45:08train this notice that calculating the
45:10mean and your standard deviation these
45:12are just mathematical formulas they're
45:13perfectly differentiable all of this is
45:15perfectly differentiable and we can just
45:17train this the problem is you actually
45:19won't achieve a very good result with
45:22this and the reason for that
45:24is we want these to be roughly Gan but
45:27only at initialization uh but we don't
45:30want these be to be forced to be Garian
45:33always we we'd like to allow the neuron
45:35net to move this around to potentially
45:38make it more diffuse to make it more
45:39sharp to make some 10 neurons maybe be
45:42more trigger more trigger happy or less
45:44trigger happy so we'd like this
45:46distribution to move around and we'd
45:47like the back propagation to tell us how
45:49the distribution should move around and
45:52so in addition to this idea of
45:54standardizing the activations that any
45:57point in the network uh we have to also
46:00introduce this additional component in
46:01the paper here described as scale and
46:04shift and so basically what we're doing
46:06is we're taking these normalized inputs
46:09and we are additionally scaling them by
46:11some gain and offsetting them by some
46:13bias to get our final output from this
46:16layer and so what that amounts to is the
46:19following we are going to allow a batch
46:21normalization gain to be initialized at
46:25just uh once
46:27and the ones will be in the shape of 1
46:29by n
46:31hidden and then we also will have a BN
46:34bias which will be torch. zeros and it
46:37will also be of the shape n by 1 by n
46:41hidden and then here the BN gain will
46:45multiply
46:46this and the BN bias will offset it
46:50here so because this is initialized to
46:52one and this to
46:53zero at initialization each neurons
46:57firing values in this batch will be
47:00exactly unit gion and will have nice
47:02numbers no matter what the distribution
47:04of the H pract is coming in coming out
47:07it will be un Gan for each neuron and
47:09that's roughly what we want at least at
47:12initialization um and then during
47:14optimization we'll be able to back
47:16propagate into BN gain and BM bias and
47:18change them so the network is given the
47:20full ability to do with this whatever it
47:23wants uh
47:24internally here we just have to make
47:26sure sure that we um include these in
47:30the parameters of the neural nut because
47:32they will be trained with back
47:34propagation so let's initialize this and
47:37then we should be able to
47:44train and then we're going to also copy
47:48this line which is the batch
47:50normalization layer here on a single
47:52line of code and we're going to swing
47:54down here and we're also going to do the
47:56exact same thing at test time
48:00here so similar to train time we're
48:03going to normalize uh and then scale and
48:06that's going to give us our train and
48:08validation
48:09loss and we'll see in a second that
48:11we're actually going to change this a
48:12little bit but for now I'm going to keep
48:14it this way so I'm just going to wait
48:16for this to converge okay so I allowed
48:18the neural nut to converge here and when
48:20we scroll down we see that our
48:21validation loss here is 2.10 roughly
48:24which I wrote down here and we see that
48:26this is actually kind of comparable to
48:28some of the results that we've achieved
48:29uh previously now I'm not actually
48:32expecting an improvement in this case
48:34and that's because we are dealing with a
48:36very simple neural nut that has just a
48:37single hidden layer so in fact in this
48:41very simple case of just one hidden
48:42layer we were able to actually calculate
48:44what the scale of w should be to make
48:47these pre activations already have a
48:48roughly Gan shape so the bat
48:50normalization is not doing much here but
48:53you might imagine that once you have a
48:54much deeper neural nut that has lots of
48:56different types of operations and
48:59there's also for example residual
49:00connections which we'll cover and so on
49:02it will become basically very very
49:05difficult to tune the scales of your
49:07weight matrices such that all the
49:09activations throughout the neural nut
49:11are roughly gsen and so that's going to
49:14become very quickly intractable but
49:16compared to that it's going to be much
49:17much easier to sprinkle batch
49:19normalization layers throughout the
49:20neural net so in particular it's common
49:24to to look at every single linear layer
49:26like this one one this is a linear layer
49:27multiplying by a weight Matrix and
49:29adding a bias or for example
49:31convolutions which we'll cover later and
49:33also perform basically a multiplication
49:36with a weight Matrix but in a more
49:37spatially structured format it's custom
49:40it's customary to take these linear
49:42layer or convolutional layer and append
49:45a b rization layer right after it to
49:47control the scale of these activations
49:49at every point in the neural nut so we'd
49:52be adding these bom layers throughout
49:53the neural nut and then this controls
49:55the scale of these AC ations throughout
49:57the neural net it doesn't require us to
49:59do uh perfect mathematics and care about
50:02the activation distributions uh for all
50:04these different types of neural network
50:06uh Lego building blocks that you might
50:07want to introduce into your neural net
50:09and it significantly stabilizes uh the
50:11training and that's why these uh layers
50:13are quite popular now the stability
50:15offered by bash normalization actually
50:17comes at a terrible cost and that cost
50:19is that if you think about what's
50:21Happening Here something something
50:23terribly strange and unnatural is
50:25happening it used to be that we have a
50:28single example feeding into a neural nut
50:30and then uh we calculate its activations
50:32and its loits and this is a
50:35deterministic sort of process so you
50:37arrive at some logits for this example
50:40and then because of efficiency of
50:41training we suddenly started to use
50:43batches of examples but those batches of
50:45examples were processed independently
50:47and it was just an efficiency thing but
50:50now suddenly in batch normalization
50:51because of the normalization through the
50:53batch we are coupling these examples
50:55mathematically and in the forward pass
50:57and the backward pass of a neural l so
50:59now the hidden State activations H pract
51:02in your log jits for any one input
51:04example are not just a function of that
51:06example and its input but they're also a
51:09function of all the other examples that
51:11happen to come for a ride in that batch
51:14and these examples are sampled randomly
51:16and so what's happening is for example
51:17when you look at H pract that's going to
51:19feed into H the hidden State activations
51:22for for example for for any one of these
51:24input examples is going to actually
51:26change slightly depending on what other
51:28examples there are in a batch and and
51:31depending on what other examples happen
51:32to come for a ride H is going to change
51:35subtly and it's going to like Jitter if
51:37you imagine sampling different examples
51:39because the the statistics of the mean
51:41and the standard deviation are going to
51:42be impacted and so you'll get a Jitter
51:44for H and you'll get a Jitter for
51:47loits and you think that this would be a
51:49bug uh or something undesirable but in a
51:53very strange way this actually turns out
51:55to be good in your Network training and
51:58as a side effect and the reason for that
52:00is that you can think of this as kind of
52:02like a regularizer because what's
52:04happening is you have your input and you
52:05get your age and then depending on the
52:07other examples this is jittering a bit
52:10and so what that does is that it's
52:11effectively padding out any one of these
52:13input examples and it's introducing a
52:15little bit of entropy and um because of
52:18the padding out it's actually kind of
52:19like a form of a data augmentation which
52:21we'll cover in the future and it's kind
52:23of like augmenting the input a little
52:25bit and jittering it and that makes it
52:27harder for the neural nut to overfit to
52:29these concrete specific examples so by
52:32introducing all this noise it actually
52:34like Pats out the examples and it
52:36regularizes the neural nut and that's
52:38one of the reasons why uh deceivingly as
52:40a second order effect uh this is
52:42actually a regularizer and that has made
52:44it harder uh for us to remove the use of
52:46batch
52:47normalization uh because basically no
52:49one likes this property that the the
52:51examples in the batch are coupled
52:53mathematically and in the forward pass
52:55and at least all kinds of like strange
52:57uh results uh we'll go into some of that
52:59in a second as well um and it leads to a
53:02lot of bugs and um and so on and so no
53:05one likes this property uh and so people
53:07have tried to um deprecate the use of
53:10bat normalization and move to other
53:11normalization techniques that do not
53:13couple the examples of a batch examples
53:15are ler normalization instance
53:17normalization group normalization and so
53:19on and we'll come we'll come some these
53:21uh later um but basically long story
53:25short bat normalization was the first
53:27kind of normalization layer to be
53:28introduced it worked extremely well it
53:31happened to have this regularizing
53:32effect it stabilized training and people
53:36have been trying to remove it and move
53:38to some of the other normalization
53:39techniques uh but it's been hard because
53:42it it just works quite well and some of
53:44the reason that it works quite well is
53:46again because of this regular rizing
53:47effect and because of the because it is
53:49quite effective at um controlling the
53:52activations and their
53:53distributions uh so that's kind of like
53:55the brief story of Bash normalization
53:57and I'd like to show you one of the
53:59other weird sort of outcomes of this
54:02coupling so here's one of the strange
54:04outcomes that I only glossed over
54:06previously when I was evaluating the
54:08loss on the validation set basically
54:11once we've trained a neural net we'd
54:13like to deploy it in some kind of a
54:15setting and we'd like to be able to feed
54:16in a single individual example and get a
54:19prediction out from our neural net but
54:21how do we do that when our neural net
54:23now in a forward pass estimates the
54:25statistics of the mean understand
54:26deviation of a batch the neur lot
54:28expects batches as an input now so how
54:30do we feed in a single example and get
54:32sensible results out and so the proposal
54:35in the batch normalization paper is the
54:37following what we would like to do here
54:40is we would like to basically have a
54:42step after training that uh calculates
54:46and sets the bach uh mean and standard
54:49deviation a single time over the
54:50training set and so I wrote this code
54:53here in interest of time and we're going
54:55to call what's called calibrate the
54:57bachor statistics and basically what we
54:59do is torch torch. nograd telling
55:02pytorch that none of this we will call
55:05Dot backward on and it's going to be a
55:07bit more efficient we're going to take
55:09the training set get the pre activations
55:11for every single training example and
55:13then one single time estimate the mean
55:15and standard deviation over the entire
55:16training set and then we're going to get
55:18B and mean and B and standard deviation
55:20and now these are fixed numbers
55:22estimating over the entire training set
55:25and here instead of estimating it
55:28dynamically we are going to instead here
55:31use B and
55:33mean and here we're just going to use B
55:35and standard
55:37deviation and so at test time we are
55:39going to fix these clamp them and use
55:41them during inference and
55:44now you see that we get basically
55:46identical result uh but the benefit that
55:49we've gained is that we can now also
55:51forward a single example because the
55:53mean and standard deviation are now
55:54fixed uh sort of tensor
55:57that said nobody actually wants to
55:58estimate this mean and standard
56:00deviation as a second stage after neural
56:03network training because everyone is
56:04lazy and so this batch normalization
56:07paper actually introduced one more idea
56:09which is that we are can we can estimate
56:11the mean and standard deviation in a
56:12running man running manner during
56:15training of the neuron nut and then we
56:17can uh simply just have a single stage
56:19of training and on the side of that
56:21training we are estimating the running
56:23mean and standard deviation so let's see
56:24what that would look like let me
56:26basically take the mean here that we are
56:29estimating on the batch and let me call
56:30this B and mean on the I
56:33iteration um and then here this is BN
56:39sdd um bnsd at I
56:45okay uh and the mean comes here and the
56:50STD comes here so so far I've done
56:53nothing I've just uh moved around and I
56:55created these EXT extra variables for
56:56the mean and standard deviation and I've
56:58put them here so so far nothing has
57:00changed but what we're going to do now
57:02is we're going to keep running mean of
57:04both of these values during training so
57:06let me swing up here and let me create a
57:08BN meanor running and I'm going to
57:12initialize it at uh
57:15zeros and then BN STD running which I'll
57:19initialize at
57:22once because um in the beginning because
57:25of the way we ized W1 uh and B1 H pract
57:29will be roughly unit gion so the mean
57:31will be roughly zero and a standard
57:33deviation roughly one so I'm going to
57:35initialize these that way but then here
57:37I'm going to update these and in pytorch
57:40um uh these uh mean and standard
57:43deviation that are running uh they're
57:45not actually part of the gradient based
57:46optimization we're never going to derive
57:48gradients with respect to them they're
57:50they're updated on the side of training
57:53and so what we're going to do here is
57:54we're going to say with torch. nograd
57:58telling pytorch that the update here is
58:01not supposed to be building out a graph
58:02because there will be no dot
58:04backward but this running is basically
58:07going to be
58:090.99 uh9 times the current
58:12Value Plus 0.001 times the um this value
58:18this new
58:19mean and in the same way bnsd running
58:23will be mostly what it used to be
58:28but it will receive a small update in
58:30the direction of what the current
58:32standard deviation
58:34is and as you're seeing here this update
58:36is outside and on the side of the
58:39gradient based optimization and it's
58:41simply being updated not using gradient
58:43scent it's just being updated using U
58:45janky like Smooth um sort of uh running
58:50mean
58:52Manner and so while the network is
58:54training and these pre activations are
58:57sort of changing and shifting around
58:58during during back propagation we are
59:00keeping track of the typical mean and
59:02standard deviation and we're estimating
59:04them once and when I run
59:08this now I'm keeping track of this in
59:10the running Manner and what we're hoping
59:12for of course is that the me BN meore
59:14running and BN meore STD are going to be
59:18very similar to the ones that we
59:20calculated here before and that way we
59:23don't need a second stage because we've
59:25sort of combined the two stages and
59:26we've put them on the side of each other
59:28if you want to look at it that way and
59:31this is how this is also implemented in
59:32The Bash normalization uh layer impi
59:34torch so during training um the exact
59:37same thing will happen and then later
59:39when you're using inference it will use
59:41the estimated running mean of both the
59:44mean and standard deviation of those
59:46hidden States so let's wait for the
59:48optimization to converge and hopefully
59:50the running mean and standard deviation
59:52are roughly equal to these two and then
59:54we can simply use it here and we don't
59:56need this stage of explicit calibration
59:58at the end okay so the optimization
1:00:00finished I'll rerun the explicit
1:00:02estimation and then the B and mean from
1:00:05the explicit estimation is here and B
1:00:08and mean from the running estimation
1:00:11during the during the optimization you
1:00:13can see is very very similar it's not
1:00:16identical but it's pretty
1:00:18close and the same way BN STD is this
1:00:22and BN STD running is this and so you
1:00:26can see that once again they are fairly
1:00:28similar values not identical but pretty
1:00:30close and so then here instead of being
1:00:33mean we can use the BN mean running
1:00:35instead of bnsd we can use bnsd
1:00:38running and uh hopefully the validation
1:00:41loss will not be impacted too
1:00:43much okay so it's basically identical
1:00:46and this way we've eliminated the need
1:00:49for this explicit stage of calibration
1:00:51because we are doing it in line over
1:00:53here okay so we're almost done with
1:00:55batch normalization there are only two
1:00:56more notes that I'd like to make number
1:00:58one I've skipped a discussion over what
1:01:00is this plus Epsilon doing here this
1:01:02Epsilon is usually like some small fixed
1:01:04number for example one5 by default and
1:01:07what it's doing is that it's basically
1:01:08preventing a division by zero in the
1:01:11case that the variance over your batch
1:01:14is exactly zero in that case uh here we
1:01:17normally have a division by zero but
1:01:19because of the plus Epsilon uh this is
1:01:21going to become a small number in the
1:01:22denominator instead and things will be
1:01:24more well behaved so feel free to also
1:01:26add a plus Epsilon here of a very small
1:01:28number it doesn't actually substantially
1:01:30change the result I'm going to skip it
1:01:31in our case just because uh this is
1:01:33unlikely to happen in our very simple
1:01:34example here and the second thing I want
1:01:37you to notice is that we're being
1:01:38wasteful here and it's very subtle but
1:01:41right here where we are adding the bias
1:01:43into H preact these biases now are
1:01:46actually useless because we're adding
1:01:48them to the H preact but then we are
1:01:50calculating the mean for every one of
1:01:53these neurons and subtracting it so
1:01:56whatever bias you add here is going to
1:01:58get subtracted right here and so these
1:02:01biases are not doing anything in fact
1:02:03they're being subtracted out and they
1:02:04don't impact the rest of the calculation
1:02:07so if you look at b1. grad it's actually
1:02:09going to be zero because it's being
1:02:10subtracted out and doesn't actually have
1:02:12any effect and so whenever you're using
1:02:14bash normalization layers then if you
1:02:16have any weight layers before like a
1:02:18linear or a c or something like that
1:02:20you're better off coming here and just
1:02:22like not using bias so you don't want to
1:02:25use bias and then here you don't want to
1:02:28add it because it's that spirous instead
1:02:31we have this B normalization bias here
1:02:33and that b rization bias is now in
1:02:35charge of the biasing of this
1:02:37distribution instead of this B1 that we
1:02:40had here originally and so uh basically
1:02:43bash normalization layer has its own
1:02:45bias and there's no need to have a bias
1:02:48in the layer before it because that bias
1:02:49is going to be subtracted out anyway so
1:02:52that's the other small detail to be
1:02:53careful with sometimes it's not going to
1:02:55do anything catastrophic this B1 will
1:02:57just be useless it will never get any
1:02:59gradient uh it will not learn it will
1:03:01stay constant and it's just wasteful but
1:03:03it doesn't actually really uh impact
1:03:05anything otherwise okay so I rearranged
1:03:07the code a little bit with comments and
1:03:09I just wanted to give a very quick
1:03:11summary of The Bash normalization layer
1:03:13we are using bash normalization to
1:03:15control the statistics of activations in
1:03:18the neural net it is common to sprinkle
1:03:20bash normalization layer across the
1:03:22neural net and usually we will place it
1:03:24after layer that have multiplications
1:03:27like for example a linear layer or
1:03:29convolutional layer which we may cover
1:03:31in the
1:03:32future now the bat normalization
1:03:34internally has parameters for the gain
1:03:38and the bias and these are trained using
1:03:40back propagation it also has two buffers
1:03:44the buffers are the mean and the
1:03:45standard deviation the running mean and
1:03:47the running mean of the standard
1:03:49deviation and these are not trained
1:03:51using back propagation these are trained
1:03:53using this uh janky update of kind of
1:03:56like a running mean
1:03:58update so
1:04:00um these are sort of the parameters and
1:04:03the buffers of Bator layer and then
1:04:05really what it's doing is it's
1:04:06calculating the mean and a standard
1:04:08deviation of the activations uh that are
1:04:10feeding into the Bator layer over that
1:04:14batch then it's centering that batch to
1:04:16be unit gion and then it's offsetting
1:04:19and scaling it by the Learned bias and
1:04:23gain and then on top of that it's
1:04:25keeping track of the mean and standard
1:04:26deviation of the inputs and it's
1:04:30maintaining this running mean and
1:04:31standard deviation and this will later
1:04:33be used at inference so that we don't
1:04:35have to reestimate the mean stand
1:04:37deviation all the time and in addition
1:04:40that allows us to basically forward
1:04:41individual examples at test time so
1:04:44that's the bash normalization layer it's
1:04:45a fairly complicated layer um but this
1:04:48is what it's doing internally now I
1:04:50wanted to show you a little bit of a
1:04:51real example so you can search resnet
1:04:55which is a residual neural network and
1:04:58these are common types of neural
1:04:59networks used for image
1:05:01classification and of course we haven't
1:05:03come resnets in detail so I'm not going
1:05:05to explain all the pieces of it but for
1:05:08now just note that the image feeds into
1:05:10a reset on the top here and there's many
1:05:12many layers with repeating structure all
1:05:15the way to predictions of what's inside
1:05:16that image this repeating structure is
1:05:19made up of these blocks and these blocks
1:05:21are just sequentially stacked up in this
1:05:23deep neural network now the code for
1:05:26this uh the block basically that's used
1:05:29and repeated sequentially in series is
1:05:32called this bottleneck block bottleneck
1:05:35block and there's a lot here this is all
1:05:37pych and of course we haven't covered
1:05:39all of it but I want to point out some
1:05:41small pieces of it here in the init is
1:05:44where we initialize the neuronet so this
1:05:46code of block here is basically the kind
1:05:47of stuff we're doing here we're
1:05:49initializing all the layers and in the
1:05:51forward we are specifying how the neuron
1:05:53lot acts once you actually have the
1:05:55input so this code here is along the
1:05:57lines of what we're doing
1:06:00here and now these blocks are replicated
1:06:04and stacked up serially and that's what
1:06:06a residual Network would be and so
1:06:09notice What's Happening Here com one um
1:06:12these are convolution layers and these
1:06:15convolution layers basically they're the
1:06:17same thing as a linear layer except
1:06:19convolutional layers don't apply um
1:06:22convolutional layers are used for images
1:06:24and so they have SP structure and
1:06:26basically this linear multiplication and
1:06:28bias offset are done on patches instead
1:06:32of math instead of the full input so
1:06:34because these images have structure
1:06:36spatial structure convolutions just
1:06:38basically do WX plus b but they do it on
1:06:41overlapping patches of the input but
1:06:44otherwise it's WX plus
1:06:45P then we have the norm layer which by
1:06:48default here is initialized to be a bash
1:06:50Norm in 2D so two- dimensional bash
1:06:52normalization layer and then we have a
1:06:54nonlinearity like reu so instead of uh
1:06:58here they use reu we are using 10 in
1:07:01this case but both both are just
1:07:03nonlinearities and you can just use them
1:07:05relatively interchangeably for very deep
1:07:07networks re typically empirically work a
1:07:10bit better so see the motif that's being
1:07:13repeated here we have convolution bat
1:07:15normalization reu convolution bat
1:07:17normalization re Etc and then here this
1:07:19is residual connection that we haven't
1:07:21covered yet but basically that's the
1:07:23exact same pattern we have here with we
1:07:25have a weight layer like a convolution
1:07:28or like a linear layer bash
1:07:31normalization and then 10h which is
1:07:34nonlinearity but basically a weight
1:07:36layer a normalization layer and
1:07:38nonlinearity and that's the motif that
1:07:40you would be stacking up when you create
1:07:42these deep neural networks exactly as
1:07:44it's done here and one more thing I'd
1:07:46like you to notice is that here when
1:07:47they are initializing the com layers
1:07:50like com 1 by one the depth for that is
1:07:53right here and so it's initializing an
1:07:55nn. Tod which is a convolution layer in
1:07:58pytorch and there's a bunch of keyword
1:07:59arguments here that I'm not going to
1:08:01explain yet but you see how there's bias
1:08:03equals false the bias equals false is
1:08:05exactly for the same reason as bias is
1:08:08not used in our case you see how I eras
1:08:10the use of bias and the use of bias is
1:08:13spous because after this weight layer
1:08:15there's a bash normalization and The
1:08:16Bash normalization subtracts that bias
1:08:19and then has its own bias so there's no
1:08:21need to introduce these spous parameters
1:08:23it wouldn't hurt performance it's just
1:08:24useless and so because they have this
1:08:27motif of C Bast umbrell they don't need
1:08:30a bias here because there's a bias
1:08:31inside here so by the way this example
1:08:35here is very easy to find just do
1:08:37resonet pie
1:08:38torch and uh it's this example here so
1:08:41this is kind of like the stock
1:08:42implementation of a residual neural
1:08:44network in pytorch and uh you can find
1:08:47that here but of course I haven't
1:08:48covered many of these parts yet and I
1:08:50would also like to briefly descend into
1:08:52the definitions of these pytorch layers
1:08:55and the the parameters that they take
1:08:57now instead of a convolutional layer
1:08:58we're going to look at a linear layer uh
1:09:01because that's the one that we're using
1:09:02here this is a linear layer and I
1:09:04haven't cover covered convolutions yet
1:09:06but as I mentioned convolutions are
1:09:07basically linear layers except on
1:09:10patches so a linear layer performs a WX
1:09:13plus b except here they're calling the W
1:09:16A
1:09:17transpose um so to calcul WX plus b very
1:09:20much like we did here to initialize this
1:09:22layer you need to know the fan in the
1:09:24fan out and that's so that they can
1:09:27initialize this W this is the fan in and
1:09:30the fan out so they know how how big the
1:09:33weight Matrix should be you need to also
1:09:36pass in whether you whether or not you
1:09:37want a bias and if you set it to false
1:09:40then no bias will be uh inside this
1:09:42layer um and you may want to do that
1:09:45exactly like in our case if your layer
1:09:47is followed by a normalization layer
1:09:49such as batch
1:09:50Norm so this allows you to basically
1:09:52disable a bias now in terms of the
1:09:55initial ation if we swing down here this
1:09:57is reporting the variables used inside
1:09:59this linear layer and our linear layer
1:10:02here has two parameters the weight and
1:10:04the bias in the same way they have a
1:10:06weight and a bias and they're talking
1:10:09about how they initialize it by default
1:10:11so by default P will initialize your
1:10:13weights by taking the
1:10:15Fanon and then um doing one over fanin
1:10:19square root and then instead of a normal
1:10:22distribution they are using a uniform
1:10:24distribution
1:10:25so it's very much the same thing but
1:10:28they are using a one instead of 5 over
1:10:30three so there's no gain being
1:10:31calculated here the gain is just one but
1:10:33otherwise is exactly one over the square
1:10:36root of fan in exactly as we have
1:10:39here so one over the square root of K is
1:10:42the is the scale of the weights but when
1:10:45they are drawing the numbers they're not
1:10:46using a gussion by default they're using
1:10:49a uniform distribution by default and so
1:10:51they draw uniformly from negative of K
1:10:54to squ of K
1:10:56but it's the exact same thing and the
1:10:57same motivation from for with respect to
1:11:00what we've seen in this lecture and the
1:11:03reason they're doing this is if you have
1:11:04a roughly gsan input this will ensure
1:11:08that out of this layer you will have a
1:11:10roughly Gan output and you you basically
1:11:13achieve that by scaling the weights by
1:11:15one over the square root of fan in so
1:11:18that's what this is
1:11:19doing and then the second thing is the
1:11:21bash normalization layer so let's look
1:11:23at what that looks like in pytorch
1:11:26so here we have a onedimensional b
1:11:27normalization layer exactly as we are
1:11:29using here and there are a number of
1:11:31keyword arguments going into it as well
1:11:33so we need to know the number of
1:11:34features uh for us that is 200 and that
1:11:37is needed so that we can initialize
1:11:39these parameters here the gain the bias
1:11:42and the buffers for the running uh mean
1:11:44and standard
1:11:46deviation then they need to know the
1:11:47value of Epsilon here and by default
1:11:50this is one5 you don't typically change
1:11:52this too much then they need to know the
1:11:54momentum
1:11:56and the momentum here as they explain is
1:11:58basically used for these uh running mean
1:12:01and running standard deviation so by
1:12:03default the momentum here is 0.1 the
1:12:05momentum we are using here in this
1:12:06example is
1:12:080.001 and basically rough you may want
1:12:12to change this sometimes and roughly
1:12:14speaking if you have a very large batch
1:12:16size then typically what you'll see is
1:12:18that when you estimate the mean and the
1:12:20standard deviation for every single
1:12:22batch size if it's large enough you're
1:12:23going to get roughly the same result
1:12:26and so therefore you can use slightly
1:12:28higher momentum like
1:12:300.1 but for a batch size as small as 32
1:12:34the mean and standard deviation here
1:12:36might take on slightly different numbers
1:12:37because there's only 32 examples we are
1:12:39using to estimate the mean and standard
1:12:41deviation so the value is changing
1:12:42around a lot and if your momentum is 0.1
1:12:46that that might not be good enough for
1:12:47this value to settle and um converge to
1:12:51the actual mean and standard deviation
1:12:53over the entire training set and so
1:12:55basically if your batch size is very
1:12:56small uh momentum of 0.1 is potentially
1:12:59dangerous and it might make it so that
1:13:00the running uh mean and stand deviation
1:13:02are is thrashing too much during
1:13:04training and it's not actually
1:13:06converging
1:13:08properly uh aine equals true determines
1:13:11whether this batch normalization layer
1:13:13has these learnable Aline parameters the
1:13:16uh the gain and the bias and this is
1:13:19almost always kept to true I'm not
1:13:21actually sure why you would want to
1:13:22change this to false um
1:13:26then track running stats is determining
1:13:28whether or not B rization layer of
1:13:30pytorch will be doing
1:13:31this and um one reason you may you may
1:13:35want to skip the running stats is
1:13:37because you may want to for example
1:13:39estimate them at the end as a stage two
1:13:42like this and in that case you don't
1:13:43want the bat normalization layer to be
1:13:45doing all this extra compute that you're
1:13:46not going to
1:13:47use and uh finally we need to know which
1:13:50device we're going to run this bash
1:13:52normalization on a CPU or a GPU and what
1:13:55the data type should be uh half
1:13:57Precision single Precision double
1:13:58precision and so
1:14:00on so that's the bat normalization layer
1:14:02otherwise they link to the paper is the
1:14:04same formula we've implemented and
1:14:06everything is the same exactly as we've
1:14:08done
1:14:09here okay so that's everything that I
1:14:11wanted to cover for this lecture really
1:14:14what I wanted to talk about is the
1:14:15importance of understanding the
1:14:16activations and the gradients and their
1:14:18statistics in neural networks and this
1:14:20becomes increasingly important
1:14:22especially as you make your neural
1:14:23networks bigger larger and deeper
1:14:25we looked at the distributions basically
1:14:27at the output layer and we saw that if
1:14:29you have two confident mispredictions
1:14:31because the activations are too messed
1:14:33up at the last layer you can end up with
1:14:35these hockey stick losses and if you fix
1:14:38this you get a better loss at the end of
1:14:39training because your training is not
1:14:41doing wasteful work then we also saw
1:14:44that we need to control the activations
1:14:45we don't want them to uh you know squash
1:14:48to zero or explode to infinity and
1:14:51because that you can run into a lot of
1:14:52trouble with all of these uh
1:14:53nonlinearities and these neural Nets and
1:14:56basically you want everything to be
1:14:57fairly homogeneous throughout the neural
1:14:58net you want roughly goshan activations
1:15:00throughout the neural net let me talked
1:15:03about okay if we want roughly Gan
1:15:05activations how do we scale these weight
1:15:08matrices and biases during
1:15:09initialization of the neural nut so that
1:15:11we don't get um you know so everything
1:15:13is as controlled as
1:15:15possible um so that give us a large
1:15:18boost in Improvement and then I talked
1:15:20about how that strategy is not actually
1:15:24uh Poss for much much deeper neural nuts
1:15:27because um when you have much deeper
1:15:29neural nuts with lots of different types
1:15:31of layers it becomes really really hard
1:15:33to precisely set the weights and the
1:15:35biases in such a way that the
1:15:37activations are roughly uniform
1:15:39throughout the neural nut so then I
1:15:41introduced the notion of a normalization
1:15:43layer now there are many normalization
1:15:45layers that that people use in practice
1:15:47bat normalization layer normalization
1:15:50instance normalization group
1:15:51normalization we haven't covered most of
1:15:53them but I've introduced the first one
1:15:55and also the one that I believe came out
1:15:57first and that's called Bat
1:15:59normalization and we saw how bat
1:16:01normalization Works uh this is a layer
1:16:03that you can sprinkle throughout your
1:16:05deep neural net and the basic idea is if
1:16:08you want roughly gsh in activations well
1:16:10then take your activations and um take
1:16:13the mean and the standard deviation and
1:16:14Center your data and you can do that
1:16:17because the centering operation is
1:16:20differentiable but and on top of that we
1:16:22actually had to add a lot of bells and
1:16:24whistles and that gave you a sense of
1:16:26the complexities of the batch
1:16:27normalization layer because now we're
1:16:29centering the data that's great but
1:16:31suddenly we need the gain and the bias
1:16:33and now those are
1:16:34trainable and then because we are
1:16:36coupling all of the training examples
1:16:38now suddenly the question is how do you
1:16:39do the inference where to do to do the
1:16:42inference we need to now estimate these
1:16:45um mean and standard deviation once uh
1:16:48or the entire training set and then use
1:16:50those at inference but then no one likes
1:16:52to do stage two so instead we fold
1:16:54everything everything into the bat
1:16:56normalization later during training and
1:16:57try to estimate these in the running
1:16:59manner so that everything is a bit
1:17:01simpler and that gives us the bat
1:17:03normalization layer um and as I
1:17:07mentioned no one likes this layer it
1:17:09causes a huge amount of bugs um and
1:17:13intuitively it's because it is coupling
1:17:15examples um in the for pass of a neural
1:17:17nut and uh I've shot myself in the foot
1:17:21with this layer over and over again in
1:17:24my life and I don't want you to suffer
1:17:26the same uh so basically try to avoid it
1:17:29as much as possible uh some of the other
1:17:32alternatives to these layers are for
1:17:34example group normalization or layer
1:17:35normalization and those have become more
1:17:37common uh in more recent deep learning
1:17:40uh but we haven't covered those yet uh
1:17:43but definitely bash normalization was
1:17:44very influential at the time when it
1:17:46came out in roughly 2015 because it was
1:17:49kind of the first time that you could
1:17:50train reliably uh much deeper neural
1:17:54nuts and fundamentally the reason for
1:17:56that is because this layer was very
1:17:58effective at controlling the statistics
1:18:00of the activations in the neural nut so
1:18:03that's the story so far and um that's
1:18:06all I wanted to cover and in the future
1:18:08lectures hopefully we can start going
1:18:09into recurrent R Nets and um recurring
1:18:13neural Nets as we'll see are just very
1:18:14very deep networks because you uh you
1:18:17unroll the loop and uh when you actually
1:18:19optimize these neurals and that's where
1:18:22a lot of this
1:18:23um analysis around the activation
1:18:26statistics and all these normalization
1:18:28layers will become very very important
1:18:30for uh good performance so we'll see
1:18:33that next time bye okay so I lied I
1:18:36would like us to do one more summary
1:18:37here as a bonus and I think it's useful
1:18:40as to have one more summary of
1:18:42everything I've presented in this
1:18:43lecture but also I would like us to
1:18:44start by torify our code a little bit so
1:18:47it looks much more like what you would
1:18:48encounter in PCH so you'll see that I
1:18:50will structure our code into these
1:18:53modules like a link
1:18:55uh module and a borm module and I'm
1:18:59putting the code inside these modules so
1:19:01that we can construct neural networks
1:19:02very much like we would construct them
1:19:04in pytorch and I will go through this in
1:19:05detail so we'll create our neural net
1:19:08then we will do the optimization loop as
1:19:11we did before and then the one more
1:19:13thing that I want to do here is I want
1:19:14to look at the activation statistics
1:19:16both in the forward pass and in the
1:19:18backward pass and then here we have the
1:19:20evaluation and sampling just like before
1:19:22so let me rewind all the way up here and
1:19:24and go a little bit slower so here I
1:19:27creating a linear layer you'll notice
1:19:29that torch.nn has lots of different
1:19:31types of layers and one of those layers
1:19:33is the linear layer torch. n. linear
1:19:36takes a number of input features output
1:19:37features whether or not we should have a
1:19:39bias and then the device that we want to
1:19:41place this layer on and the data type so
1:19:44I will emit these two but otherwise we
1:19:46have the exact same thing we have the
1:19:48fan in which is the number of inputs fan
1:19:50out the number of outputs and whether or
1:19:53not we want to use a bias
1:19:55and internally inside this layer there's
1:19:57a weight and a bias if you'd like it it
1:20:00is typical to initialize the weight
1:20:02using um say random numbers drawn from
1:20:05aashan and then here's the coming
1:20:07initialization um that we discussed
1:20:09already in this lecture and that's a
1:20:11good default and also the default that I
1:20:13believe pytor chooses and by default the
1:20:15bias is usually initialized to zeros now
1:20:18when you call this module uh this will
1:20:21basically calculate W * X plus b if you
1:20:23have a b and then when you also call
1:20:25that parameters on this module it will
1:20:27return uh the tensors that are the
1:20:30parameters of this layer now next we
1:20:32have the bash normalization layer so
1:20:35I've written that here and this is very
1:20:38similar to pytorch nn. bashor 1D layer
1:20:42as shown
1:20:43here so I'm kind of um taking these
1:20:46three parameters here the dimensionality
1:20:49the Epsilon that we will use in the
1:20:50division and the momentum that we will
1:20:52use in keeping track of these running
1:20:54stats the running mean and the running
1:20:56variance um now py actually takes quite
1:20:59a few more things but I'm assuming some
1:21:01of their settings so for us Aline will
1:21:03be true that means that we will be using
1:21:05a gamma and beta after the normalization
1:21:08the track running stats will be true so
1:21:09we will be keeping track of the running
1:21:11mean and the running variance in the in
1:21:13the bat Norm our device by default is
1:21:15the CPU and the data type by default is
1:21:18uh float float
1:21:2132 so those are the defaults otherwise
1:21:24uh we are taking all the same parameters
1:21:26in this bachom layer so first I'm just
1:21:28saving them now here's something new
1:21:31there's a doc training which by default
1:21:32is true and pytorch andn modules also
1:21:35have this attribute. training and that's
1:21:37because many modules in borm is included
1:21:40in that have a different Behavior
1:21:43whether you are training your interet
1:21:44and or whether you are running it in an
1:21:46evaluation mode and calculating your
1:21:48evaluation loss or using it for
1:21:50inference on some test examples and
1:21:53bashor is an example of this because
1:21:54when we are training we are going to be
1:21:56using the mean and the variance
1:21:57estimated from the current batch but
1:22:00during inference we are using the
1:22:01running mean and running variance and so
1:22:04also if we are training we are updating
1:22:06mean and variance but if we are testing
1:22:08then these are not being updated they're
1:22:10kept fixed and so this flag is necessary
1:22:13and by default true just like in
1:22:15pytorch now the parameters of B 1D are
1:22:18the gamma and the beta
1:22:20here and then the running mean and
1:22:22running variance are called buffers in
1:22:25pyto
1:22:26nomenclature and these buffers are
1:22:29trained using exponential moving average
1:22:32here explicitly and they are not part of
1:22:34the back propagation and stochastic
1:22:36radient descent so they are not sort of
1:22:37like parameters of this layer and that's
1:22:40why when we C when we have a parameters
1:22:42here we only return gamma and beta we do
1:22:44not return the mean and the variance
1:22:46this is trained sort of like internally
1:22:48here um every forward pass using
1:22:51exponential moving average so that's the
1:22:55initialization now in a forward pass if
1:22:58we are training then we use the mean and
1:23:00the variance estimated by the batch let
1:23:03me pull up the paper
1:23:04here we calculate the mean and the
1:23:07variance now up above I was estimating
1:23:10the standard deviation and keeping track
1:23:12of the standard deviation here in the
1:23:15running standard deviation instead of
1:23:16running variance but let's follow the
1:23:18paper exactly here they calculate the
1:23:21variance which is the standard deviation
1:23:23squared and that's what's get track of
1:23:25in a running variance instead of a
1:23:27running standard
1:23:28deviation uh but those two would be very
1:23:31very similar I
1:23:32believe um if we are not training then
1:23:34we use running mean and variance we
1:23:38normalize and then here I am calculating
1:23:40the output of this layer and I'm also
1:23:42assigning it to an attribute called out
1:23:45now out is something that I'm using in
1:23:48our modules here uh this is not what you
1:23:50would find in pytorch we are slightly
1:23:51deviating from it I'm creating a DOT out
1:23:54because I would like to very easily um
1:23:57maintain all those variables so that we
1:23:59can create statistics of them and plot
1:24:00them but pytorch and modules will not
1:24:03have a do out attribute and finally here
1:24:05we are updating the buffers using again
1:24:08as I mentioned exponential moving
1:24:09average uh provide given the provided
1:24:11momentum and importantly you'll notice
1:24:14that I'm using the torch. nogra context
1:24:16manager and I doing this because if we
1:24:18don't use this then pytorch will start
1:24:20building out an entire computational
1:24:22graph out of these tensors because it is
1:24:25expecting that we will eventually call
1:24:26Dot backward but we are never going to
1:24:28be calling dot backward on anything that
1:24:30includes running mean and running
1:24:31variance so that's why we need to use
1:24:33this context manager so that we are not
1:24:36um sort of maintaining them using all
1:24:38this additional memory um so this will
1:24:40make it more efficient and it's just
1:24:42telling pyour that there will no
1:24:43backward we just have a bunch of tensors
1:24:45we want to update them that's it and
1:24:48then we
1:24:49return okay now scrolling down we have
1:24:51the 10h layer this is very very similar
1:24:53to uh torch. 10h and it doesn't do too
1:24:57much it just calculates 10 as you might
1:24:59expect so uh that's torch. 10h and uh
1:25:03there's no parameters in this layer but
1:25:05because these are layers um it now
1:25:07becomes very easy to sort of like stack
1:25:09them up into uh basically just a list um
1:25:13and uh we can do all the initializations
1:25:15that we're used to so we have the
1:25:17initial sort of embedding Matrix we have
1:25:19our layers and we can call them
1:25:21sequentially and then again with Tor no
1:25:24grb but there's some initializations
1:25:25here so we want to make the output
1:25:27softmax a bit less confident like we saw
1:25:30and in addition to that because we are
1:25:31using a six layer multi-layer percep on
1:25:34here so you see how I'm stacking linear
1:25:3610age linear Tage Etc uh I'm going to be
1:25:39using the gain here and I'm going to
1:25:41play with this in a second so you'll see
1:25:43how uh when we change this what happens
1:25:45to the
1:25:46statistics finally the parameters are
1:25:48basically the embedding Matrix and all
1:25:50the parameters in all the layers and
1:25:52notice here I'm using a double list
1:25:54apprehension if you want to call it that
1:25:56but for every layer in layers and for
1:25:58every parameter in each of those layers
1:26:00we are just stacking up all those piece
1:26:03uh all those parameters now in total we
1:26:05have 46,000 um
1:26:08parameters and I'm telling P that all of
1:26:10them require
1:26:15gradient then here uh we have everything
1:26:18here we are actually mostly used to uh
1:26:20we are sampling a batch we are doing a
1:26:22forward pass the forward pass now is
1:26:24just the linear application of all the
1:26:25layers in order followed by the cross
1:26:28entropy and then in the backward pass
1:26:30you'll notice that for every single
1:26:31layer I now iterate over all the outputs
1:26:34and I'm telling pytorch to retain the
1:26:35gradient of them and then here we are
1:26:38already used to uh all the all the
1:26:40gradient set To None do the backward to
1:26:42fill in the gradients uh do an update
1:26:44using stochastic gradient sent and then
1:26:46uh track some statistics and then I am
1:26:49going to break after a single iteration
1:26:52now here in this cell in this diagram I
1:26:54I'm visualizing the histogram the
1:26:56histograms of the for pass activations
1:26:58and I'm specifically doing it at the 10
1:27:00each layers so iterating over all the
1:27:03layers except for the very last one
1:27:05which is basically just the U soft Max
1:27:08layer um if it is a 10h layer and I'm
1:27:11using a 10h layer just because they have
1:27:13a finite output netive 1 to 1 and so
1:27:15it's very easy to visualize here so you
1:27:17see 1 to one and it's a finite range and
1:27:20easy to work with I take the out tensor
1:27:23from that layer into T and then I'm
1:27:25calculating the mean the standard
1:27:27deviation and the percent saturation of
1:27:29T and the way I Define the percent
1:27:31saturation is that t. absolute value is
1:27:33greater than 97 so that means we are
1:27:36here at the tals of the 10 H and
1:27:38remember that when we are in the tales
1:27:40of the 10 H that will actually stop
1:27:41gradients so we don't want this to be
1:27:43too
1:27:44high now here I'm calling torch.
1:27:48histogram and then I am plotting this
1:27:50histogram so basically what this is
1:27:51doing is that every different type of
1:27:53layer and they have a different color we
1:27:55are looking at how many um values in
1:27:58these tensors take on any of the values
1:28:01Below on this axis here so the first
1:28:04layer is fairly saturated uh here at 20%
1:28:07so you can see that it's got Tails here
1:28:10but then everything sort of stabilizes
1:28:12and if we had more layers here it would
1:28:14actually just stabilize at around the
1:28:15standard deviation of about 65 and the
1:28:18saturation would be roughly 5% and the
1:28:21reason that the stabilizes and gives us
1:28:22a nice distribution here is because gain
1:28:25is set to 5
1:28:26over3 now here this gain you see that by
1:28:31default we initialize with 1 /un of fan
1:28:34in but then here during initialization I
1:28:37come in and I erator all the layers and
1:28:38if it's a linear layer I boost that by
1:28:40the gain now we saw that one so
1:28:44basically if we just do not use a gain
1:28:47then what happens if I redraw this you
1:28:50will see that the standard deviation is
1:28:53shrinking and the saturation is coming
1:28:55to zero and basically what's happening
1:28:58is the first layer is you know pretty
1:29:00decent but then further layers are just
1:29:02kind of like shrinking down to zero and
1:29:05it's happening slowly but it's shrinking
1:29:06to zero and the reason for that is when
1:29:09you just have a sandwich of linear
1:29:11layers alone then a then initializing
1:29:15our weights in this manner we saw
1:29:18previously would have conserved the
1:29:20standard deviation of one but because we
1:29:22have this interspersed 10 in layers in
1:29:25there these 10h layers are squashing
1:29:28functions and so they take your
1:29:30distribution and they slightly squash it
1:29:32and so some gain is necessary to keep
1:29:35expanding it to fight the
1:29:39squashing so it just turns out that 5
1:29:41over3 is a good value so if we have
1:29:44something too small like one we saw that
1:29:46things will come toward zero but if it's
1:29:49something too high let's do
1:29:51two then here we see that um
1:29:56well let me do something a bit more
1:29:58extreme because so it's a bit more
1:30:00visible let's try
1:30:01three okay so we see here that the
1:30:03saturations are going to be way too
1:30:05large okay so three would create way too
1:30:08saturated activations so 5 over3 is a
1:30:12good setting for a sandwich of linear
1:30:15layers with 10h activations and it
1:30:18roughly stabilizes the standard
1:30:19deviation at a reasonable point now
1:30:23honestly I have no idea where 5 over3
1:30:25came from in pytorch um when we were
1:30:27looking at the coming initialization um
1:30:30I see empirically that it stabilizes
1:30:32this sandwich of linear an 10age and
1:30:34that the saturation is in a good range
1:30:36um but I don't actually know if this
1:30:37came out of some math formula I tried
1:30:39searching briefly for where this comes
1:30:41from uh but I wasn't able to find
1:30:43anything uh but certainly we see that
1:30:45empirically these are very nice ranges
1:30:47our saturation is roughly 5% which is a
1:30:49pretty good number and uh this is a good
1:30:52setting of The gain in this context
1:30:55similarly we can do the exact same thing
1:30:57with the gradients so here is a very
1:30:59same Loop if it's a 10h but instead of
1:31:01taking a layer do out I'm taking the
1:31:03grad and then I'm also showing the mean
1:31:05and the standard deviation and I'm
1:31:07plotting the histogram of these values
1:31:09and so you'll see that the gradient
1:31:11distribution is uh fairly reasonable and
1:31:13in particular what we're looking for is
1:31:15that all the different layers in this
1:31:16sandwich has roughly the same gradient
1:31:19things are not shrinking or exploding so
1:31:22uh we can for example come here and we
1:31:24can take a look at what happens if this
1:31:25gain was way too small so this was
1:31:290.5 then you see the first of all the
1:31:32activations are shrinking to zero but
1:31:34also the gradients are doing something
1:31:35weird the gradients started out here and
1:31:38then now they're like expanding
1:31:40out and similarly if we for example have
1:31:43a too high of a gain so like
1:31:45three then we see that also the
1:31:47gradients have there's some asymmetry
1:31:49going on where as you go into deeper and
1:31:51deeper layers the activation CS are
1:31:53changing and so that's not what we want
1:31:55and in this case we saw that without the
1:31:57use of batro as we are going through
1:31:59right now we had to very carefully set
1:32:02those gains to get nice activations in
1:32:04both the forward pass and the backward
1:32:07pass now before we move on to bat
1:32:09normalization I would also like to take
1:32:11a look at what happens when we have no
1:32:1210h units here so erasing all the 10
1:32:15nonlinearities but keeping the gain at 5
1:32:18over3 we now have just a giant linear
1:32:21sandwich so let's see what happens to
1:32:22the activations
1:32:24as we saw before the correct gain here
1:32:26is one that is the standard deviation
1:32:28preserving gain so 1.66 7 is too high
1:32:33and so what's going to happen now is the
1:32:36following uh I have to change this to be
1:32:38linear so we are because there's no more
1:32:4010h layers and let me change this to
1:32:43linear as
1:32:45well so what we're seeing is um the
1:32:48activations started out on the blue and
1:32:51have by layer four become very diffuse
1:32:55so what's happening to the activations
1:32:56is this and with the gradients on the
1:32:59top layer the activation the gradient
1:33:02statistics are the purple and then they
1:33:05diminish as you go down deeper in the
1:33:06layers and so basically you have an
1:33:08asymmetry like in the neuron net and you
1:33:11might imagine that if you have very deep
1:33:12neural networks say like 50 layers or
1:33:14something like that this just uh this is
1:33:16not a good place to be uh so that's why
1:33:19before bash normalization this was
1:33:21incredibly tricky to to set in
1:33:24particular if this is too large of a
1:33:26gain this happens and if it's too little
1:33:27of a
1:33:28gain then this happens so the opposite
1:33:32of that basically happens here we have a
1:33:34um shrinking and a uh diffusion
1:33:39depending on which direction you look at
1:33:40it from and so certainly this is not
1:33:43what you want and in this case the
1:33:44correct setting of The gain is exactly
1:33:47one just like we're doing at
1:33:49initialization and then we see that the
1:33:52uh statistics for the forward and a
1:33:54backward pass are well behaved and so
1:33:57the reason I want to show you this is
1:33:59that basically like getting neural nness
1:34:02to train before these normalization
1:34:03layers and before the use of advanced
1:34:05optimizers like adom which we still have
1:34:07to cover and residual connections and so
1:34:09on uh training neurs basically looked
1:34:12like this it's like a total Balancing
1:34:14Act you have to make sure that
1:34:15everything is precisely orchestrated and
1:34:18you have to care about the activations
1:34:19and the gradients and their statistics
1:34:21and then maybe you can train something
1:34:23uh but it was it was basically
1:34:24impossible to train very deep networks
1:34:25and this is fundamentally the the reason
1:34:27for that you'd have to be very very
1:34:29careful with your
1:34:30initialization um the other point here
1:34:33is you might be asking yourself by the
1:34:35way I'm not sure if I covered this why
1:34:37do we need these 10h layers at all uh
1:34:40why do we include them and then have to
1:34:42worry about the gain and uh the reason
1:34:44for that of course is that if you just
1:34:45have a stack of linear layers then
1:34:48certainly we're getting very easily nice
1:34:50activations and so on uh but this is
1:34:53just massive linear sandwich and it
1:34:54turns out that it collapses to a single
1:34:56linear layer in terms of its uh
1:34:58representation power so if you were to
1:35:00plot the output as a function of the
1:35:02input you're just getting a linear
1:35:03function no matter how many linear
1:35:05layers you stack up you still just end
1:35:07up with a linear transformation all the
1:35:09WX plus BS just collapse into a large WX
1:35:13plus b with slightly different W's and
1:35:15slightly different B um but
1:35:17interestingly even though the forward
1:35:19pass collapses to just a linear layer
1:35:21because of back propagation and uh the
1:35:23dynamics of the backward pass the
1:35:26optimization natur is not identical you
1:35:28actually end up with uh all kinds of
1:35:30interesting um Dynamics in the backward
1:35:33pass uh because of the uh the way the
1:35:36chain Ru is calculating it and so
1:35:38optimizing a linear layer by itself and
1:35:41optimizing a sandwich of 10 linear
1:35:43layers in both cases those are just a
1:35:44linear transformation in the forward
1:35:46pass but the training Dynamics would be
1:35:47different and there's entire papers that
1:35:49analyze in fact like infinitely layered
1:35:52uh linear layers and and so on and so
1:35:55there's a lot of things to that you can
1:35:56play with
1:35:57there uh but basically the tal
1:35:59linearities allow us to
1:36:02um turn this sandwich from just a
1:36:07linear uh function into uh a neural
1:36:10network that can in principle um
1:36:13approximate any arbitrary function okay
1:36:15so now I've reset the code to use the
1:36:17linear tanh sandwich like before and I
1:36:20reset everything so the gain is 5 over
1:36:23three uh we can run a single step of
1:36:25optimization and we can look at the
1:36:27activation statistics of the forward
1:36:28pass and the backward pass but I've
1:36:30added one more plot here that I think is
1:36:32really important to look at when you're
1:36:33training your neural nuts and to
1:36:35consider and ultimately what we're doing
1:36:37is we're updating the parameters of the
1:36:39neural nut so we care about the
1:36:40parameters and their values and their
1:36:43gradients so here what I'm doing is I'm
1:36:45actually iterating over all the
1:36:46parameters available and then I'm only
1:36:49um restricting it to the two-dimensional
1:36:51parameters which are basically the
1:36:52weights of the linear layers and I'm
1:36:54skipping the biases and I'm skipping the
1:36:57um gamas and the betas in the bom just
1:37:00for Simplicity but you can also take a
1:37:03look at those as well but what's
1:37:04happening with the weights is um
1:37:06instructive by
1:37:07itself so here we have all the different
1:37:10weights their shapes uh so this is the
1:37:13embedding layer the first linear layer
1:37:15all the way to the very last linear
1:37:16layer and then we have the mean the
1:37:18standard deviation of all these
1:37:20parameters the histogram and you can see
1:37:23that actually doesn't look that amazing
1:37:24so there's some trouble in Paradise even
1:37:26though these gradients looked okay
1:37:28there's something weird going on here
1:37:30I'll get to that in a second and the
1:37:32last thing here is the gradient to data
1:37:34ratio so sometimes I like to visualize
1:37:37this as well because what this gives you
1:37:39a sense of is what is the scale of the
1:37:41gradient compared to the scale of the
1:37:44actual values and this is important
1:37:46because we're going to end up taking a
1:37:48step update um that is the learning rate
1:37:51times the gradient onto the data
1:37:54and so if the gradient has too large of
1:37:55magnitude if the numbers in there are
1:37:57too large compared to the numbers in
1:37:59data then you'd be in trouble but in
1:38:02this case the gradient to data is our
1:38:04low numbers so the values inside grad
1:38:07are 1,000 times smaller than the values
1:38:09inside data in these weights most of
1:38:13them now notably that is not true about
1:38:15the last layer and so the last layer
1:38:18actually here the output layer is a bit
1:38:19of a troublemaker in the way that this
1:38:21is currently arranged because you can
1:38:23see that the um last layer here in pink
1:38:28takes on values that are much larger
1:38:30than some of the values inside um inside
1:38:34the neural nut so the standard
1:38:36deviations are roughly 1 and3 throughout
1:38:39except for the last last uh layer which
1:38:41actually has roughly one -2 standard
1:38:44deviation of gradients and so the
1:38:46gradients on the last layer are
1:38:47currently about 100 times greater sorry
1:38:5110 times greater than all the other
1:38:53weights inside the neural net and so
1:38:56that's problematic because in the simple
1:38:58stochastic rting theend setup you would
1:39:00be training this last layer about 10
1:39:02times faster than you would be training
1:39:04the other layers at
1:39:06initialization now this actually like
1:39:08kind of fixes itself a little bit if you
1:39:10train for a bit longer so for example if
1:39:12I greater than 1,000 only then do a
1:39:15break let me reinitialize and then let
1:39:17me do it 1,000 steps and after 1,000
1:39:20steps we can look at the forward pass
1:39:24okay so you see how the neurons are a
1:39:26bit are saturating a bit and we can also
1:39:28look at the backward pass but otherwise
1:39:30they look good they're about equal and
1:39:32there's no shrinking to zero or
1:39:34exploding to Infinities and you can see
1:39:36that here in the weights uh things are
1:39:39also stabilizing a little bit so the
1:39:41Tails of the last pink layer are
1:39:42actually coming coming in during the
1:39:45optimization but certainly this is like
1:39:47a little bit of troubling especially if
1:39:49you are using a very simple update rule
1:39:51like stochastic gradient descent instead
1:39:52of a modern Optimizer like Adam now I'd
1:39:55like to show you one more plot that I
1:39:56usually look at when I train neural
1:39:58networks and basically the gradient to
1:40:01data ratio is not actually that
1:40:02informative because what matters at the
1:40:04end is not the gradient to data ratio
1:40:06but the update to the data ratio because
1:40:08that is the amount by which we will
1:40:10actually change the data in these
1:40:11tensors so coming up here what I'd like
1:40:14to do is I'd like to introduce a new
1:40:16update to data uh ratio it's going to be
1:40:20list and we're going to build it out
1:40:21every single iteration and here I'd like
1:40:23to keep track of basically the
1:40:26ratio every single
1:40:29iteration so without any gradients I'm
1:40:33comparing the update which is learning
1:40:35rate times the times the
1:40:37gradient that is the update that we're
1:40:39going to apply to every
1:40:41parameter uh so see I'm iterating over
1:40:43all the parameters and then I'm taking
1:40:45the basically standard deviation of the
1:40:46update we're going to apply and divided
1:40:49by the um actual content the data of of
1:40:53that parameter and its standard
1:40:55deviation so this is the ratio of
1:40:57basically how great are the updates to
1:41:00the values in these tensors then we're
1:41:02going to take a log of it and actually
1:41:03I'd like to take a log
1:41:0510 um just so it's a nicer
1:41:09visualization um so we're going to be
1:41:10basically looking at the exponents of uh
1:41:14the of this division here and then that
1:41:17item to pop out the float and we're
1:41:19going to be keeping track of this for
1:41:20all the parameters and adding it to
1:41:22these UD answer so now let me
1:41:24reinitialize and run a th iterations we
1:41:27can look at the activations the
1:41:30gradients and the parameter gradients as
1:41:33we did before but now I have one more
1:41:35plot here to
1:41:36introduce and what's Happening Here is
1:41:38we're are interval parameters and I'm
1:41:40constraining it again like I did here to
1:41:42just the
1:41:43weights so the number of dimensions in
1:41:46these sensors is two and then I'm
1:41:48basically plotting all of these um
1:41:50update ratios over time
1:41:54so when I plot this I plot those ratios
1:41:57and you can see that they evolve over
1:41:58time during initialization they take on
1:42:00certain values and then these updates s
1:42:02of like start stabilizing usually during
1:42:04training then the other thing that I'm
1:42:06plotting here is I'm plotting here like
1:42:08an approximate value that is a Rough
1:42:10Guide for what it roughly should be and
1:42:12it should be like roughly
1:42:14one3 and so that means that basically
1:42:17there's some values in the tensor um and
1:42:20they take on certain values and the
1:42:22updates to them at every iteration are
1:42:24no more than roughly 1,000th of the
1:42:27actual like magnitude in those tensors
1:42:30uh if this was much larger like for
1:42:32example if this was um if the log of
1:42:36this was like say negative 1 this is
1:42:37actually updating those values quite a
1:42:39lot they're undergoing a lot of change
1:42:42but the reason that the final rate the
1:42:44final uh layer here is an outlier is
1:42:46because this layer was artificially
1:42:49shrunk down to keep the soft Max um
1:42:51incom unconfident
1:42:54so here you see how we multiplied The
1:42:57Weight by
1:42:580.1 uh in the initialization to make the
1:43:00last layer prediction less confident
1:43:04that made that artificially made the
1:43:07values inside that tensor way too low
1:43:09and that's why we're getting temporarily
1:43:10a very high ratio but you see that that
1:43:12stabilizes over time once uh that weight
1:43:15starts to learn starts to learn but
1:43:18basically I like to look at the
1:43:19evolution of this update ratio for all
1:43:21my parameters usually and I like to make
1:43:23sure that it's not too much above onean
1:43:27neg3 roughly uh so around3 on this log
1:43:32plot if it's below -3 usually that means
1:43:34that the parameters are not trained fast
1:43:36enough so if our learning rate was very
1:43:38low let's do that
1:43:40experiment uh let's initialize and then
1:43:43let's actually do a learning rate of say
1:43:45one3 here so
1:43:480.001 if your learning rate is way too
1:43:50low
1:43:53this plot will typically reveal it so
1:43:56you see how all of these updates are way
1:43:59too small so the size of the update is
1:44:02uh basically uh 10,000 times um in
1:44:06magnitude to the size of the numbers in
1:44:09that tensor in the first place so this
1:44:10is a symptom of training way too
1:44:13slow so this is another way to sometimes
1:44:16set the learning rate and to get a sense
1:44:17of what that learning rate should be and
1:44:19ultimately this is something that you
1:44:20would uh keep track of
1:44:25if anything the learning rate here is a
1:44:27little bit on the higher side uh because
1:44:30you see that um we're above the black
1:44:33line of3 we're somewhere around -2.5
1:44:35it's like okay and uh but everything is
1:44:38like somewhat stabilizing and so this
1:44:40looks like a pretty decent setting of of
1:44:42um learning rates and so on but this is
1:44:44something to look at and when things are
1:44:46miscalibrated you will you will see very
1:44:47quickly so for
1:44:49example everything looks pretty well
1:44:51behaved right but just as a comparison
1:44:53when things are not properly calibrated
1:44:55what does that look like let me come up
1:44:57here and let's say that for example uh
1:45:00what do we do let's say that we forgot
1:45:02to apply this a fan in normalization so
1:45:05the weights inside the linear layers are
1:45:07just sampled from aaan and all the
1:45:09stages what happens to our how do we
1:45:12notice that something's off well the
1:45:15activation plot will tell you whoa your
1:45:16neurons are way too saturated uh the
1:45:18gradients are going to be all messed up
1:45:21uh the histogram for these weights are
1:45:22going to be all messed up as well and
1:45:25there's a lot of asymmetry and then if
1:45:27we look here I suspect it's all going to
1:45:29be also pretty messed up so uh you see
1:45:31there's a lot of uh discrepancy in how
1:45:34fast these layers are learning and some
1:45:36of them are learning way too fast so uh1
1:45:401.5 those are very large numbers in
1:45:42terms of this ratio again you should be
1:45:44somewhere around3 and not much more
1:45:46about that um so this is how
1:45:49miscalibrations of your neuron nuts are
1:45:51going to manifest and these kinds of
1:45:53plots here are a good way of um sort of
1:45:56bringing um those miscalibrations sort
1:45:59of uh to your attention and so you can
1:46:03address them okay so so far we've seen
1:46:05that when we have this linear tanh
1:46:07sandwich we can actually precisely
1:46:09calibrate the gains and make the
1:46:10activations the gradients and the
1:46:12parameters and the updates all look
1:46:14pretty decent but it definitely feels a
1:46:16little bit like balancing of a pencil on
1:46:19your finger and that's because this gain
1:46:22has to be very precisely calibrated so
1:46:26now let's introduce bat normalization
1:46:27layers into the fix into the mix and
1:46:30let's let's see how that helps fix the
1:46:32problem so
1:46:35here I'm going to take the bachom 1D
1:46:37class and I'm going to start placing it
1:46:40inside and as I mentioned before the
1:46:43standard typical place you would place
1:46:44it is between the linear layer so right
1:46:47after it but before the nonlinearity but
1:46:49people have definitely played with that
1:46:51and uh in fact you can get very similar
1:46:53results even if you place it after the
1:46:55nonlinearity um and the other thing that
1:46:58I wanted to mention is it's totally fine
1:46:59to also place it at the end uh after the
1:47:02last linear layer and before the L
1:47:04function so this is potentially fine as
1:47:06well um and in this case this would be
1:47:10output would be WAP
1:47:12size um now because the last layer is
1:47:16Bash we would not be changing the weight
1:47:18to make the softmax less confident we'd
1:47:20be changing the gamma because gamma
1:47:23remember in the bathroom is the variable
1:47:26that multiplicatively interacts with the
1:47:28output of that
1:47:31normalization so we can initialize this
1:47:34sandwich now we can train and we can see
1:47:37that the activations uh are going to of
1:47:39course look uh very good and they are
1:47:42going to necessarily look good because
1:47:44now before every single 10h layer there
1:47:46is a normalization in the bashor so this
1:47:50is unsurprisingly all uh looks pretty
1:47:52good it's going to be standard deviation
1:47:54of roughly 65 2% and roughly equal
1:47:57standard deviation throughout the entire
1:47:59layers so everything looks very
1:48:01homogeneous the gradients look good the
1:48:04weights look good and their
1:48:08distributions and then the
1:48:10updates also look um pretty reasonable
1:48:13uh we are going above3 a little bit but
1:48:16not by too much so all the parameters
1:48:19are training at roughly the same rate um
1:48:22here
1:48:24but now what we've gained is um we are
1:48:26going to be slightly less
1:48:30um brittle with respect to the gain of
1:48:33these so for example I can make the gain
1:48:35be say2 here um which is much much much
1:48:40slower than what we had with the tan
1:48:41H but as we'll see the activations will
1:48:44actually be exactly unaffected uh and
1:48:46that's because of again this explicit
1:48:48normalization the gradients are going to
1:48:50look okay the weight gradients are going
1:48:52to look okay okay but actually the
1:48:54updates will
1:48:56change and so even though the forward
1:48:59and backward pass to a very large extent
1:49:00look okay because of the backward pass
1:49:02of the Bator and how the scale of the
1:49:04incoming activations interacts in the
1:49:07Bator and its uh backward pass this is
1:49:10actually changing the um the scale of
1:49:14the updates on these parameters so the
1:49:16grades on gradients of these weights are
1:49:18affected so we still don't get it
1:49:21completely free pass to pass in arbitral
1:49:23um weights here but it everything else
1:49:26is significantly more robust in terms of
1:49:29the forward backward and the weight
1:49:32gradients it's just that you may have to
1:49:33retune your learning rate if you are
1:49:35changing sufficiently the the scale of
1:49:38the activations that are coming into the
1:49:40batch Norms so here for example this um
1:49:43we changed the gains of these linear
1:49:45layers to be greater and we're seeing
1:49:47that the updates are coming out lower as
1:49:49a
1:49:50result and then finally we can also so
1:49:53if we are using borms we don't actually
1:49:55need to necessarily let me reset this to
1:49:57one so there's no gain we don't
1:49:59necessarily even have to um normalize by
1:50:02fan in sometimes so if I take out the
1:50:04fan in so these are just now uh random
1:50:06gsh in we'll see that because of borm
1:50:09this will actually be relatively well
1:50:10behaved
1:50:13so the statistic look of course in the
1:50:16forward pass look good the gradients
1:50:18look good the uh backward uh the weight
1:50:21updates look okay A little bit of fat
1:50:24tails on some of the
1:50:25layers and uh this looks okay as well
1:50:29but as you as you can see uh we're
1:50:32significantly below ne3 so we'd have to
1:50:34bump up the learning rate of this bachor
1:50:36uh so that we are training more properly
1:50:39and in particular looking at this
1:50:40roughly looks like we have to 10x the
1:50:42learning rate to get to about
1:50:45one3 so we' come here and we would
1:50:48change this to be update of 1.0 and if I
1:50:52reinitialize
1:50:59then we'll see that everything still of
1:51:00course looks good and now we are roughly
1:51:03here and we expect this to be an okay
1:51:05training run so long story short we are
1:51:08significantly more robust to the gain of
1:51:10these linear layers whether or not we
1:51:12have to apply the fan in and then we can
1:51:14change the gain uh but we actually do
1:51:17have to worry a little bit about the
1:51:18update um scales and making sure that uh
1:51:21the learning rate is properly calibrated
1:51:23here but this the activations of the
1:51:25forward backward pass and the updates
1:51:27are are looking significantly more well
1:51:29behaved except for the global scale that
1:51:32is potentially being adjusted here okay
1:51:34so now let me summarize there are three
1:51:36things I was hoping to achieve with this
1:51:38section number one I wanted to introduce
1:51:40you to bat normalization which is one of
1:51:42the first modern innovations that we're
1:51:44looking into that helped stabilize very
1:51:47deep neural networks and their training
1:51:49and I hope you understand how the B
1:51:51normalization works and um how it would
1:51:54be used in a neural network number two I
1:51:56was hoping to py torify some of our code
1:51:59and wrap it up into these uh modules so
1:52:02like linear bash 1D 10h Etc these are
1:52:04layers or modules and they can be
1:52:07stacked up into neural nuts like Lego
1:52:09building blocks and these layers
1:52:12actually exist in pytorch and if you
1:52:15import torch NN then you can actually
1:52:17the way I've constructed it you can
1:52:19simply just use pytorch by prepending n
1:52:21and Dot to all these different
1:52:24layers and actually everything will just
1:52:27work because the API that I've developed
1:52:29here is identical to the API that
1:52:31pytorch uses and the implementation also
1:52:33is basically as far as I'm Weare
1:52:36identical to the one in pytorch and
1:52:38number three I tried to introduce you to
1:52:40the diagnostic tools that you would use
1:52:42to understand whether your neural
1:52:43network is in a good State dynamically
1:52:46so we are looking at the statistics and
1:52:48histograms and activation of the forward
1:52:50pass activ activations the backward pass
1:52:53gradients and then also we're looking at
1:52:55the weights that are going to be updated
1:52:56as part of stochastic gradi in ascent
1:52:58and we're looking at their means
1:53:00standard deviations and also the ratio
1:53:02of gradients to data or even better the
1:53:05updates to data and we saw that
1:53:08typically we don't actually look at it
1:53:10as a single snapshot Frozen in time at
1:53:12some particular iteration typically
1:53:14people look at this as a over time just
1:53:16like I've done here and they look at
1:53:18these update to data ratios and they
1:53:19make sure everything looks okay and in
1:53:21particular I said said that um
1:53:24W3 or basically ne3 on the lock scale is
1:53:27a good uh rough euristic for what you
1:53:30want this ratio to be and if it's way
1:53:32too high then probably the learning rate
1:53:34or the updates are a little too too big
1:53:36and if it's way too small that the
1:53:37learning rate is probably too small so
1:53:39that's just some of the things that you
1:53:41may want to play with when you try to
1:53:42get your neural network to uh work with
1:53:45very
1:53:45well now there's a number of things I
1:53:47did not try to achieve I did not try to
1:53:50beat our previous performance as an
1:53:51example by introducing using the bash
1:53:53layer actually I did try um and I found
1:53:56the new I used the learning rate finding
1:53:58mechanism that I've described before I
1:54:00tried to train a borm layer a borm
1:54:02neural nut and uh I actually ended up
1:54:04with results that are very very similar
1:54:06to what we've obtained before and that's
1:54:08because our performance now is not
1:54:10bottlenecked by the optimization which
1:54:13is what borm is helping with the
1:54:15performance at this stage is bottleneck
1:54:17by what I suspect is the context length
1:54:19of our context so currently we are
1:54:22taking three characters to predict the
1:54:24fourth one and I think we need to go
1:54:25beyond that and we need to look at more
1:54:27powerful architectures like recurrent
1:54:29neural networks and Transformers in
1:54:30order to further push um the lock
1:54:33probabilities that we're achieving on
1:54:34this data
1:54:35set and I also did not try to have a
1:54:39full explanation of all of these
1:54:41activations the gradients and the
1:54:42backward pass and the statistics of all
1:54:44these gradients and so you may have
1:54:46found some of the parts here un
1:54:47intuitive and maybe you're slightly
1:54:48confused about okay if I change the uh
1:54:51gain here how come that we need a
1:54:53different learning rate and I didn't go
1:54:54into the full detail because you'd have
1:54:56to actually look at the backward pass of
1:54:57all these different layers and get an
1:54:59intuitive understanding of how that
1:55:00works and I did not go into that in this
1:55:03lecture the purpose really was just to
1:55:05introduce you to the diagnostic tools
1:55:07and what they look like but there's
1:55:08still a lot of work remaining on the
1:55:10intuitive level to understand the
1:55:11initialization the backward pass and how
1:55:13all of that interacts uh but you
1:55:15shouldn't feel too bad because honestly
1:55:18we are getting to The Cutting Edge of
1:55:21where the field is
1:55:22we certainly haven't I would say soled
1:55:24initialization and we haven't soled back
1:55:27propagation and these are still very
1:55:29much an active area of research people
1:55:30are still trying to figure out what is
1:55:32the best way to initialize these
1:55:33networks what is the best update rule to
1:55:35use um and so on so none of this is
1:55:38really solved and we don't really have
1:55:39all the answers to all the to you know
1:55:42all these cases but at least uh you know
1:55:45we're making progress and at least we
1:55:46have some tools to tell us uh whether or
1:55:48not things are on the right track for
1:55:50now so
1:55:53I think we've made positive progress in
1:55:54this lecture and I hope you enjoyed that
1:55:56and I will see you next time

About this transcript

This page contains the full transcript of Building makemore Part 3: Activations & Gradients, BatchNorm by Andrej Karpathy, generated from the public captions YouTube serves with the video. The transcript has 21,386 words across 3,051 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.