YT YouTube2Text

YouTube2Text → Transcripts → Building makemore Part 5: Building a WaveNet

Building makemore Part 5: Building a WaveNet — Transcript

by Andrej Karpathy · 10,385 words · 1,559 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00hi everyone today we are continuing our
0:02implementation of make more our favorite
0:04character level language model
0:06now you'll notice that the background
0:07behind me is different that's because I
0:09am in Kyoto and it is awesome so I'm in
0:12a hotel room here
0:13now over the last few lectures we've
0:15built up to this architecture that is a
0:17multi-layer perceptron character level
0:19language model so we see that it
0:21receives three previous characters and
0:23tries to predict the fourth character in
0:24a sequence using a very simple multi
0:26perceptron using one hidden layer of
0:28neurons with 10ational neuralities
0:31so we'd like to do now in this lecture
0:33is I'd like to complexify this
0:34architecture in particular we would like
0:36to take more characters in a sequence as
0:38an input not just three and in addition
0:41to that we don't just want to feed them
0:42all into a single hidden layer because
0:45that squashes too much information too
0:46quickly instead we would like to make a
0:49deeper model that progressively fuses
0:51this information to make its guess about
0:53the next character in a sequence
0:55and so we'll see that as we make this
0:57architecture more complex we're actually
0:59going to arrive at something that looks
1:01very much like a wavenet
1:03the witness is this paper published by
1:05the point in 2016 and it is also a
1:09language model basically but it tries to
1:11predict audio sequences instead of
1:13character level sequences or Word level
1:15sequences but fundamentally the modeling
1:18setup is identical it is an auto
1:20aggressive model and it tries to predict
1:23next character in a sequence and the
1:25architecture actually takes this
1:26interesting hierarchical sort of
1:29approach to predicting the next
1:31character in a sequence uh with the
1:33street-like structure and this is the
1:35architecture and we're going to
1:36implement it in the course of this video
1:38so let's get started so the starter code
1:41for part five is very similar to where
1:43we ended up in in part three recall that
1:46part four was the manual black
1:47replication exercise that is kind of an
1:49aside so we are coming back to part
1:51three copy pasting chunks out of it and
1:53that is our starter code for part five
1:55I've changed very few things otherwise
1:57so a lot of this should look familiar to
1:59if you've gone through part three so in
2:01particular very briefly we are doing
2:03Imports we are reading our our data set
2:05of words and we are processing their set
2:09of words into individual examples and
2:11none of this data generation code has
2:13changed and basically we have lots and
2:15lots of examples in particular we have
2:17182 000 examples of three characters try
2:21to predict the fourth one and we've
2:24broken up every one of these words into
2:25little problems of given three
2:27characters predict the fourth one so
2:29this is our data set and this is what
2:30we're trying to get the neural lot to do
2:32now in part three we started to develop
2:35our code around these layer modules
2:39um that are for example like class
2:40linear and we're doing this because we
2:42want to think of these modules as
2:44building blocks and like a Lego building
2:47block bricks that we can sort of like
2:49stack up into neural networks and we can
2:51feed data between these layers and stack
2:53them up into a sort of graphs
2:56now we also developed these layers to
2:59have apis and signatures very similar to
3:01those that are found in pytorch so we
3:04have torch.nn and it's got all these
3:05layer building blocks that you would use
3:07in practice and we were developing all
3:09of these to mimic the apis of these so
3:11for example we have linear so there will
3:13also be a torch.nn.linear and its
3:17signature will be very similar to our
3:18signature and the functionality will be
3:20also quite identical as far as I'm aware
3:22so we have the linear layer with the
3:24Bass from 1D layer and the 10h layer
3:27that we developed previously
3:29and linear just as a matrix multiply in
3:32the forward pass of this module batch
3:35number of course is this crazy layer
3:36that we developed in the previous
3:37lecture and what's crazy about it is
3:40well there's many things number one it
3:42has these running mean and variances
3:44that are trained outside of back
3:46propagation they are trained using
3:49exponential moving average inside this
3:52layer when we call the forward pass
3:54in addition to that
3:56there's this training plug because the
3:58behavior of bathroom is different during
3:59train time and evaluation time and so
4:02suddenly we have to be very careful that
4:03bash form is in its correct state that
4:05it's in the evaluation state or training
4:07state so that's something to now keep
4:08track of something that sometimes
4:10introduces bugs
4:11uh because you forget to put it into the
4:13right mode and finally we saw that
4:15Bachelor couples the statistics or the
4:18the activations across the examples in
4:20the batch so normally we thought of the
4:22bat as just an efficiency thing but now
4:25we are coupling the computation across
4:28batch elements and it's done for the
4:30purposes of controlling the automation
4:32statistics as we saw in the previous
4:33video
4:34so it's a very weird layer at least a
4:36lot of bugs
4:38partly for example because you have to
4:40modulate the training in eval phase and
4:42so on
4:44um in addition for example you have to
4:46wait for uh the mean and the variance to
4:49settle and to actually reach a steady
4:51state and so um you have to make sure
4:53that you basically there's state in this
4:55layer and state is harmful uh usually
4:59now I brought out the generator object
5:02previously we had a generator equals g
5:04and so on inside these layers I've
5:07discarded that in favor of just
5:08initializing the torch RNG outside here
5:12use it just once globally just for
5:15Simplicity
5:16and then here we are starting to build
5:18out some of the neural network elements
5:19this should look very familiar we are we
5:22have our embedding table C and then we
5:24have a list of players and uh it's a
5:27linear feeds to Bachelor feeds to 10h
5:29and then a linear output layer and its
5:32weights are scaled down so we are not
5:33confidently wrong at the initialization
5:36we see that this is about 12 000
5:38parameters we're telling pytorch that
5:40the parameters require gradients
5:42the optimization is as far as I'm aware
5:44identical and should look very very
5:46familiar
5:47nothing changed here
5:49uh loss function looks very crazy we
5:52should probably fix this and that's
5:54because 32 batch elements are too few
5:56and so you can get very lucky lucky or
5:59unlucky in any one of these batches and
6:01it creates a very thick loss function
6:04um so we're going to fix that soon
6:06now once we want to evaluate the trained
6:08neural network we need to remember
6:09because of the bathroom layers to set
6:11all the layers to be training equals
6:13false so this only matters for the
6:15bathroom layer so far
6:17and then we evaluate
6:19we see that currently we have validation
6:22loss of 2.10 which is fairly good but
6:25there's still ways to go but even at
6:282.10 we see that when we sample from the
6:30model we actually get relatively
6:31name-like results that do not exist in a
6:34training set so for example Yvonne kilo
6:37Pros
6:40Alaia Etc so certainly not
6:43reasonable not unreasonable I would say
6:46but not amazing and we can still push
6:48this validation loss even lower and get
6:50much better samples that are even more
6:52name-like
6:53so let's improve this model
6:56okay first let's fix this graph because
6:58it is daggers in my eyes and I just
7:00can't take it anymore
7:01um so last I if you recall is a python
7:05list of floats so for example the first
7:0710 elements
7:10now what we'd like to do basically is we
7:12need to average up
7:14um some of these values to get a more
7:16sort of Representative uh value along
7:19the way so one way to do this is the
7:20following
7:21in part torch if I create for example
7:24a tensor of the first 10 numbers
7:27then this is currently a one-dimensional
7:29array but recall that I can view this
7:31array as two-dimensional so for example
7:33I can use it as a two by five array and
7:36this is a 2d tensor now two by five and
7:39you see what petroch has done is that
7:40the first row of this tensor is the
7:42first five elements and the second row
7:44is the second five elements
7:46I can also view it as a five by two as
7:48an example
7:50and then recall that I can also
7:52use negative one in place of one of
7:55these numbers
7:55and pytorch will calculate what that
7:58number must be in order to make the
7:59number of elements work out so this can
8:01be
8:03this or like that but it will work of
8:06course this would not work
8:09okay so this allows it to spread out
8:11some of the consecutive values into rows
8:13so that's very helpful because what we
8:15can do now is first of all we're going
8:17to create a torshot tensor out of the a
8:21list of floats
8:22and then we're going to view it as
8:24whatever it is but we're going to
8:26stretch it out into rows of 1000
8:29consecutive elements so the shape of
8:31this now becomes 200 by 1000. and each
8:35row is one thousand um consecutive
8:37elements in this list
8:39so that's very helpful because now we
8:41can do a mean along the rows
8:43and the shape of this will just be 200.
8:47and so we've taken basically the mean on
8:48every row so plt.plot of that should be
8:51something nicer
8:53much better
8:55so we see that we basically made a lot
8:56of progress and then here this is the
8:59learning rate Decay so here we see that
9:01the learning rate Decay subtracted a ton
9:03of energy out of the system and allowed
9:05us to settle into sort of the local
9:07minimum in this optimization
9:09so this is a much nicer plot let me come
9:12up and delete the monster and we're
9:15going to be using this going forward now
9:16next up what I'm bothered by is that you
9:19see our forward pass is a little bit
9:22gnarly and takes way too many lines of
9:24code
9:24so in particular we see that we've
9:26organized some of the layers inside the
9:28layers list but not all of them uh for
9:30no reason so in particular we see that
9:32we still have the embedding table a
9:34special case outside of the layers and
9:37in addition to that the viewing
9:39operation here is also outside of our
9:40layers so let's create layers for these
9:43and then we can add those layers to just
9:45our list
9:46so in particular the two things that we
9:48need is here we have this embedding
9:50table and we are indexing at the
9:53integers inside uh the batch XB uh
9:56inside the tensor xB
9:58so that's an embedding table lookup just
10:00done with indexing and then here we see
10:03that we have this view operation which
10:04if you recall from the previous video
10:06Simply rearranges the character
10:09embeddings and stretches them out into a
10:12row and effectively what print that does
10:14is the concatenation operation basically
10:16except it's free because viewing is very
10:19cheap in pytorch no no memory is being
10:22copied we're just re-representing how we
10:24view that tensor so let's create
10:27um
10:28modules for both of these operations the
10:31embedding operation and flattening
10:32operation
10:33so I actually wrote the code in just to
10:37save some time
10:38so we have a module embedding and a
10:40module pattern and both of them simply
10:43do the indexing operation in the forward
10:45pass and the flattening operation here
10:49and this C now will just become a salt
10:53dot weight inside an embedding module
10:56and I'm calling these layers
10:58specifically embedding a platinum
10:59because it turns out that both of them
11:01actually exist in pi torch so in
11:03phytorch we have n and Dot embedding and
11:06it also takes the number of embeddings
11:07and the dimensionality of the bedding
11:09just like we have here but in addition
11:11python takes in a lot of other keyword
11:13arguments that we are not using for our
11:15purposes yet
11:17and for flatten that also exists in
11:19pytorch and it also takes additional
11:21keyword arguments that we are not using
11:23so we have a very simple platform
11:26but both of them exist in pytorch
11:28they're just a bit more simpler and now
11:30that we have these we can simply take
11:33out some of these special cased
11:36um things so instead of C we're just
11:40going to have an embedding
11:41and of a cup size and N embed
11:45and then after the embedding we are
11:47going to flatten
11:48so let's construct those modules and now
11:51I can take out this the
11:53and here I don't have to special case
11:54anymore because now C is the embeddings
11:57weight and it's inside layers
12:01so this should just work
12:03and then here our forward pass
12:06simplifies substantially because we
12:08don't need to do these now outside of
12:10these layer outside and explicitly
12:13they're now inside layers
12:15so we can delete those
12:17but now to to kick things off we want
12:19this little X which in the beginning is
12:21just XB uh the tensor of integers
12:24specifying the identities of these
12:26characters at the input
12:27and so these characters can now directly
12:29feed into the first layer and this
12:31should just work
12:32so let me come here and insert a break
12:35because I just want to make sure that
12:36the first iteration of this runs and
12:38then there's no mistake so that ran
12:40properly and basically we substantially
12:42simplified the forward pass here okay
12:45I'm sorry I changed my microphone so
12:46hopefully the audio is a little bit
12:48better
12:49now one more thing that I would like to
12:51do in order to pytortify our code even
12:53further is that right now we are
12:54maintaining all of our modules in a
12:56naked list of layers and we can also
12:59simplify this uh because we can
13:01introduce the concept of Pi torch
13:03containers so in tors.nn which we are
13:05basically rebuilding from scratch here
13:07there's a concept of containers
13:09and these containers are basically a way
13:10of organizing layers into
13:13lists or dicts and so on so in
13:16particular there's a sequential which
13:18maintains a list of layers and is a
13:20module class in pytorch and it basically
13:23just passes a given input through all
13:25the layers sequentially exactly as we
13:27are doing here
13:28so let's write our own sequential
13:31I've written a code here and basically
13:33the code for sequential is quite
13:35straightforward we pass in a list of
13:37layers which we keep here and then given
13:39any input in a forward pass we just call
13:41all the layers sequentially and return
13:43the result in terms of the parameters
13:45it's just all the parameters of the
13:46child modules
13:48so we can run this and we can again
13:50simplify this substantially because we
13:52don't maintain this naked list of layers
13:54we now have a notion of a model which is
13:57a module and in particular is a
14:00sequential of all these layers
14:04and now parameters are simply just a
14:07model about parameters
14:09and so that list comprehension now lives
14:11here
14:13and then here we are press here we are
14:15doing all the things we used to do
14:17now here the code again simplifies
14:19substantially because we don't have to
14:22do this forwarding here instead of just
14:24call the model on the input data and the
14:26input data here are the integers inside
14:28xB so we can simply do logits which are
14:31the outputs of our model are simply the
14:33model called on xB
14:36and then the cross entropy here takes
14:38the logits and the targets
14:41so this simplifies substantially
14:43and then this looks good so let's just
14:46make sure this runs that looks good
14:49now here we actually have some work to
14:51do still here but I'm going to come back
14:52later for now there's no more layers
14:54there's a model that layers but it's not
14:57a to access attributes of these classes
15:00directly so we'll come back and fix this
15:01later
15:03and then here of course this simplifies
15:05substantially as well because logits are
15:07the model called on x
15:10and then these low Jets come here
15:14so we can evaluate the train and
15:15validation loss which currently is
15:17terrible because we just initialized the
15:19neural net and then we can also sample
15:21from the model and this simplifies
15:22dramatically as well
15:24because we just want to call the model
15:25onto the context and outcome logits
15:30and these logits go into softmax and get
15:32the probabilities Etc so we can sample
15:35from this model
15:37what did I screw up
15:42okay so I fixed the issue and we now get
15:44the result that we expect which is
15:46gibberish because the model is not
15:48trained because we re-initialize it from
15:49scratch
15:50the problem was that when I fixed this
15:52cell to be modeled out layers instead of
15:54just layers I did not actually run the
15:56cell and so our neural net was in a
15:58training mode and what caused the issue
16:01here is the bathroom layer as bathroom
16:03layer of the likes to do because
16:05Bachelor was in a training mode and here
16:07we are passing in an input which is a
16:09batch of just a single example made up
16:11of the context
16:12and so if you are trying to pass in a
16:15single example into a bash Norm that is
16:16in the training mode you're going to end
16:18up estimating the variance using the
16:20input and the variance of a single
16:21number is is not a number because it is
16:24a measure of a spread so for example the
16:26variance of just the single number five
16:28you can see is not a number and so
16:31that's what happened in the master
16:33basically caused an issue and then that
16:35polluted all of the further processing
16:37so all that we have to do was make sure
16:39that this runs and we basically made the
16:43issue of
16:45again we didn't actually see the issue
16:46with the loss we could have evaluated
16:48the loss but we got the wrong result
16:49because basharm was in the training mode
16:52and uh and so we still get a result it's
16:54just the wrong result because it's using
16:56the uh sample statistics of the batch
16:59whereas we want to use the running mean
17:00and running variants inside the bachelor
17:02and so
17:04again an example of introducing a bug
17:06inline because we did not properly
17:09maintain the state of what is training
17:10or not okay so I Rewritten everything
17:12and here's where we are as a reminder we
17:15have the training loss of 2.05 and
17:17validation 2.10
17:18now because these losses are very
17:21similar to each other we have a sense
17:22that we are not overfitting too much on
17:24this task and we can make additional
17:26progress in our performance by scaling
17:28up the size of the neural network and
17:29making everything bigger and deeper
17:32now currently we are using this
17:33architecture here where we are taking in
17:35some number of characters going into a
17:37single hidden layer and then going to
17:39the prediction of the next character
17:41the problem here is we don't have a
17:43naive way of making this bigger in a
17:46productive way we could of course use
17:48our layers sort of building blocks and
17:51materials to introduce additional layers
17:53here and make the network deeper but it
17:55is still the case that we are crushing
17:56all of the characters into a single
17:58layer all the way at the beginning
18:00and even if we make this a bigger layer
18:02and add neurons it's still kind of like
18:04silly to squash all that information so
18:07fast in a single step
18:09so we'd like to do instead is we'd like
18:11our Network to look a lot more like this
18:13in the wavenet case so you see in the
18:15wavenet when we are trying to make the
18:17prediction for the next character in the
18:18sequence it is a function of the
18:20previous characters that are feeding
18:22that feed in but not all of these
18:25different characters are not just
18:26crushed to a single layer and then you
18:28have a sandwich they are crushed slowly
18:31so in particular we take two characters
18:34and we fuse them into sort of like a
18:36diagram representation and we do that
18:38for all these characters consecutively
18:40and then we take the bigrams and we fuse
18:42those into four character level chunks
18:46and then we fuse that again and so we do
18:49that in this like tree-like hierarchical
18:51manner so we fuse the information from
18:53the previous context slowly into the
18:56network as it gets deeper and so this is
18:58the kind of architecture that we want to
18:59implement
19:00now in the wave Nets case this is a
19:02visualization of a stack of dilated
19:04causal convolution layers and this makes
19:07it sound very scary but actually the
19:08idea is very simple and the fact that
19:10it's a dilated causal convolution layer
19:12is really just an implementation detail
19:14to make everything fast we're going to
19:16see that later but for now let's just
19:18keep the basic idea of it which is this
19:20Progressive Fusion so we want to make
19:22the network deeper and at each level we
19:24want to fuse only two consecutive
19:26elements two characters then two bigrams
19:29then two four grams and so on so let's
19:32unplant this okay so first up let me
19:34scroll to where we built the data set
19:35and let's change the block size from 3
19:37to 8. so we're going to be taking eight
19:39characters of context to predict the
19:42ninth character so the data set now
19:44looks like this we have a lot more
19:45context feeding in to predict any next
19:47character in a sequence and these eight
19:49characters are going to be processed in
19:51this tree like structure
19:53now if we scroll here everything here
19:56should just be able to work so we should
19:58be able to redefine the network
19:59you see the number of parameters has
20:01increased by 10 000 and that's because
20:03the block size has grown so this first
20:06linear layer is much much bigger our
20:08linear layer now takes eight characters
20:10into this middle layer so there's a lot
20:13more parameters there but this should
20:15just run let me just break right after
20:18the very first iteration so you see that
20:20this runs just fine it's just that this
20:22network doesn't make too much sense
20:23we're crushing way too much information
20:25way too fast
20:26so let's now come in and see how we
20:29could try to implement the hierarchical
20:30scheme now before we dive into the
20:33detail of the re-implementation here I
20:35was just curious to actually run it and
20:37see where we are in terms of the
20:38Baseline performance of just lazily
20:40scaling up the context length so I'll
20:42let it run we get a nice loss curve and
20:45then evaluating the loss we actually see
20:46quite a bit of improvement just from
20:48increasing the context line length so I
20:51started a little bit of a performance
20:52log here and previously where we were is
20:54we were getting a performance of 2.10 on
20:57the validation loss and now simply
20:59scaling up the contact length from 3 to
21:018 gives us a performance of 2.02 so
21:05quite a bit of an improvement here and
21:07also when you sample from the model you
21:08see that the names are definitely
21:10improving qualitatively as well
21:13so we could of course spend a lot of
21:14time here tuning
21:16um uh tuning things and making it even
21:18bigger and scaling up the network
21:19further even with the simple
21:21um sort of setup here but let's continue
21:24and let's Implement here model and treat
21:27this as just a rough Baseline
21:28performance but there's a lot of
21:30optimization like left on the table in
21:32terms of some of the hyper parameters
21:34that you're hopefully getting a sense of
21:35now okay so let's scroll up now
21:38and come back up and what I've done here
21:41is I've created a bit of a scratch space
21:42for us to just like look at the forward
21:45pass of the neural net and inspect the
21:47shape of the tensor along the way as the
21:49neural net uh forwards so here I'm just
21:53temporarily for debugging creating a
21:55batch of just say four examples so four
21:58random integers then I'm plucking out
22:00those rows from our training set
22:02and then I'm passing into the model the
22:04input xB
22:06now the shape of XB here because we have
22:08only four examples is four by eight and
22:11this eight is now the current block size
22:14so uh inspecting XP we just see that we
22:18have four examples each one of them is a
22:19row of xB
22:21and we have eight characters here and
22:24this integer tensor just contains the
22:26identities of those characters
22:29so the first layer of our neural net is
22:31the embedding layer so passing XB this
22:33integer tensor through the embedding
22:35layer creates an output that is four by
22:37eight by ten
22:39so our embedding table has for each
22:42character a 10-dimensional vector that
22:44we are trying to learn
22:46and so what the embedding layer does
22:48here is it plucks out the embedding
22:50Vector for each one of these integers
22:53and organizes it all in a four by eight
22:56by ten tensor now
22:58so all of these integers are translated
23:00into 10 dimensional vectors inside this
23:02three-dimensional tensor now
23:04passing that through the flattened layer
23:06as you recall what this does is it views
23:09this tensor as just a 4 by 80 tensor and
23:12what that effectively does is that all
23:15these 10 dimensional embeddings for all
23:16these eight characters just end up being
23:18stretched out into a long row
23:21and that looks kind of like a
23:22concatenation operation basically so by
23:25viewing the tensor differently we now
23:27have a four by eighty and inside this 80
23:29it's all the 10 dimensional uh
23:32vectors just uh concatenate next to each
23:35other
23:36and then the linear layer of course
23:37takes uh 80 and creates 200 channels
23:40just via matrix multiplication
23:43so so far so good now I'd like to show
23:45you something surprising
23:47let's look at the insides of the linear
23:50layer and remind ourselves how it works
23:52the linear layer here in the forward
23:54pass takes the input X multiplies it
23:56with a weight and then optionally adds
23:58bias and the weight here is
24:00two-dimensional as defined here and the
24:02bias is one dimensional here
24:04so effectively in terms of the shapes
24:06involved what's happening inside this
24:08linear layer looks like this right now
24:10and I'm using random numbers here but
24:12I'm just illustrating the shapes and
24:15what happens
24:16basically a 4 by 80 input comes into the
24:18linear layer that's multiplied by this
24:2080 by 200 weight Matrix inside and
24:23there's a plus 200 bias and the shape of
24:25the whole thing that comes out of the
24:26linear layer is four by two hundred as
24:28we see here
24:30now notice here by the way that this
24:32here will create a 4x200 tensor and then
24:36plus 200 there's a broadcasting
24:38happening here about 4 by 200 broadcasts
24:41with 200 uh so everything works here
24:44so now the surprising thing that I'd
24:46like to show you that you may not expect
24:47is that this input here that is being
24:49multiplied uh doesn't actually have to
24:52be two-dimensional this Matrix multiply
24:55operator in pytorch is quite powerful
24:56and in fact you can actually pass in
24:58higher dimensional arrays or tensors and
25:00everything works fine so for example
25:02this could be four by five by eighty and
25:04the result in that case will become four
25:06by five by two hundred
25:08you can add as many dimensions as you
25:09like on the left here
25:11and so effectively what's happening is
25:13that the matrix multiplication only
25:15works on the last Dimension and the
25:17dimensions before it in the input tensor
25:19are left unchanged
25:24so that is basically these um these
25:27dimensions on the left are all treated
25:29as just a batch Dimension so we can have
25:32multiple batch dimensions and then in
25:34parallel over all those Dimensions we
25:36are doing the matrix multiplication on
25:38the last dimension
25:39so this is quite convenient because we
25:41can use that in our Network now
25:44because remember that we have these
25:46eight characters coming in
25:49and we don't want to now uh flatten all
25:51of it out into a large eight-dimensional
25:53vector
25:54because we don't want to Matrix multiply
25:5780.
25:59into a weight Matrix multiply
26:01immediately instead we want to group
26:03these
26:04like this
26:06so every consecutive two elements
26:09one two and three and four and five and
26:11six and seven and eight all of these
26:12should be now
26:14basically flattened out and multiplied
26:17by weight Matrix but all of these four
26:19groups here we'd like to process in
26:21parallel so it's kind of like a batch
26:23Dimension that we can introduce
26:25and then we can in parallel basically
26:28process all of these uh bigram groups in
26:33the four batch dimensions of an
26:34individual example and also over the
26:37actual batch dimension of the you know
26:39four examples in our example here so
26:42let's see how that works effectively
26:43what we want is right now we take a 4 by
26:4680
26:47and multiply it by 80 by 200
26:50to in the linear layer this is what
26:52happens
26:53but instead what we want is we don't
26:56want 80 characters or 80 numbers to come
26:58in we only want two characters to come
27:00in on the very first layer and those two
27:02characters should be fused
27:04so in other words we just want 20 to
27:07come in right 20 numbers would come in
27:11and here we don't want a 4 by 80 to feed
27:13into the linear layer we actually want
27:15these groups of two to feed in so
27:17instead of four by eighty we want this
27:19to be a 4 by 4 by 20.
27:23so these are the four groups of two and
27:27each one of them is ten dimensional
27:28vector
27:29so what we want is now is we need to
27:31change the flattened layer so it doesn't
27:33output a four by eighty but it outputs a
27:35four by four by Twenty where basically
27:38these um
27:39every two consecutive characters are uh
27:43packed in on the very last Dimension and
27:46then these four is the first batch
27:48Dimension and this four is the second
27:50batch Dimension referring to the four
27:52groups inside every one of these
27:54examples
27:55and then this will just multiply like
27:57this so this is what we want to get to
27:59so we're going to have to change the
28:01linear layer in terms of how many inputs
28:02it expects it shouldn't expect 80 it
28:05should just expect 20 numbers and we
28:07have to change our flattened layer so it
28:09doesn't just fully flatten out this
28:11entire example it needs to create a 4x4
28:14by 20 instead of four by eighty so let's
28:17see how this could be implemented
28:19basically right now we have an input
28:21that is a four by eight by ten that
28:23feeds into the flattened layer and
28:25currently the flattened layer just
28:27stretches it out so if you remember the
28:29implementation of flatten
28:31it takes RX and it just views it as
28:34whatever the batch Dimension is and then
28:35negative one
28:37so effectively what it does right now is
28:39it does e dot view of 4 negative one and
28:42the shape of this of course is 4 by 80.
28:45so that's what currently happens and we
28:48instead want this to be a four by four
28:49by Twenty where these consecutive
28:51ten-dimensional vectors get concatenated
28:54so you know how in Python you can take a
28:57list of range of 10
29:00so we have numbers from zero to nine and
29:03we can index like this to get all the
29:05even parts
29:06and we can also index like starting at
29:08one and going in steps up two to get all
29:11the odd parts
29:13so one way to implement this it would be
29:15as follows we can take e and we can
29:18index into it for all the batch elements
29:21and then just even elements in this
29:24Dimension so at indexes 0 2 4 and 8.
29:29and then all the parts here from this
29:31last dimension
29:33and this gives us the even characters
29:37and then here
29:39this gives us all the odd characters and
29:42basically what we want to do is we make
29:43sure we want to make sure that these get
29:44concatenated in pi torch and then we
29:47want to concatenate these two tensors
29:49along the second dimension
29:53so this and the shape of it would be
29:55four by four by Twenty this is
29:57definitely the result we want we are
29:58explicitly grabbing the even parts and
30:01the odd parts and we're arranging those
30:03four by four by ten right next to each
30:06other and concatenate
30:08so this works but it turns out that what
30:10also works is you can simply use a view
30:13again and just request the right shape
30:16and it just so happens that in this case
30:18those vectors will again end up being
30:21arranged in exactly the way we want so
30:23in particular if we take e and we just
30:25view it as a four by four by Twenty
30:27which is what we want
30:28we can check that this is exactly equal
30:30to but let me call this this is the
30:33explicit concatenation I suppose
30:36um
30:36so explosives dot shape is 4x4 by 20. if
30:40you just view it as 4x4 by 20 you can
30:42check that when you compare to explicit
30:46uh you got a big this is element wise
30:48operation so making sure that all of
30:49them are true that is the truth so
30:53basically long story short we don't need
30:54to make an explicit call to concatenate
30:56Etc we can simply take this input tensor
31:00to flatten and we can just view it in
31:03whatever way we want
31:04and in particular you don't want to
31:07stretch things out with negative one we
31:09want to actually create a
31:10three-dimensional array and depending on
31:12how many vectors that are consecutive we
31:15want to
31:16um fuse like for example two then we can
31:20just simply ask for this Dimension to be
31:2120. and um
31:24use a negative 1 here and python will
31:26figure out how many groups it needs to
31:27pack into this additional batch
31:29dimension
31:30so let's now go into flatten and
31:32implement this okay so I scroll up here
31:34to flatten and what we'd like to do is
31:36we'd like to change it now so let me
31:38create a Constructor and take the number
31:40of elements that are consecutive that we
31:42would like to concatenate now in the
31:44last dimension of the output
31:46so here we're just going to remember
31:48solve.n equals n
31:50and then I want to be careful here
31:52because pipe pytorch actually has a
31:54torch to flatten and its keyword
31:56arguments are different and they kind of
31:58like function differently so R flatten
32:00is going to start to depart from patreon
32:02flatten so let me call it flat flatten
32:04consecutive or something like that just
32:06to make sure that our apis are about
32:08equal
32:09so this uh basically flattens only some
32:13n consecutive elements and puts them
32:15into the last dimension
32:17now here the shape of X is B by T by C
32:21so let me
32:23pop those out into variables and recall
32:26that in our example down below B was 4 T
32:28was 8 and C was 10.
32:33now instead of doing x dot view of B by
32:37negative one
32:39right this is what we had before
32:44we want this to be B by
32:47um negative 1 by
32:49and basically here we want c times n
32:52that's how many consecutive elements we
32:55want
32:56and here instead of negative one I don't
32:58super love the use of negative one
33:00because I like to be very explicit so
33:02that you get error messages when things
33:03don't go according to your expectation
33:04so what do we expect here we expect this
33:07to become t
33:09divide n using integer division here
33:12so that's what I expect to happen
33:14and then one more thing I want to do
33:15here is remember previously all the way
33:18in the beginning n was three and uh
33:21basically we're concatenating
33:23um all the three characters that existed
33:25there
33:26so we basically are concatenated
33:28everything
33:29and so sometimes I can create a spurious
33:31dimension of one here so if it is the
33:34case that x dot shape at one is one then
33:37it's kind of like a spurious dimension
33:39um so we don't want to return a
33:41three-dimensional tensor with a one here
33:44we just want to return a two-dimensional
33:46tensor exactly as we did before
33:48so in this case basically we will just
33:50say x equals x dot squeeze that is a
33:54pytorch function
33:56and squeeze takes a dimension that it
34:01either squeezes out all the dimensions
34:02of a tensor that are one or you can
34:05specify the exact Dimension that you
34:08want to be squeezed and again I like to
34:10be as explicit as possible always so I
34:12expect to squeeze out the First
34:13Dimension only
34:15of this tensor
34:17this three-dimensional tensor and if
34:19this Dimension here is one then I just
34:21want to return B by c times n
34:24and so self dot out will be X and then
34:26we return salt dot out
34:28so that's the candidate implementation
34:30and of course this should be self.n
34:33instead of just n
34:34so let's run
34:36and let's come here now
34:39and take it for a spin so flatten
34:41consecutive
34:44and in the beginning let's just use
34:47eight so this should recover the
34:49previous Behavior so flagging
34:51consecutive of eight uh which is the
34:53current block size
34:55we can do this uh that should recover
34:57the previous Behavior
34:59so we should be able to run the model
35:02and here we can inspect I have a little
35:06code snippet here where I iterate over
35:08all the layers I print the name of this
35:11class and the shape
35:14and so we see the shapes as we expect
35:17them after every single layer in the top
35:19bit so now let's try to restructure it
35:22using our flattened consecutive and do
35:25it hierarchically so in particular
35:28we want to flatten consecutive not just
35:30not block size but just two
35:33and then we want to process this with
35:34linear now then the number of inputs to
35:37this linear will not be an embed times
35:38block size it will now only be n embed
35:41times two
35:4220.
35:44this goes through the first layer and
35:46now we can in principle just copy paste
35:48this
35:49now the next linear layer should expect
35:51and hidden times two
35:53and the last piece of it should expect
35:58and it enters 2 again
36:01so this is sort of like the naive
36:03version of it
36:04um
36:05so running this we now have a much much
36:07bigger model
36:09and we should be able to basically just
36:10forward the model
36:13and now we can inspect uh the numbers in
36:16between
36:17so four byte by 20
36:19was Platinum consecutively into four by
36:21four by Twenty
36:23this was projected into four by four by
36:24two hundred
36:26and then bash storm just worked out of
36:29the box we have to verify that bastron
36:31does the correct thing even though it
36:33takes a three-dimensional impedance that
36:34are two dimensional input
36:36then we have 10h which is element wise
36:38then we crushed it again so if we
36:41flatten consecutively and ended up with
36:42a four by two by 400 now
36:45then linear brought it back down to 200
36:47batch room 10h and lastly we get a 4 by
36:50400 and we see that the flattened
36:52consecutive for the last flatten here uh
36:54it squeezed out that dimension of one so
36:57we only ended up with four by four
36:58hundred and then linear Bachelor on 10h
37:00and uh the last linear layer to get our
37:04logents and so The Lodges end up in the
37:06same shape as they were before but now
37:08we actually have a nice three layer
37:10neural nut and it basically corresponds
37:12to whoops sorry it basically corresponds
37:15exactly to this network now except only
37:18this piece here because we only have
37:20three layers whereas here in this
37:22example there's uh four layers with the
37:25total receptive field size of 16
37:28characters instead of just eight
37:29characters so the block size here is 16.
37:32so this piece of it's basically
37:34implemented here
37:36um now we just have to kind of figure
37:38out some good Channel numbers to use
37:40here now in particular I changed the
37:42number of hidden units to be 68 in this
37:45architecture because when I use 68 the
37:47number of parameters comes out to be 22
37:49000 so that's exactly the same that we
37:52had before and we have the same amount
37:54of capacity at this neural net in terms
37:56of the number of parameters but the
37:57question is whether we are utilizing
37:59those parameters in a more efficient
38:00architecture so what I did then is I got
38:03rid of a lot of the debugging cells here
38:05and I rerun the optimization and
38:07scrolling down to the result we see that
38:09we get the identical performance roughly
38:12so our validation loss now is 2.029 and
38:15previously it was 2.027 so controlling
38:18for the number of parameters changing
38:20from the flat to hierarchical is not
38:21giving us anything yet
38:23that said there are two things
38:25um to point out number one we didn't
38:27really torture the um architecture here
38:29very much this is just my first guess
38:31and there's a bunch of hyper parameters
38:33search that we could do in order in
38:35terms of how we allocate uh our budget
38:37of parameters to what layers number two
38:39we still may have a bug inside the
38:42bachelor 1D layer so let's take a look
38:44at
38:45um uh that because it runs but does it
38:49do the right thing
38:50so I pulled up the layer inspector sort
38:53of that we have here and printed out the
38:55shape along the way and currently it
38:57looks like the batch form is receiving
38:58an input that is 32 by 4 by 68 right and
39:03here on the right I have the current
39:04implementation of Bachelor that we have
39:05right now
39:06now this bachelor assumed in the way we
39:09wrote it and at the time that X is
39:11two-dimensional so it was n by D where n
39:15was the batch size so that's why we only
39:17reduced uh the mean and the variance
39:19over the zeroth dimension but now X will
39:21basically become three-dimensional so
39:23what's happening inside the bachelor
39:24right now and how come it's working at
39:26all and not giving any errors the reason
39:28for that is basically because everything
39:30broadcasts properly but the bachelor is
39:32not doing what we need what we wanted to
39:34do
39:35so in particular let's basically think
39:37through what's happening inside the
39:38bathroom uh looking at what's what's do
39:41What's Happening Here
39:43I have the code here
39:45so we're receiving an input of 32 by 4
39:47by 68 and then we are doing uh here x
39:52dot mean here I have e instead of X but
39:54we're doing the mean over zero and
39:57that's actually giving us 1 by 4 by 68.
39:59so we're doing the mean only over the
40:01very first Dimension and it's giving us
40:03a mean and a variance that still
40:05maintain this Dimension here
40:07so these means are only taking over 32
40:10numbers in the First Dimension and then
40:12when we perform this everything
40:14broadcasts correctly still
40:16but basically what ends up happening is
40:20when we also look at the running mean
40:26the shape of it so I'm looking at the
40:27model that layers at three which is the
40:28first bathroom layer and they're looking
40:30at whatever the running mean became and
40:32its shape
40:34the shape of this running mean now is 1
40:35by 4 by 68.
40:38right instead of it being
40:39um you know just a size of dimension
40:43because we have 68 channels we expect to
40:45have 68 means and variances that we're
40:47maintaining but actually we have an
40:49array of 4 by 68 and so basically what
40:51this is telling us is this bash Norm is
40:54only
40:55this bachelor is currently working in
40:57parallel
40:58over
41:014 times 68 instead of just 68 channels
41:06so basically we are maintaining
41:08statistics for every one of these four
41:10positions individually and independently
41:13and instead what we want to do is we
41:15want to treat this four as a batch
41:16Dimension just like the zeroth dimension
41:19so as far as the bachelor is concerned
41:22it doesn't want to average we don't want
41:24to average over 32 numbers we want to
41:26now average over 32 times four numbers
41:29for every single one of these 68
41:31channels
41:32and uh so let me now
41:34remove this
41:36it turns out that when you look at the
41:38documentation of torch.mean
41:42so let's go to torch.me
41:49in one of its signatures when we specify
41:51the dimension
41:53we see that the dimension here is not
41:54just it can be in or it can also be a
41:56tuple of ins so we can reduce over
41:59multiple integers at the same time over
42:02multiple Dimensions at the same time so
42:04instead of just reducing over zero we
42:05can pass in a tuple 0 1.
42:08and here zero one as well and then
42:10what's going to happen is the output of
42:12course is going to be the same
42:13but now what's going to happen is
42:15because we reduce over 0 and 1 if we
42:17look at immin.shape
42:20we see that now we've reduced we took
42:22the mean over both the zeroth and the
42:25First Dimension
42:26so we're just getting 68 numbers and a
42:28bunch of spurious Dimensions here
42:30so now this becomes 1 by 1 by 68 and the
42:34running mean and the running variance
42:35analogously will become one by one by
42:3768. so even though there are the
42:39spurious Dimensions uh the current the
42:41current the correct thing will happen in
42:43that we are only maintaining means and
42:45variances for 64 sorry for 68 channels
42:49and we're not calculating the mean
42:50variance across 32 times 4 dimensions so
42:54that's exactly what we want and let's
42:56change the implementation of bash term
42:581D that we have so that it can take in
43:01two-dimensional or three-dimensional
43:02inputs and perform accordingly so at the
43:05end of the day the fix is relatively
43:07straightforward basically the dimension
43:09we want to reduce over is either 0 or
43:12the Tuple zero and one depending on the
43:14dimensionality of X so if x dot and dim
43:16is two so it's a two dimensional tensor
43:18then Dimension we want to reduce over is
43:20just the integer zero
43:22L if x dot ending is three so it's a
43:24three-dimensional tensor then the dims
43:26we're going to assume are zero and one
43:29that we want to reduce over and then
43:31here we just pass in dim
43:33and if the dimensionality of X is
43:35anything else we'll now get an error
43:36which is good
43:38um so that should be the fix now I want
43:41to point out one more thing we're
43:42actually departing from the API of Pi
43:44torch here a little bit because when you
43:46come to batch room 1D and pytorch you
43:48can scroll down and you can see that the
43:50input to this layer can either be n by C
43:53where n is the batch size and C is the
43:55number of features or channels or it
43:57actually does accept three-dimensional
43:59inputs but it expects it to be n by C by
44:01L
44:02where LSA like the sequence length or
44:04something like that
44:05so um
44:07this is problem because you see how C is
44:09nested here in the middle and so when it
44:12gets three-dimensional inputs this bash
44:14term layer will reduce over zero and two
44:17instead of zero and one so it basically
44:20Pi torch batch number one D layer
44:22assumes that c will always be the First
44:25Dimension whereas we'll we assume here
44:28that c is the last Dimension and there
44:30are some number of batch Dimensions
44:32beforehand
44:34um
44:34and so
44:36it expects n by C or M by C by all we
44:39expect and by C or n by L by C
44:42and so it's a deviation
44:45um
44:46I think it's okay I prefer it this way
44:49honestly so this is the way that we will
44:50keep it for our purposes
44:52so I redefined the layers re-initialize
44:54the neural net and did a single forward
44:55pass with a break just for one step
44:57looking at the shapes along the way
44:59they're of course identical all the
45:01shapes are the same but the way we see
45:03that things are actually working as we
45:05want them to now is that when we look at
45:07the bathroom layer the running mean
45:08shape is now one by one by 68. so we're
45:11only maintaining 68 means for every one
45:13of our channels and we're treating both
45:15the zeroth and the First Dimension as a
45:17batch Dimension which is exactly what we
45:19want so let me retrain the neural lot
45:21now okay so I retrained the neural net
45:22with the bug fix we get a nice curve and
45:25when we look at the validation
45:25performance we do actually see a slight
45:27Improvement so we went from 2.029 to
45:302.022 so basically the bug inside the
45:32bathroom was holding up us back like a
45:35little bit it looks like and we are
45:37getting a tiny Improvement now but it's
45:39not clear if this is statistical
45:40significant
45:41um
45:42and the reason we slightly expect an
45:44improvement is because we're not
45:46maintaining so many different means and
45:47variances that are only estimated using
45:49using 32 numbers effectively now we are
45:52estimating them using 32 times 4 numbers
45:54so you just have a lot more numbers that
45:56go into any one estimate of the mean and
45:58variance and it allows things to be a
46:01bit more stable and less Wiggly inside
46:03those estimates of those statistics so
46:07pretty nice with this more General
46:08architecture in place we are now set up
46:10to push the performance further by
46:12increasing the size of the network so
46:14for example I bumped up the number of
46:16embeddings to 24 instead of 10 and also
46:19increased number of hidden units but
46:21using the exact same architecture we now
46:23have 76 000 parameters and the training
46:25takes a lot longer but we do get a nice
46:28curve and then when you actually
46:29evaluate the performance we are now
46:31getting validation performance of 1.993
46:33so we've crossed over the 2.0 sort of
46:36territory and right about 1.99 but we
46:39are starting to have to wait quite a bit
46:42longer and we're a little bit in the
46:44dark with respect to the correct setting
46:46of the hyper parameters here and the
46:47learning rates and so on because the
46:48experiments are starting to take longer
46:50to train and so we are missing sort of
46:52like an experimental harness on which we
46:54could run a number of experiments and
46:56really tune this architecture very well
46:58so I'd like to conclude now with a few
46:59notes we basically improved our
47:02performance from a starting of 2.1 down
47:04to 1.9 but I don't want that to be the
47:06focus because honestly we're kind of in
47:08the dark we have no experimental harness
47:10we're just guessing and checking and
47:12this whole thing is terrible we're just
47:13looking at the training loss normally
47:15you want to look at both the training
47:17and the validation loss together and the
47:19whole thing looks different if you're
47:20actually trying to squeeze out numbers
47:23that said we did implement this
47:25architecture from the wavenet paper but
47:28we did not implement this specific uh
47:31forward pass of it where you have a more
47:33complicated a linear layer sort of that
47:35is this gated linear layer kind of and
47:38there's residual connections and Skip
47:40connections and so on so we did not
47:42Implement that we just implemented this
47:44structure I would like to briefly hint
47:46or preview how what we've done here
47:48relates to convolutional neural networks
47:50as used in the wavenet paper and
47:52basically the use of convolutions is
47:54strictly for efficiency it doesn't
47:56actually change the model we've
47:57implemented
47:58so here for example
48:00let me look at a specific name to work
48:02with an example so there's a name in our
48:05training set and it's DeAndre and it has
48:08seven letters so that is eight
48:10independent examples in our model so all
48:12these rows here are independent examples
48:14of the Android
48:16now you can forward of course any one of
48:18these rows independently so I can take
48:20my model and call call it on any
48:24individual index notice by the way here
48:26I'm being a little bit tricky
48:28the reason for this is that extra at
48:30seven that shape is just
48:33um one dimensional array of eight so you
48:36can't actually call the model on it
48:37you're going to get an error because
48:39there's no batch dimension
48:41so when you do extra at
48:45a list of seven then the shape of this
48:47becomes one by eight so I get an extra
48:49batch dimension of one and then we can
48:52forward the model
48:53so
48:55that forwards a single example and you
48:57might imagine that you actually may want
48:59to forward all of these eight
49:01um at the same time
49:03so pre-allocating some memory and then
49:05doing a for Loop eight times and
49:07forwarding all of those eight here will
49:10give us all the logits in all these
49:11different cases
49:13now for us with the model as we've
49:14implemented it right now this is eight
49:16independent calls to our model
49:18but what convolutions allow you to do is
49:20it allow you to basically slide this
49:22model efficiently over the input
49:24sequence and so this for Loop can be
49:27done not outside in Python but inside of
49:31kernels in Cuda and so this for Loop
49:33gets hidden into the convolution
49:35so the convolution basically you can
49:37cover this it's a for Loop applying a
49:40little linear filter over space of some
49:43input sequence and in our case the space
49:45we're interested in is one dimensional
49:46and we're interested in sliding these
49:48filters over the input data
49:51so this diagram actually is fairly good
49:54as well
49:55basically what we've done is here they
49:57are highlighting in Black one individ
49:59one single sort of like tree of this
50:01calculation so just calculating the
50:03single output example here
50:06um
50:07and so this is basically what we've
50:08implemented here we've implemented a
50:10single this black structure we've
50:13implemented that and calculated a single
50:15output like a single example
50:17but what collusions allow you to do is
50:19it allows you to take this black
50:20structure and kind of like slide it over
50:23the input sequence here and calculate
50:26all of these orange outputs at the same
50:29time or here that corresponds to
50:31calculating all of these outputs of
50:34um at all the positions of DeAndre at
50:37the same time
50:38and the reason that this is much more
50:41efficient is because number one as I
50:43mentioned the for Loop is inside the
50:45Cuda kernels in the sliding so that
50:48makes it efficient but number two notice
50:50the variable reuse here for example if
50:52we look at this circle this node here
50:54this node here is the right child of
50:56this node but is also the left child of
50:59the node here
51:01and so basically this node and its value
51:03is used twice
51:05and so right now in this naive way we'd
51:08have to recalculate it but here we are
51:11allowed to reuse it
51:12so in the convolutional neural network
51:14you think of these linear layers that we
51:16have up above as filters and we take
51:19these filters and they're linear filters
51:21and you slide them over input sequence
51:23and we calculate the first layer and
51:25then the second layer and then the third
51:26layer and then the output layer of the
51:28sandwich and it's all done very
51:30efficiently using these convolutions
51:32so we're going to cover that in a future
51:34video the second thing I hope you took
51:35away from this video is you've seen me
51:37basically Implement all of these layer
51:40Lego building blocks or module building
51:42blocks and I'm implementing them over
51:45here and we've implemented a number of
51:46layers together and we've also
51:48implemented these these containers and
51:51we've overall pytorchified our code
51:53quite a bit more
51:54now basically what we're doing here is
51:56we're re-implementing torch.nn which is
51:59the neural networks library on top of
52:02torch.tensor and it looks very much like
52:04this except it is much better because
52:07because it's in pi torch instead of
52:08jingling my Jupiter notebook so I think
52:11going forward I will probably have
52:13considered us having unlocked
52:15um torch.nn we understand roughly what's
52:18in there how these modules work how
52:19they're nested and what they're doing on
52:21top of torture tensor so hopefully we'll
52:24just uh we'll just switch over and
52:25continue and start using torch.net
52:27directly the next thing I hope you got a
52:29bit of a sense of is what the
52:31development process of building deep
52:33neural networks looks like which I think
52:35was relatively representative to some
52:36extent so number one we are spending a
52:39lot of time in the documentation page of
52:41pytorch and we're reading through all
52:44the layers looking at documentations
52:45where the shapes of the inputs what can
52:48they be what does the layer do and so on
52:51unfortunately I have to say the
52:53patreon's documentation is not are very
52:55good they spend a ton of time on
52:57Hardcore engineering of all kinds of
52:59distributed Primitives Etc but as far as
53:01I can tell no one is maintaining any
53:03documentation it will lie to you it will
53:06be wrong it will be incomplete it will
53:08be unclear so unfortunately it is what
53:12it is and you just kind of do your best
53:14um with what they've given us
53:18um number two
53:20uh the other thing that I hope you got a
53:22sense of is there's a ton of trying to
53:24make the shapes work and there's a lot
53:26of gymnastics around these
53:27multi-dimensional arrays and are they
53:29two-dimensional three-dimensional
53:30four-dimensional uh what layers take
53:32what shapes is it NCL or NLC and you're
53:36promoting and viewing and it just can
53:39get pretty messy and so that brings me
53:40to number three I very often prototype
53:43these layers and implementations in
53:44jupyter notebooks and make sure that all
53:46the shapes work out and I'm spending a
53:48lot of time basically babysitting the
53:50shapes and making sure everything is
53:52correct and then once I'm satisfied with
53:54the functionality in the Jupiter
53:55notebook I will take that code and copy
53:57paste it into my repository of actual
53:59code that I'm training with and so then
54:02I'm working with vs code on the side so
54:04I usually have jupyter notebook and vs
54:06code I develop in Jupiter notebook I
54:07paste into vs code and then I kick off
54:09experiments from from the reaper of
54:11course from the code repository so
54:14that's roughly some notes on the
54:16development process of working with
54:17neurons lastly I think this lecture
54:19unlocks a lot of potential further
54:21lectures because number one we have to
54:23convert our neural network to actually
54:25use these dilated causal convolutional
54:27layers so implementing the comnet number
54:30two potentially starting to get into
54:32what this means whatever residual
54:34connections and Skip connections and why
54:36are they useful
54:37number three we as I mentioned we don't
54:40have any experimental harness so right
54:42now I'm just guessing checking
54:44everything this is not representative of
54:45typical deep learning workflows you have
54:47to set up your evaluation harness you
54:49can kick off experiments you have lots
54:51of arguments that your script can take
54:53you're you're kicking off a lot of
54:54experimentation you're looking at a lot
54:56of plots of training and validation
54:57losses and you're looking at what is
54:59working and what is not working and
55:01you're working on this like population
55:02level and you're doing all these hyper
55:04parameter searches and so we've done
55:06none of that so far so how to set that
55:09up and how to make it good I think as a
55:11whole another topic number three we
55:14should probably cover recurring neural
55:16networks RNs lstm's grooves and of
55:19course Transformers so many uh places to
55:22go and we'll cover that in the future
55:24for now bye sorry I forgot to say that
55:27if you are interested I think it is kind
55:30of interesting to try to beat this
55:31number 1.993 because I really haven't
55:34tried a lot of experimentation here and
55:36there's quite a bit of fruit potentially
55:37to still purchase further so I haven't
55:40tried any other ways of allocating these
55:42channels in this neural net maybe the
55:44number of dimensions for the embedding
55:47is all wrong maybe it's possible to
55:49actually take the original network with
55:50just one hidden layer and make it big
55:53enough and actually beat my fancy
55:54hierarchical Network it's not obvious
55:56that would be kind of embarrassing if
55:59this did not do better even once you
56:01torture it a little bit maybe you can
56:03read the weight net paper and try to
56:04figure out how some of these layers work
56:06and Implement them yourselves using what
56:07we have
56:08and of course you can always tune some
56:10of the initialization or some of the
56:12optimization and see if you can improve
56:15it that way so I'd be curious if people
56:16can come up with some ways to beat this
56:18and yeah that's it for now bye

About this transcript

This page contains the full transcript of Building makemore Part 5: Building a WaveNet by Andrej Karpathy, generated from the public captions YouTube serves with the video. The transcript has 10,385 words across 1,559 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.