YT YouTube2Text

YouTube2Text → Transcripts → The spelled-out intro to language modeling: building makemore

The spelled-out intro to language modeling: building makemore — Transcript

by Andrej Karpathy · 19,445 words · 3,332 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00Hi everyone. Hope you're well.
0:02And uh next up what I'd like to do is
0:03I'd like to build out make more.
0:06Like micrograd before it, make more is a
0:08repository that I have on my GitHub
0:10webpage. Uh you can look at it. Uh but
0:12just like with micrograd, I'm going to
0:14build it out step-by-step and I'm going
0:16to spell everything out. So, we're going
0:18to build it out slowly and together.
0:20Now, what is make more?
0:22Make more, uh as the name suggests, uh
0:24makes more of things that you give it.
0:27So, here's an example. names.txt is an
0:30example data set to make more.
0:32And when you look at names.txt, you'll
0:34find that it's a very large data set of
0:36names.
0:38So,
0:40here's lots of different types of names.
0:41In fact, I believe there are 32,000
0:43names that I've sort of found randomly
0:45on a government website.
0:47And if you train make more on this data
0:50set, it will learn to make more of
0:52things like this. Um and in particular
0:56in this case, that will mean more things
0:58that sound name-like, but are actually
1:01unique names. And maybe if you have a
1:03baby and you're trying to assign name,
1:05maybe you're looking for a cool new
1:06sounding unique name, make more might
1:08help you.
1:09So, here are some example generations
1:11from the neural network once we train it
1:14on our data set.
1:16So, here's some example unique names
1:18that it will generate. Dontal,
1:21uh Irot,
1:23uh Zendy,
1:24and so on.
1:25And so, all these sort of sound
1:27name-like, uh but they're not, of
1:28course, names.
1:30So, under the hood, make more is a
1:32character-level language model. So, what
1:35that means is that it is treating every
1:37single line here as an example.
1:39And within each example, it's treating
1:41them all as sequences of individual
1:44characters. So, r e e s e is this
1:48example, and that's the sequence of
1:49characters, and that's the level on
1:51which we are building out make more.
1:53And what it means to to a
1:55character-level language model then is
1:57that it's just sort of modeling those
1:59sequences of characters and it knows how
2:00to predict the next character in the
2:02sequence.
2:03Now, we're actually going to implement a
2:05large number of character level language
2:07models in terms of the neural networks
2:09that are involved in predicting the next
2:10character in a sequence. So, very simple
2:13bigram and bag of word models,
2:15multi-layer perceptrons, recurrent
2:17neural networks, all the way to modern
2:19transformers. In fact, the transformer
2:21that we will build will be basically the
2:23equivalent transformer to GPT-2 if you
2:26have heard of GPT. Uh so, that's kind of
2:28a big deal. It's a modern network and by
2:30the end of this series you will actually
2:32understand how that works
2:34um on the level of characters.
2:36Now, to give you a sense of the
2:38extensions here, uh after characters we
2:40will probably spend some time on the
2:42word level so that we can generate
2:43documents of words, not just little, you
2:45know, segments of characters.
2:47Uh but we can generate entire large,
2:49much larger documents.
2:51And then we're probably going to go into
2:52images and image text uh networks such
2:55as DALL-E, Stable Diffusion, and so on.
2:58But for now we have to start
2:59uh here, character level language
3:01modeling. Let's go.
3:03So, like before we are starting with a
3:04completely blank Jupyter notebook page.
3:07The first thing is I would like to
3:08basically load up the data set
3:10names.txt.
3:11So, we're going to open up names.txt for
3:13reading.
3:15And we're going to read in everything
3:17into a massive string.
3:19And then, because it's a massive string,
3:21we'd only like the individual words and
3:23put them in a list.
3:24So, let's call splitlines on that string
3:27to get all of our words as a Python list
3:30of strings.
3:32So, basically we can look at for example
3:33the first 10 words
3:35and we have that it's a list of Emma,
3:39Olivia, Ava, and so on.
3:41And if we look at
3:43the top of the page here, that is indeed
3:45what we see.
3:47Um
3:48So, that's good.
3:49This list actually makes me feel that
3:52this is probably sorted by frequency.
3:55But okay, so these are the words. Now,
3:58we'd like to actually like learn a
4:00little bit more about this data set.
4:01Let's look at the total number of words.
4:03We expect this to be roughly 32,000.
4:06And then what is the for example
4:07shortest word?
4:09So min of
4:11len of each word for W in words. So the
4:13shortest word will be length two.
4:18And max of len W for W in words. So the
4:21longest word will be
4:2315 characters.
4:24So let's now think through our very
4:25first language model.
4:27As I mentioned, a character-level
4:28language model is predicting the next
4:30character in a sequence given already
4:33some concrete sequence of characters
4:35before it.
4:36Now, what we have to realize here is
4:37that every single word here, like
4:39Isabella,
4:40is actually quite a few examples packed
4:43in to that single word.
4:45Because what is a an existence of a word
4:47like Isabella in the data set telling us
4:48really? It's saying that
4:51the character I is a very likely
4:53character to come first in the sequence
4:56of a name.
4:58The character S is likely to come
5:01after I.
5:04The character A is likely to come after
5:06IS.
5:07The character B is very likely to come
5:09after ISA. And so on all the way to A
5:12following Isabella.
5:14And then there's one more example
5:15actually packed in here.
5:17And that is that
5:19after there's Isabella,
5:21the word is very likely to end.
5:23So that's one more sort of explicit
5:25piece of information that we have here.
5:27That we have to be careful with.
5:29And so there's a lot packed into a
5:31single individual word in terms of the
5:33statistical structure of what's likely
5:35to follow in these character sequences.
5:38And then of course we don't have just an
5:39individual word. We actually have 32,000
5:41of these. And so there's a lot of
5:42structure here to model.
5:44Now, in the beginning what I'd like to
5:46start with is I would like to start with
5:48building a bigram language model.
5:51Now, in a bigram language model, we're
5:53always working with just two characters
5:55at a time.
5:56So, we're only looking at one character
5:59that we are given, and we're trying to
6:00predict the next character in the
6:02sequence.
6:03So, um what characters are likely to
6:06follow R? What characters are likely to
6:08follow A? And so on. And we're just
6:10modeling that kind of a little local
6:11structure.
6:13And we're forgetting the fact that we
6:15may have a lot more information. We're
6:17always just looking at the previous
6:18character to predict the next one. So,
6:20it's a very simple and weak language
6:21model, but I think it's a great place to
6:23start.
6:24So, now let's begin by looking at these
6:25bigrams in our data set and what they
6:27look like. And these bigrams, again, are
6:29just two characters in a row.
6:31So, for W words,
6:33each W here is an individual word
6:35string.
6:36We want to iterate uh for
6:39in We want to iterate this word
6:41with consecutive characters. So, two
6:44characters at a time, sliding it through
6:46the word. Now, a interesting, nice way,
6:49cute way to do this in Python, by the
6:50way, is doing something like this. For
6:52character one, character two in zip off
6:56W and W at one.
7:00one colon
7:01print
7:03character one, character two
7:04And let's not do all the words. Let's
7:05just do the first three words. And I'm
7:07going to show you in a second how this
7:09works.
7:10But for now, basically, as an example,
7:12let's just do the very first word alone,
7:13Emma.
7:15You see how we have a Emma, and this
7:18will just print EM, MM, MA.
7:21And the reason this works is because W
7:23is the string Emma, W at one colon is
7:26the string MMA,
7:28and zip takes two iterators, and it
7:32pairs them up, and then creates an
7:34iterator over the tuples of their
7:35consecutive entries.
7:37And if any one of these lists is shorter
7:39than the other, then it will just uh
7:41halt and return.
7:43So basically, uh that's why we return E
7:46M M M M M M A,
7:50but then because this iterator, the
7:51second one here, runs out of elements,
7:54zip just ends, and that's why we only
7:56get these tuples. So, pretty cute.
7:59So, these are the consecutive elements
8:01in the first word.
8:03Now, we have to be careful because we
8:04actually have more information here than
8:05just these three examples. As I
8:08mentioned, we know that E is the is very
8:11come first, and we know that A in this
8:13case is coming last.
8:15So, one way to do this is basically
8:17we're going to create a special array
8:20here of characters.
8:23And
8:24we're going to hallucinate a special
8:25start token here.
8:28I'm going to
8:29call it like special start.
8:32So, this is a list of one element plus
8:36W
8:37and then plus a special end character.
8:41And the reason I'm wrapping the list of
8:42W here is because W is a string in a
8:46list of W will just have the individual
8:48characters in the list.
8:50And then doing this again now, but not
8:54iterating over Ws but over the
8:56characters,
8:58will give us something like this.
9:00So, E is likely So, this is a bigram of
9:02the start character and E, and this is a
9:05bigram of the A and the special end
9:07character.
9:09And now we can look at, for example,
9:10what this looks like for
9:12Olivia or Ava.
9:14And indeed, we can actually
9:16potentially do this for the entire data
9:17set, but we won't print that. That's
9:19going to be too much.
9:20But these are the individual character
9:22bigrams, and we can print them.
9:25Now, in order to learn the statistics
9:26about which characters are likely to
9:28follow other characters, the simplest
9:30way in the bigram language models is to
9:32simply do it by counting.
9:34So, we're basically just going to count
9:36how often any one of these combinations
9:38occurs in the training set.
9:40In these words. So we're going to need
9:42some kind of a dictionary that's going
9:44to maintain some counts for every one of
9:46these bigrams. So let's use a dictionary
9:48B.
9:49And this will map these bigrams. So
9:52bigram is a tuple of character one
9:54character two.
9:56And then B at bigram
9:58will be B.get of bigram
10:01which is basically the same as B at
10:03bigram.
10:04But in the case that bigram is not in
10:07the dictionary B, we would like to by
10:09default return a zero.
10:11Plus one.
10:13So this will basically add up all the
10:15bigrams and count how often they occur.
10:18Let's get rid of printing.
10:20Or rather
10:22let's keep the printing and let's just
10:24inspect what B is in this case.
10:27And we see that many bigrams occur just
10:29a single time. This one allegedly
10:31occurred three times.
10:33So A was an ending character three
10:34times. And that's true for all of these
10:36words.
10:38All of Emma, Olivia, and Ava end with A.
10:41Uh so that's why this occurred three
10:43times.
10:45Um
10:46now let's do it for all the words.
10:51Oops, I should not have printed.
10:55I meant to erase that.
10:56Let's kill this.
10:58Let's just run.
11:00And now B will have the statistics of
11:02the entire data set.
11:04So these are the counts across all the
11:05words of the individual bigrams.
11:08And we could for example look at some of
11:10the most common ones and least common
11:11ones.
11:12Um this kind of grows in Python, but the
11:14way to do this, the simplest way I like,
11:17is we just use B.items.
11:19B.items returns
11:21the tuples of
11:24key value. In this case the keys are the
11:27character bigrams and the values are the
11:29counts.
11:31And so then what we want to do is we
11:32want to do um
11:35sorted of this.
11:38Uh but by default sort is on the first
11:41um
11:43on the first item of a tuple, but we
11:45want to sort by the values, which are
11:47the second element of a tuple, that is
11:49the key value.
11:50So we want to use the key
11:53equals lambda
11:55uh that takes the key value
11:57and returns the key value at the at one,
12:01not at zero, but at one, which is the
12:03count. So we want to sort by the count
12:07of these elements.
12:10And actually we want it to go backwards.
12:12So here what we have is the bigram QNR
12:16occurs only a single time.
12:18Uh DZ occurred only a single time.
12:20And when we sort this the other way
12:21around,
12:23we're going to see the most likely
12:25bigrams. So we see that N was very often
12:29an ending character
12:30many, many times. And apparently N
12:32almost always follows an A. And that's a
12:34very likely combination as well.
12:37Um
12:38so
12:39this is kind of the individual counts
12:42that we achieve over the entire data
12:43set.
12:45Now it's actually going to be
12:46significantly more convenient for us to
12:48keep this information in a
12:49two-dimensional array instead of a
12:51Python dictionary.
12:53So we're going to store this information
12:56in a 2D array
12:58and
13:00the rows are going to be the first
13:01character of the bigram and the columns
13:03are going to be the second character.
13:05And each entry in this two-dimensional
13:06array will tell us how often that first
13:08character follows the second character
13:10in the data set.
13:12So in particular, the array
13:14representation that we're going to use
13:16or the library is that of PyTorch.
13:18And PyTorch is a deep learning neural
13:21network framework, but part of it is
13:23also this torch.tensor, uh which allows
13:25us to create multi-dimensional arrays,
13:27and manipulate them very efficiently.
13:29So, let's import PyTorch, which you can
13:32do by import torch.
13:34And then we can create uh arrays.
13:37So, let's create a array of zeros.
13:40And we give it a um size of this array.
13:43Let's create a 3x5 array as an example.
13:47And
13:48this is a 3x5 array of zeros.
13:51And by default, you'll notice a data D
13:53type, which is short for data type, is
13:55float 32. So, these are single-precision
13:57floating-point numbers.
13:59Because we are going to represent
14:00counts, let's actually use D type as
14:03torch.int32.
14:05So, these are uh
14:0732-bit integers.
14:10So, now you see that we have integer
14:12data inside this tensor.
14:14Now, tensors allow us to really um
14:17manipulate all the individual entries
14:19and do it very efficiently.
14:20So, for example, if we want to change
14:22this bit, we have to index into the
14:24tensor. And in particular, here, this is
14:28the first row, and the um because it's
14:31zero-indexed. So, this is row index one,
14:35and column index 0 1 2 3.
14:38So, A at 1,3, we can set that to 1.
14:43And then A will have a 1 over there.
14:47We can of course also do things like
14:48this. So, now A will be 2 over there.
14:52Or 3.
14:53And also, we can for example say A 0 0
14:56is 5.
14:57And then A will have a 5 over here.
15:00So, that's how we can index into the
15:02arrays. Now, of course, the array that
15:04we are interested in is much much
15:05bigger. So, for our purposes, we have 26
15:08letters of the alphabet, and then we
15:10have two special characters, S and E.
15:13So, uh we want 26 + 2 or 28x28 array.
15:19And let's call it the the N, because
15:21it's going to represent sort of the
15:22counts.
15:24Let me erase this stuff.
15:26So, that's the array that starts at
15:28zeros, 28 by 28.
15:30And now let's copy paste this
15:33here.
15:34But instead of having a dictionary be
15:37which we're going to erase, we now have
15:39an N.
15:41Now, the problem here is that we have
15:42these characters which are strings, but
15:44we have to now
15:45um basically index into a um array, and
15:49we have to index using integers. So, we
15:51need some kind of a lookup table from
15:53characters to integers.
15:55So, let's construct such a character
15:56array.
15:58And the way we're going to do this is
15:59we're going to take all the words, which
16:01is a list of strings.
16:02We're going to concatenate all of it
16:04into a massive string. So, this is just
16:06simply the entire data set as a single
16:07string.
16:09We're going to pass this to the set
16:10constructor, which takes this massive
16:13string and throws out duplicates because
16:16sets do not allow duplicates.
16:18So, set of this will just be the set of
16:21all the lowercase characters.
16:24And there should be a total of 26 of
16:26them.
16:28And now we actually don't want a set, we
16:30want a list.
16:32But we don't want a list sorted in some
16:34weird arbitrary way, we want it to be
16:36sorted
16:37from A to Z.
16:39So, a sorted list.
16:41So, those are our characters.
16:45Now, what we want is this lookup table
16:47as I mentioned. So, let's create a
16:49special S2I, I will call it.
16:53Um S is string or character, and this
16:55will be an S2I mapping
16:58for
17:00IS in enumerate of these characters.
17:04So, enumerate basically gives us this
17:06iterator over the integer index and the
17:10actual element of the list. And then we
17:12are mapping the character to the
17:13integer.
17:15So, S2I
17:16is a mapping from A to 0, B to 1, etc.
17:19All the way from Z to 25.
17:24And that's going to be useful here, but
17:25we actually also have to specifically
17:27set that S will be 26.
17:29And S 2 I at E
17:32will be 27, right? Because Z was 25.
17:35So, those are the lookups. And now we
17:38can come here and we can map both
17:40character one and character two to their
17:41integers.
17:42So, this will be S 2 I at character one.
17:45And IX2 will be S 2 I of character two.
17:49And now we should be able to
17:52do this line, but using our array. So, N
17:55at IX1, IX2, this is the two-dimensional
17:58array indexing I showed you before,
18:00and honestly just plus equals one.
18:02Because everything starts at zero.
18:06So, this should work and give us a large
18:1028 by 28 array
18:13of all these counts. So, if we print N,
18:16this is the array, but of course it
18:18looks ugly. So, let's erase this ugly
18:21mess and let's try to visualize it a bit
18:22more nicer.
18:24So, for that we're going to use a
18:26library called Matplotlib.
18:28So, Matplotlib allows us to create
18:30figures. So, we can do things like
18:32plt.imshow of the counter array.
18:36So, this is the 28 by 28 array.
18:38And this is a structure, but even this,
18:41I would say, is still pretty ugly. So,
18:44we're going to try to create a much
18:45nicer visualization of it, and I wrote a
18:47bunch of code for that.
18:49Uh, the first thing we're going to need
18:50is
18:52we're going to need to invert this array
18:54here, this um dictionary. So, S 2 I is a
18:57mapping from S to I.
18:59And in I 2 S, we're going to reverse
19:02this dictionary. So, iterate over all
19:03the items and just reverse that array.
19:06So, I 2 S maps inversely from 0 to A, 1
19:10to B, etc.
19:12So, we'll need that.
19:14And then here's the code that I came up
19:16with to try to make this a little bit
19:17nicer.
19:20We create a figure.
19:22We plot N.
19:24And then we do and then we visualize a
19:26bunch of things later. Let me just run
19:28it so you get a sense of what all this
19:29is.
19:32Okay.
19:33So, you see here that we have
19:35the array spaced out. And every one of
19:38these is basically like B follows G zero
19:41times.
19:42B follows H 41 times. Um so, A follows J
19:46175 times.
19:48And so, what you can see that I'm doing
19:49here is first I show that entire array.
19:53And then I iterate over all the
19:54individual little cells here.
19:56And I create a character string here.
19:59Which is the inverse mapping I to S of
20:02the integer I and the integer J. So,
20:04that's These are the bigrams in a
20:06character representation.
20:08And then I plot just the bigram text.
20:12And then I plot the number of times that
20:14this bigram occurs.
20:16Now, the reason that there's a dot item
20:17here is because when you index into
20:20these arrays, these are torch tensors.
20:23You see that we still get a tensor back.
20:26So, the type of this thing, you'd think
20:28it would be just an integer 149, but
20:29it's actually torch.tensor.
20:32And so, if you do dot item, then it will
20:34pop out that individual integer.
20:38So, it'll just be 149.
20:40So, that's what's happening there. And
20:42these are just some options to make it
20:43look nice.
20:45So, what is the structure of this array?
20:47Um
20:49We have all these counts, and we see
20:50that some of them occur often and some
20:52of them do not occur often.
20:54Now, if you scrutinize this carefully,
20:56you will notice that we're not actually
20:57being very clever.
20:58That's because when you come over here,
21:00you'll notice that for example, we have
21:02an entire row of completely zeros. And
21:04that's because the end character
21:07is never possibly going to be the first
21:08character of a bigram because we're
21:10always placing these end tokens all at
21:12the end of the bigram.
21:14Similarly, we have entire columns of
21:16zeros here because the S
21:19character will never possibly be the
21:21second element of a bigram because we
21:23always start with S and we end with E
21:25and we only have the words in between.
21:27So, we have an entire column of zeros,
21:29an entire row of zeros. And in this
21:32little 2 by 2 matrix here as well, the
21:34only one that can possibly happen is if
21:36S directly follows E.
21:38That can be non-zero if we have a word
21:41that has no letters. So, in that case
21:43there's no letters in the word, it's an
21:44empty word and we just have S follows E.
21:47But, the other ones are just not
21:49possible.
21:50And so, we're basically wasting space
21:51and not only that, but the S and the E
21:53are getting very crowded here.
21:55I was using these brackets because
21:57there's convention in natural language
21:58processing to use these kinds of
22:00brackets to denote special tokens. Uh
22:03but, we're going to use something else.
22:05So, let's fix all of this and make it
22:06prettier.
22:08We're not actually going to have two
22:09special tokens, we're only going to have
22:11one special token.
22:13So, we're going to have n by n array of
22:1527 by 27 instead.
22:18Instead of having two, we will just have
22:21one and I will call it a dot.
22:24Okay?
22:27Let me swing this over here.
22:30Now, one more thing that I would like to
22:31do is I would actually like to make this
22:33special character have position zero and
22:36I would like to offset all the other
22:37letters off. I find that a little bit
22:39more pleasing.
22:41Um
22:42so,
22:44we need a plus one here so that the
22:46first character, which is A, will start
22:48at one.
22:49So, S to I will now be A starts at one
22:53and dot is zero.
22:55And uh I to S, of course, we're not
22:58changing this because I to S just
22:59creates a reverse mapping and this will
23:01work fine. So, one is A, two is B, zero
23:04is dot.
23:06So, we reverse that. Here,
23:09we have
23:10a dot and a dot.
23:13This should work fine.
23:14Make sure I start at zeros.
23:17Count. And then here, we don't go up to
23:1928, we go up to 27.
23:22And this should just work.
23:30Okay.
23:31So, we see that dot dot never happened.
23:33It's at zero because we don't have empty
23:35words.
23:36Then this row here now is just very
23:38simply the
23:40counts for all the first letters. So, G
23:45J starts a word, H starts a word, I
23:47starts a word, etc. And then these are
23:50all the ending
23:51characters.
23:53And in between, we have the structure of
23:54what characters follow each other.
23:57So, this is the counts array of our
23:59entire uh data set. So, this array
24:02actually has all of the information
24:03necessary for us to actually sample from
24:06this bigram
24:07character level language model.
24:09And
24:10roughly speaking, what we're going to do
24:12is we're just going to start following
24:13these probabilities and these counts,
24:15and we're going to start sampling from
24:17the from the model.
24:18So, in the beginning, of course,
24:20we start with the dot, the start token.
24:23Dot. So, to sample the first character
24:26of a name, we're looking at this row
24:28here.
24:30So, we see that we have the counts, and
24:32those counts are telling are telling us
24:34how often any one of these characters is
24:37to start a word.
24:39So, if we take this N,
24:41and we grab the first row,
24:44we can do that by using just indexing at
24:47zero,
24:48and then using this notation colon for
24:51the rest of that row.
24:53So, N zero colon
24:56is indexing into the zeroth um row and
24:59then it's grabbing all the columns.
25:02And so this will give us a
25:03one-dimensional array
25:05of the first row. So 0 4 4 10.
25:08You know it's 0 4 4 10 1 3 0 6 1 5 4 2
25:12etc. It's just the first row.
25:14The shape of this is 27. It's just a row
25:17of 27.
25:19And the other way that you can do this
25:21also is you just you don't need to
25:22actually give this
25:23you just uh grab the zeroth row like
25:25this. This is equivalent.
25:28Now these are the counts.
25:30And now what we'd like to do is we'd
25:31like to basically um sample from this.
25:35Since these are the raw counts, we
25:36actually have to convert this to
25:37probabilities.
25:39So we create a probability vector.
25:42So we'll take n of zero
25:45and we'll actually convert this to float
25:48first.
25:50Okay, so these integers are converted to
25:51float
25:52a floating point numbers. And the reason
25:54we're creating floats is because we're
25:56about to normalize these counts.
25:58So to create a probability distribution
26:00here, we want to divide
26:03we basically want to do p p p divide
26:05p.sum.
26:09And now we get a vector of smaller
26:11numbers and these are now probabilities.
26:13So of course because we divided by the
26:15sum, the sum of p now is one.
26:18So this is a nice proper probability
26:20distribution. It sums to one and this is
26:22giving us the probability for any single
26:24character to be the first uh character
26:26of a word.
26:28So now we can try to sample from this
26:29distribution. To sample from these
26:31distributions, we're going to use
26:32torch.multinomial which I've pulled up
26:34here.
26:36So torch.multinomial returns uh
26:39um samples from the multinomial
26:41probability distribution which is a
26:43complicated way of saying you give me
26:45probabilities and I will give you
26:47integers which are sampled according to
26:49the probability distribution.
26:51So this is the signature of the method,
26:53and to make everything deterministic,
26:54we're going to use a generator object in
26:57PyTorch.
26:59Uh so, this makes everything
27:00deterministic. So, you when you run this
27:01on your computer, you're going to the
27:03exact get the exact same results that
27:04I'm getting here on my computer.
27:07So, let me show you how this works.
27:09Um
27:12Here's the deterministic way of creating
27:15a torch generator object,
27:18seeding it with some number that we can
27:20agree on.
27:21So, that seeds a generator, gets gives
27:23us an object G,
27:25and then we can pass that G to a
27:27function
27:28that creates um
27:30here random numbers. torch.rand creates
27:32random numbers, three of them,
27:35and it's using this generator object to
27:37as a source of randomness.
27:40Uh so, uh
27:41without normalizing it,
27:44I can just print.
27:46Uh this is sort of like numbers between
27:48zero and one that are random according
27:50to this thing. And whenever I run it
27:52again,
27:53I'm always going to get the same result
27:54because I keep using the same generator
27:56object, which I'm seeding here.
27:58And then if I divide
28:01to normalize, I'm going to get a nice
28:04probability distribution of just three
28:05elements.
28:07And then we can use torch.multinomial to
28:09draw samples from it. So, this is what
28:11that looks like.
28:13torch.multinomial will take
28:16the torch tensor
28:18of probability distributions.
28:21Then we can ask for a number of samples,
28:22let's say 20.
28:24replacement equals true means that when
28:27we draw an element, uh we will uh we can
28:30draw it, and then we can put it back
28:31into the list of eligible indices to
28:34draw again.
28:35And we have to specify replacement as
28:37true because by default, uh for for some
28:39reason, it's false.
28:41Um and I think,
28:43you know, it's just something to be
28:44careful with.
28:45Uh and the generator is passed in here.
28:47So, we are going to always get
28:48deterministic results, the same results.
28:51So, if I run these two,
28:53we're going to get a bunch of samples
28:55from this distribution.
28:57Now, you'll notice here that the
28:58probability for the first element in
29:01this tensor is 60%.
29:04So, in these 20 samples, we'd expect 60%
29:08of them to be zero.
29:10We'd expect 30% of them to be one.
29:14And because the uh element index two
29:17has only 10% probability, very few of
29:20these samples should be two. And indeed,
29:22we only have a small number of twos.
29:25And we can sample as many as we would
29:26like.
29:29And the more we sample, the more uh
29:31these numbers should um roughly have the
29:33distribution here.
29:35So, we should have lots of zeros, half
29:38as many um
29:41ones, and we should have um
29:44three times as few
29:46uh sorry, as few ones, and three times
29:48as few uh
29:50twos.
29:51So, you see that we have very few twos,
29:53we have some ones, and most of them are
29:54zero. So, that's what torch.multinomial
29:57is doing.
29:58For us here,
30:01we are interested in this row. We've
30:02created this um
30:05P here,
30:06and now we can sample from it.
30:09So, if we use the same seed,
30:12and then we sample from this
30:14distribution, let's just get one sample.
30:18Then, we see that the sample is, say,
30:2013.
30:21Um so, this will be the index.
30:25And let's You see how it's a tensor that
30:27wraps 13. We again have to use that item
30:30to pop out that integer.
30:32And now, index would be just the number
30:3513.
30:37And of course, the um we can do we can
30:40map the I2S of IX to figure out exactly
30:43which character we're sampling here.
30:46We're sampling M.
30:48So, we're saying that the first
30:49character is M in our generation.
30:53And just looking at the row here,
30:55M was drawn and you we can see that M
30:57actually starts a large number of words.
31:00M started 2,500 words out of 32,000
31:04words. So, almost
31:06a bit less than 10% of the words start
31:08with M. So, this is actually fairly
31:10likely character to draw.
31:13Um
31:15So, that would be the first character of
31:16our word and now we can continue to
31:18sample more characters because now we
31:20know that M started
31:22M is already sampled.
31:24So, now to draw the next character, we
31:26will come back here and we will look for
31:29the row
31:30that starts with M.
31:32So, you see M
31:34and we have a row here.
31:36So, we see that M.
31:38is
31:39516, MA is this many, MB is this many,
31:43etc. So, these are the counts for the
31:44next row and that's the next character
31:46that we are going to now generate. So, I
31:48think we are ready to actually just
31:50write out the loop because I think
31:51you're starting to get a sense of how
31:52this is going to go.
31:54The um
31:56We always begin at index zero because
31:59that's the start token.
32:02And then while true,
32:04we're going to grab the row
32:06corresponding to index
32:08that we're currently on. So, that's P.
32:11So, that's N array at IX.
32:14Convert it to float is RP.
32:18Then, we normalize this P to sum to one.
32:25Accidentally ran the infinite loop.
32:28We normalize P to sum to one.
32:30Then we need this generator object
32:33that we're going to initialize up here
32:35and we're going to draw a single sample
32:37from this distribution.
32:40And then this is going to tell us what
32:42index is going to be next.
32:46If the index sampled is zero, then
32:49that's now the end token.
32:52So, we will break.
32:55Otherwise, we are going to print
32:57s2i of ix.
33:02i2s of ix.
33:05And uh that's pretty much it. We're just
33:08uh this should work.
33:10Okay, more.
33:12So, that's the that's the name that
33:13we've sampled. We started with M. The
33:16next step was O, then R, and then dot.
33:21And this dot we printed here as well.
33:24So,
33:26let's now do this a few times.
33:28Um
33:30So, let's actually create an
33:33out list here.
33:37And instead of printing, we're going to
33:38append. So, out.append this character.
33:44And then here, let's just print it at
33:46the end. So, let's just join up all the
33:48outs, and we're just going to print
33:50more.
33:51Okay? Now, we're always getting the same
33:53result because of the generator.
33:55So, if you want to do this a few times,
33:56we can go for I in range
34:0010. We can sample 10 names.
34:02And we can just do that 10 times.
34:05And these are the names that we're
34:06getting out.
34:08Let's do 20.
34:14I'll be honest with you, this doesn't
34:15look right.
34:16So, I stared at it a few minutes to
34:17convince myself that it actually is
34:19right.
34:20The reason these samples are so terrible
34:22is that bigram language model
34:24is actually like just like really
34:26terrible.
34:27We can generate a few more here.
34:30And you can see that they're kind of
34:31like their name-like a little bit, like
34:33Keanu, O'Reilly, etc. Uh but they're
34:35just like totally messed up. Um
34:38And I mean, the reason that this is so
34:40bad, like we're generating H as a name.
34:43But you have to think through it from
34:45the model's eyes. It doesn't know that
34:47this H is the very first H. All it knows
34:50is that H was previously, and now how
34:52likely is H the last character? Well,
34:55it's somewhat likely, and so it just
34:58makes it last character. It doesn't know
34:59that there were other things before it
35:01or there were not other things before
35:03it. And so, that's why it's generating
35:05all these like sun- nonsense names.
35:08Another way to do this is
35:12to convince yourself that this is
35:13actually doing something reasonable even
35:14though it's so terrible is
35:17these little piece here are 27, right?
35:20Like 27.
35:23So, how about if we did something like
35:24this?
35:26Instead of P having any structure
35:27whatsoever,
35:28how about if P was just a torch.ones of
35:3127?
35:34of 27.
35:37By default, this is a float 32, so this
35:39is fine. Divide 27.
35:42So, what I'm doing here is this is the
35:45uniform distribution, which will make
35:47everything equally likely.
35:49And we can sample from that. So, let's
35:52see if that does any better.
35:54Okay? So, it's This is what you have
35:56from a model that is completely
35:58untrained, where everything is equally
35:59likely. So, it's obviously garbage. And
36:02then if we have a trained model, which
36:04is trained on just bigrams,
36:07this is what we get. So, you can see
36:08that it is more name-like. It is
36:10actually working. It's just um
36:14bigram is so terrible, and we have to do
36:15better. Now, next I would like to fix an
36:17inefficiency that we have going on here.
36:20Because what we're doing here is we're
36:21always fetching a row of N from the
36:24counts matrix up ahead.
36:26And we we're always doing the same
36:27things. We're converting to float and
36:29we're dividing and we're doing this
36:30every single iteration of this loop and
36:32we just keep re-normalizing these rows
36:34over and over again and it's extremely
36:35inefficient and wasteful.
36:37So, what I'd like to do is I'd like to
36:38actually prepare a matrix capital P that
36:41will just have the probabilities in it.
36:43So, in other words, it's going to be the
36:45same as the capital N matrix here of
36:47counts, but every single row will have
36:49the row of probabilities uh that is
36:51normalized to one indicating the
36:53probability distribution for the next
36:55character given the character before it.
36:58Um as defined by which row we're in.
37:01So, basically what we'd like to do is
37:03we'd like to just do it up front here
37:05and then we would like to just use that
37:06row here.
37:07Uh so, here we would like to just do P
37:10equals P of IX instead.
37:12Okay.
37:14The other reason I want to do this is
37:16not just for efficiency, but also I
37:17would like us to practice uh these
37:19N-dimensional tensors and I'd like us to
37:21practice uh their manipulation and
37:23especially something that's called
37:24broadcasting that we'll go into in a
37:25second.
37:27We're actually going to have to become
37:28very good at these tensor manipulations
37:30because if we're going to build out all
37:32the way to transformers, we're going to
37:33be doing some pretty complicated um
37:35array operations for efficiency and uh
37:37we need to really understand that and be
37:39very good at it.
37:42So, intuitively what we want to do is we
37:43first want to grab the floating point
37:45copy of N.
37:48And I'm mimicking the line here
37:49basically.
37:50And then we want to divide all the rows
37:53so that they sum to one.
37:55So, we'd like to do something like this.
37:57P divide P.sum.
38:00But, now we have to be careful
38:02because P.sum actually
38:05produces a sum
38:08Sorry. P equals N.float.copy.
38:10P.sum produces a um
38:14sums up all of the counts of this entire
38:17matrix N
38:18and gives us a single number of just the
38:19summation of everything. So, that's not
38:21the way we want to defi- divide. We want
38:23to simultaneously and in parallel divide
38:26all the rows by their respective sums.
38:30So, what we have to do now is we have to
38:32go into documentation for torch.sum.
38:35And we can scroll down here to a
38:37definition that is relevant to us, which
38:38is where we don't only provide an input
38:41array that we want to sum, but we also
38:43provide the dimension along which we
38:45want to sum.
38:47And in particular, we want to sum up uh
38:49over rows, right?
38:52Now, one more argument that I want you
38:53to pay attention to here is the keepdim
38:56is false.
38:57If keepdim is true, then the output
39:00tensor is of the same size as input,
39:02except of course the dimension along
39:03which you summed, which will become just
39:05one.
39:07But, if you pass in uh keepdim as false,
39:12then this dimension is squeezed out. And
39:14so, torch.sum not only does the sum and
39:16collapses dimension to be of size one,
39:18but in addition, it does what's called a
39:20squeeze, where it squeeze out it
39:22squeezes out that dimension.
39:24So,
39:25basically what we want here is we
39:27instead want to do p.sum of sum axis.
39:30And in particular, notice that p.shape
39:33is 27 by 27.
39:35So, when we sum up across axis zero,
39:37then we would be taking the zeroth
39:39dimension, and we would be summing
39:40across it.
39:42So, when keepdim is true,
39:45then this thing will not only give us
39:46the counts across um
39:50along the columns,
39:52but notice that basically the shape of
39:53this is 1 by 27. We just get a row
39:56vector.
39:57And the reason we get a row vector here
39:59again is because we pass in zero
40:00dimension, so this zero dimension
40:02becomes one, and we've done a sum,
40:04and we get a row. And so, basically,
40:06we've done the sum
40:08this way,
40:09vertically, and arrived at just a single
40:111 by 27
40:12vector of counts.
40:15What happens when you take out keepdim
40:17is that we just get 27. So, it squeezes
40:20out that dimension and we just get um
40:23a one-dimensional vector of size 27.
40:28Now, we don't actually want
40:311 by 27 row vector because that gives us
40:33the counts or the sums across um
40:37the uh columns.
40:39We actually want to sum the other way
40:41along dimension one.
40:42And you'll see that the shape of this is
40:4427 by one. So, it's a column vector.
40:47It's a 27 by one
40:49vector of counts.
40:52Okay?
40:53And that's because what's happened here
40:55is that we're going horizontally and
40:57this 27 by 27 matrix becomes a 27 by one
41:01array.
41:03Now, you'll notice by the way that um
41:06the actual numbers
41:08of these counts are identical.
41:10And that's because this special array of
41:12counts here comes from bigram
41:14statistics. And actually, it just so
41:15happens by chance or because of the way
41:18this array is constructed that the sums
41:20along the columns or along the rows,
41:23horizontally or vertically, is
41:24identical.
41:26But actually, what we want to do in this
41:27case is we want to sum across the uh
41:30rows
41:31horizontally. So, what we want here is
41:33p.sum of one with keep dim true.
41:3727 by one column vector.
41:39And now, what we want to do is we want
41:40to divide by that.
41:44Now, we have to be careful here again.
41:46Is it possible to take
41:48what's a um p.shape you see here is 27
41:51by 27. Is it possible to take a 27 by 27
41:55array and divide it by what is a 27 by
41:58one array?
42:01Is that an operation that you can do?
42:03And whether or not you can perform this
42:05operation is determined by what's called
42:06broadcasting rules. So, if you just
42:08search broadcasting semantics in torch,
42:12you'll notice that there's a special
42:13definition for uh what's called
42:14broadcasting that for whether or not
42:18these two arrays can be combined in a
42:21binary operation like division.
42:24So the first condition is each tensor
42:25has at least one dimension, which is the
42:27case for us.
42:28And then when iterating over the
42:29dimension sizes starting at the trailing
42:31dimension,
42:32the dimension sizes must either be
42:33equal, one of them is one, or one of
42:35them does not exist.
42:38Okay? So, let's do that. We need to
42:40align the two arrays and their shapes,
42:44which is very easy because both of these
42:45shapes have two elements, so they're
42:46aligned.
42:48Then we iterate over from the from the
42:50right and going to the left.
42:52Each dimension must be either equal, one
42:55of them is a one, or one of them does
42:56not exist.
42:58So in this case they're not equal, but
42:59one of them is a one. So, this is fine.
43:02And then this dimension, they're both
43:03equal. So, this is fine.
43:05So, all the dimensions are fine and
43:08therefore the this operation is
43:10broadcastable.
43:12So that means that this operation is
43:13allowed.
43:14And what is it that these arrays do when
43:16you divide 27 by 27 by 27 by one?
43:19What it does is that it takes this
43:21dimension one and it stretches it out.
43:24It copies it
43:25to match 27 here in this case.
43:29So in our case, it takes this column
43:30vector which is 27 by one
43:32and it copies it 27 times
43:36to make
43:37these both be 27 by 27 internally. You
43:40can think that way. And so it copies
43:42those counts
43:44and then it does an element-wise
43:45division,
43:47which is what we want because these
43:48counts we want to divide by them on
43:50every single one of these columns in
43:53this matrix.
43:54So this actually we expect will
43:56normalize every single row.
43:59And we can check that this is true by
44:01taking the first row for example and
44:04taking its sum. We expect this to be
44:06one.
44:08Because it's now normalized.
44:10And then we expect this now, because if
44:13we actually correctly normalize all the
44:14rows, we expect to get the exact same
44:16result here. So, let's run this.
44:19It's the exact same result.
44:21So, this is correct. So, now I would
44:23like to scare you a little bit. Uh you
44:25actually have to like I basically
44:27encourage you very strongly to read
44:28through broadcasting semantics.
44:30And I encourage you to treat this with
44:31respect. And it's not something to play
44:34fast and loose with. It's something to
44:35really respect, really understand, and
44:37look up maybe some tutorials for
44:39broadcasting and practice it, and be
44:40careful with it because you can very
44:42quickly run into bugs. Let me show you
44:44what I mean.
44:47You see how here we have p.sum of one,
44:49keep dims is true.
44:50The shape of this is 27 by one.
44:53Let me take out this line just so we
44:54have the n, and then we can see the
44:57counts.
44:58We can see that this is all the counts
45:00across all the
45:02rows.
45:03And it's a 27 by one column vector,
45:05right?
45:07Now, suppose that I tried to do the
45:09following,
45:10but I erase keep dims is true here.
45:14What does that do? If keep dims is not
45:15true, it's false, then remember,
45:17according to documentation, it gets rid
45:19of this dimension one. It squeezes it
45:22out. So, basically we just get all the
45:24same counts, the same result, except the
45:27shape of it is not 27 by one, it is just
45:2927. The one disappears.
45:31But all the counts are the same.
45:34So, you think that this divide that
45:37would would work.
45:40First of all, can we even write this?
45:42And will it Is it even Is it even
45:44expected to run? Is it broadcastable?
45:46Let's determine if this result is
45:47broadcastable.
45:49p.sum at one is shape
45:51is 27.
45:53This is 27 by 27. So, 27 by 27
45:57broadcasting into 27.
46:00So, now
46:01rules of broadcasting, number one align
46:03all the dimensions on the right done now
46:06iteration over all the dimensions
46:07starting from the right going to the
46:09left
46:10all the dimensions must either be equal
46:13one of them must be one or one of them
46:14does not exist so here they are all
46:16equal here the dimension does not exist
46:20so internally what broadcasting will do
46:21is it will create a one here
46:24and then
46:25we see that one of them is a one and
46:27this will get copied and this will run
46:30this will broadcast
46:32okay so you'd expect this to work
46:37because we we are
46:41this broadcasting this we can divide
46:42this
46:43now if I run this you'd expect it to
46:45work but
46:46it doesn't
46:48you actually get garbage you get a wrong
46:49result because this is actually a bug
46:52this keep them equals true
46:57makes it work
47:00this is a bug in
47:03both cases we are doing the correct
47:05counts we are summing up across the rows
47:09but keep them is saving us and making it
47:10work so in this case
47:12I'd like you to encourage you to
47:13potentially like pause this video at
47:15this point and try to think about why
47:17this is buggy and why the keep them was
47:19necessary here
47:22okay
47:23so the reason to do
47:24for this is I'm trying to hint it here
47:26when I was sort of giving you a bit of a
47:27hint on how this works
47:29this
47:3027 vector
47:32internally inside the broadcasting this
47:34becomes a one by 27
47:36and one by 27 is a row vector right and
47:39now we are dividing 27 by 27 by one by
47:4227
47:43and torch will replicate this dimension
47:45so basically
47:47it will take
47:49it will take this
47:51row vector and it will copy it
47:53vertically now
47:5527 times so the 27 by 27 lines exactly,
47:58and element twice divides.
48:00And so, basically, what's happening here
48:02is
48:03um
48:04we're actually normalizing the columns
48:06instead of normalizing the rows.
48:09So, you can check that what's happening
48:11here is that P at zero, which is the
48:13first row of P, dot sum is not one, it's
48:17seven.
48:18It is the first column, as an example,
48:20that sums to one.
48:23So,
48:24to summarize, where does the issue come
48:26from? The issue comes from the silent
48:28adding of a dimension here because in
48:30broadcasting rules, you align on the
48:31right and go from right to left, and if
48:34dimension doesn't exist, you create it.
48:36So, that's where the problem happens. We
48:38still did the counts correctly. We did
48:39the counts across the rows, and we got
48:41the the counts on the right here as a
48:44column vector. But, because the keepdims
48:46was true, this this uh this dimension
48:48was discarded, and now we just have a
48:50vector 27.
48:51And because of broadcasting the way it
48:53works, this vector of 27 suddenly
48:55becomes a row vector.
48:57And then this row vector gets replicated
48:58vertically, and then every single point
49:01we are dividing by the by the count
49:05uh in the opposite direction.
49:07So, uh
49:08so, this thing just uh doesn't work. You
49:11This needs to be keepdims equals true in
49:13this case.
49:14So, then
49:15um then we have that P at zero is
49:17normalized.
49:19And conversely, the first column, you'd
49:21expect to potentially not be normalized.
49:24And this is what makes that work.
49:27So, pretty subtle, and uh hopefully this
49:31helps to scare you that you should have
49:33a respect for broadcasting, be careful,
49:35check your work,
49:36uh and uh understand how it works under
49:38the hood, and make sure that it's
49:39broadcasting in the direction that you
49:40like. Otherwise, you're going to
49:41introduce very subtle bugs, very hard to
49:43hard to find bugs, and uh just be
49:46careful. One more note on efficiency, we
49:48don't want to be doing this here because
49:50uh this creates a completely new tensor
49:52that we store into P. We prefer to use
49:55in-place operations if possible.
49:57Also, this would be an in-place
49:59operation. It has the potential to be
50:01faster. It doesn't create new memory
50:03under the hood. And then let's erase
50:05this. We don't need it.
50:07And let's also
50:10um just do fewer just so I'm not wasting
50:13space.
50:14Okay, so we're actually in a pretty good
50:15spot now.
50:17We trained a bigram language model, and
50:19we trained it really just by counting
50:21uh how frequently any pairing occurs and
50:24then normalizing so that we get a nice
50:26probability distribution.
50:28So really these elements of this array P
50:31are really the parameters of our bigram
50:32language model giving us and summarizing
50:34the statistics of these bigrams.
50:37So we trained the model and then we know
50:38how to sample from the model. We just
50:40iteratively uh sample the next character
50:43and uh feed it in each time and get the
50:45next character.
50:47Now what I'd like to do is I'd like to
50:48somehow evaluate the quality of this
50:50model.
50:51We'd like to somehow summarize the
50:53quality of this model into a single
50:54number. How good is it at predicting uh
50:57the training set?
50:59And as an example, so in the training
51:00set we can evaluate now the training
51:03loss. And this training loss is telling
51:05us about uh sort of the quality of this
51:07model in a single number just like we
51:09saw in micrograd.
51:11So let's try to think through the
51:13quality of the model and how we would
51:14evaluate it.
51:17Basically what we're going to do is
51:18we're going to copy paste this code
51:20that we previously used for counting.
51:23Okay?
51:24And let me just print the bigrams first.
51:26We're going to use F-strings.
51:27And I'm going to print character one
51:29followed by character two. These are the
51:31bigrams. And then I don't want to do it
51:32for all the words. Let's just do the
51:34first three words.
51:36So here we have Emma, Olivia, and Ava
51:38bigrams.
51:40Now what we'd like to do is we'd like to
51:41basically look at the probability that
51:44the model assigns to every one of these
51:46bigrams.
51:48So, in other words, we can look at the
51:49probability, which is summarized in the
51:51matrix P,
51:53of IX1 IX2.
51:56And then we can print it here as
51:58probability.
52:00And because these probabilities are way
52:01too large, let me percent uh
52:04or colon point four F
52:06to like truncate it a bit.
52:09So, what do we have here, right? We're
52:10looking at the probabilities that the
52:11model assigns to every one of these
52:13bigrams in the data set.
52:15And so, we can see some of them are 4%,
52:173%, etc. Just to have a measuring stick
52:19in our mind, by the way, um
52:21with 27 possible characters or tokens,
52:24and if everything was equally likely,
52:26then you'd expect all these
52:27probabilities to be
52:304% roughly.
52:32So, anything above 4% means that we've
52:34learned something useful from these
52:36bigram statistics.
52:37And you see that roughly some of these
52:38are 4%, but some of them are as high as
52:4040%,
52:4135%, and so on. So, you see that the
52:44model actually assigned a pretty high
52:45probability to whatever's in the
52:47training set. And so, that that's a good
52:49thing.
52:50Um basically, if you have a very good
52:51model, you'd expect that these
52:53probabilities should be near one,
52:54because that means that uh your model is
52:56correctly predicting what's going to
52:57come next, especially in the training
52:59set, where you where you trained your
53:01model.
53:02So, now we'd like to think about how can
53:05we summarize these probabilities into a
53:07single number that measures the quality
53:09of this model.
53:11Now, when you look at the literature
53:13into maximum likelihood estimation and
53:15statistical modeling and so on,
53:17you'll see that what's typically used
53:18here is something called the likelihood.
53:21And the likelihood is the product of all
53:23of these probabilities.
53:25And so, the product of all of these
53:27probabilities is the likelihood, and
53:29it's really telling us about the
53:31probability of the entire data set
53:33assigned uh assigned by the model that
53:36we've trained. And that is a measure of
53:38quality.
53:39So, the product of these should be as
53:41high as possible
53:43when you are training the model and when
53:44you have a good model, your your product
53:46of these probabilities should be very
53:48high.
53:49Um
53:50Now, because the product of these
53:51probabilities is an unwieldy thing to
53:53work with, uh you can see that all of
53:55them are between 0 and 1. So, your
53:56product of these probabilities will be a
53:58very tiny number.
54:00Um so, for convenience, what people work
54:03with usually is not the likelihood, but
54:04they work with what's called the log
54:06likelihood.
54:07So,
54:09the product of these is likelihood.
54:10To get likelihood, we just have to take
54:13the log of the probability.
54:15And so, the log of the probability here,
54:17I have a log of X from 0 to 1.
54:19The log is a you see here monotonic
54:21transformation of the probability,
54:24where if you pass in 1, you get 0.
54:28So, probability 1 gets you log
54:30probability of 0.
54:32And then, as you go lower and lower
54:33probability, the log will grow more and
54:35more negative until all the way to
54:37negative infinity at 0.
54:41So, here we have a log prob, which is
54:44really just a torch. log of probability.
54:46Let's print it out to get a sense of
54:48what that looks like.
54:50Log prob,
54:51also point four if.
54:54Okay.
54:56So, as you can see, when we plug in
54:58numbers that are very close, some of our
55:00higher numbers, we get closer and closer
55:02to 0.
55:03And then, if we plug in very bad
55:05probabilities, we get more and more
55:06negative number. That's bad.
55:09So,
55:10and the reason we work with this is for
55:13large extent convenience, right? Because
55:15we have mathematically that if you have
55:17some product A * B * C of all these
55:19probabilities, right? Or the likelihood
55:22is the product of all these
55:23probabilities,
55:25then the log
55:27of these is just log of A plus log of B
55:33plus log of C. If you remember your logs
55:36from your
55:37high school or undergrad and so on.
55:39So we have that basically
55:41the likelihood is the product of
55:42probabilities, the log likelihood is
55:44just the sum of the logs of the
55:46individual probabilities.
55:48So
55:49log likelihood
55:52starts at zero.
55:54And then log likelihood here we can just
55:57accumulate simply.
56:00And in the end we can print this.
56:05Print the log likelihood.
56:09F strings.
56:11Maybe you're familiar with this.
56:13So log likelihood is -38.
56:19Okay.
56:21Now
56:22we actually want um
56:25So how high can log likelihood get? It
56:27can go to zero. So when all the
56:30probabilities are one, log likelihood
56:31will be zero. And then when all the
56:33probabilities are lower, this will grow
56:35more and more negative.
56:37Now we don't actually like this because
56:39what we'd like is a loss function. And a
56:41loss function has the semantics that low
56:44is good. Because we're trying to
56:46minimize the loss. So we actually need
56:49to invert this. And that's what gives us
56:51something called the negative log
56:53likelihood.
56:54Um
56:55negative log likelihood is just negative
56:58of the log likelihood.
57:03These are F strings by the way if you'd
57:05like to look this up.
57:06Negative log likelihood equals
57:09So negative log likelihood now is just
57:10the negative of it.
57:12And so the negative log likelihood is a
57:13very nice loss function because um
57:17the lowest it can get is zero. And the
57:20higher it is, the worse off the
57:22predictions are that you're making.
57:24And then one more modification to this
57:26that sometimes people do is that for
57:27convenience uh they actually like to
57:29normalize by they like to make it an
57:32average instead of a sum.
57:34And so here
57:37let's just keep some counts as well.
57:39So n plus equals one starts at zero. And
57:42then here we can have sort of like a
57:45normalized log likelihood.
57:47Um
57:50if we just normalize it by the count
57:52then we will sort of get the average log
57:54likelihood. So this would be usually our
57:57loss function here.
57:58This this we would this is what we would
58:00use.
58:02Uh so our loss function for the training
58:03set assigned by the model is 2.4. That's
58:06the quality of this model.
58:08And the lower it is, the better off we
58:10are. And the higher it is, the worse off
58:12we are.
58:13And the job of our, you know, training
58:16is to find the parameters that minimize
58:19the negative log likelihood loss.
58:22And that would be like a high quality
58:24model. Okay, so to summarize I actually
58:26wrote it out here.
58:28So our goal is to maximize likelihood
58:30which is the product of all the
58:32probabilities assigned by the model.
58:35And we want to maximize this likelihood
58:37with respect to the model parameters.
58:39And in our case, the model parameters
58:41here are defined in the table. These
58:43numbers, the probabilities
58:45are
58:46uh the model parameters sort of in our
58:47bigram language model so far.
58:50Uh but you have to keep in mind that
58:51here we are storing everything in a
58:52table format, the probabilities. But
58:54what's coming up as a brief preview is
58:57that these numbers will not be kept
58:59explicitly but these numbers will be
59:01calculated by a neural network.
59:03So that's coming up.
59:04And we want to change and tune the
59:06parameters of these neural networks. We
59:08want to change these parameters to
59:09maximize the likelihood, the product of
59:11the probabilities.
59:13Now maximizing the likelihood is
59:15equivalent to maximizing the log
59:16likelihood because log is a monotonic
59:18function.
59:19Here's the graph of log.
59:22And basically all it is doing is it's uh
59:24just a scaling your um you can look at
59:27it as just a scaling of the loss
59:28function.
59:29And so the optimization problem here and
59:32here are actually equivalent because
59:34this is just a scaling. You can look at
59:35it that way.
59:37And so these are two identical
59:38optimization problems.
59:41Um
59:41maximizing the log likelihood is
59:43equivalent to minimizing the negative
59:44log likelihood.
59:46And then in practice people actually
59:47minimize the average negative log
59:49likelihood to get numbers like 2.4.
59:53And then this summarizes the quality of
59:55your model.
59:56And we'd like to minimize it and make it
59:57as small as possible.
59:59And the lowest it can get is zero.
1:00:02And the lower it is
1:00:04the better off your model is because
1:00:06it's assigning it's assigning high
1:00:07probabilities to your data.
1:00:09Now let's estimate the probability over
1:00:10the entire training set just to make
1:00:12sure that we get something around 2.4.
1:00:15Let's run this over the entire oops.
1:00:17Let's take out the print statement as
1:00:18well.
1:00:20Okay, 2.45 over the entire training set.
1:00:24Now what I'd like to show you is that
1:00:25you can actually evaluate the
1:00:26probability for any word that you want.
1:00:28Like for example
1:00:30if we just test a single word Andre and
1:00:32bring back the print statement
1:00:35then you see that Andre is actually kind
1:00:37of like an unlikely word. Like on
1:00:39average
1:00:40um we take three log probability to
1:00:43represent it. And roughly that's because
1:00:45EJ apparently is very uncommon as an
1:00:47example.
1:00:50Now
1:00:51think through this um
1:00:53when I take Andre and I append Q and I
1:00:55test the probability of it Andre Q
1:01:00we actually get um infinity.
1:01:03And that's because JQ has a 0%
1:01:05probability according to our model. So
1:01:07the log likelihood
1:01:09so the log of zero will be negative
1:01:11infinity. We get infinite loss.
1:01:14So this is kind of undesirable, right?
1:01:15Because we plugged in a string that
1:01:16could be like a somewhat reasonable
1:01:18name. But basically what this is saying
1:01:20is that this model is exactly 0% likely
1:01:23to uh to predict this name
1:01:26and our loss is infinity on this
1:01:28example.
1:01:29And really what the reason for that is
1:01:31that J
1:01:32is followed by Q
1:01:35uh zero times. Uh where is Q? JQ is zero
1:01:39and so JQ is 0% likely.
1:01:42So it's actually kind of gross and
1:01:43people don't like this too much. To fix
1:01:45this, there's a very simple fix that
1:01:47people like to do to sort of like smooth
1:01:49out your model a little bit that is
1:01:50called model smoothing.
1:01:52And roughly what's happening is that we
1:01:53will we will add some fake counts.
1:01:56So imagine adding a count of one to
1:01:59everything.
1:02:00So we add a count of one
1:02:03like this
1:02:04and then we recalculate the
1:02:05probabilities.
1:02:07And that's model smoothing and you can
1:02:09add as much as you like. You can add
1:02:10five and that will give you a smoother
1:02:11model.
1:02:12And the more you add here
1:02:14the more uniform model you're going to
1:02:16have. And the less you add,
1:02:19um the more more peaked model you're
1:02:21going to have, of course.
1:02:22So one is like a pretty decent count to
1:02:24add and that will ensure that there will
1:02:27be no zeros in our probability matrix P.
1:02:30And so this will of course change the
1:02:32generations a little bit. In this case
1:02:34it didn't but it in principle it could.
1:02:36But what that's going to do now is that
1:02:38nothing will be infinity unlikely.
1:02:41So now
1:02:42our model will predict some other
1:02:44probability and we see that JQ now has a
1:02:46very small probability. So the model
1:02:48still finds it very surprising that this
1:02:49was a word or bigram but we don't get
1:02:52negative infinity.
1:02:53So it's kind of like a nice fix that
1:02:54people like to apply sometimes and it's
1:02:55called model smoothing. Okay, so we've
1:02:57now trained a respectable bigram
1:03:00character level language model and we
1:03:01saw that we both
1:03:04a sort of trained the model by looking
1:03:05at the counts of all the bigrams and
1:03:08normalizing the rows to get probability
1:03:10distributions.
1:03:11We saw that we can also then use those
1:03:14parameters of this model to perform
1:03:16sampling of new words.
1:03:19So, we sample new names according to
1:03:21those distributions. And we also saw
1:03:22that we can evaluate the quality of this
1:03:24model.
1:03:25And the quality of this model is
1:03:26summarized in a single number, which is
1:03:28the negative log likelihood. And the
1:03:30lower this number is, the better the
1:03:32model is
1:03:33because it is giving high probabilities
1:03:35to the actual next characters in all the
1:03:37bigrams in our training set.
1:03:40So, that's all well and good. But we've
1:03:42arrived at this model explicitly by
1:03:44doing something that felt sensible. We
1:03:46were just performing counts, and then we
1:03:48were normalizing those counts.
1:03:51Now, what I would like to do is I would
1:03:52like to take an alternative approach. We
1:03:54will end up in a very, very similar
1:03:55position, but the approach will look
1:03:57very different because I would like to
1:03:58cast the problem of bigram character
1:04:00level language modeling into the neural
1:04:02network framework.
1:04:04And in the neural network framework,
1:04:05we're going to approach things slightly
1:04:07differently, but again, end up in a very
1:04:09similar spot. I'll go into that later.
1:04:12Now, our neural network is going to be a
1:04:15still a bigram character level language
1:04:16model. So, it receives a single
1:04:18character as an input.
1:04:20Then there's neural network with some
1:04:21weights or some parameters W.
1:04:24And it's going to output the probability
1:04:26distribution over the next character in
1:04:28a sequence. It's going to make guesses
1:04:30as to what is likely to follow this
1:04:32character that was input to the model.
1:04:36And then in addition to that, we're
1:04:37going to be able to evaluate any setting
1:04:39of the parameters of the neural net
1:04:41because we have the loss function.
1:04:43The negative log likelihood. So, we're
1:04:45going to take a look at these
1:04:46probability distributions, and we're
1:04:47going to use the labels
1:04:49which are basically just the identity of
1:04:51the next character in that bigram, the
1:04:53second character.
1:04:54So, knowing what second character
1:04:56actually comes next in the bigram allows
1:04:58us to then look at what how high of a
1:05:00probability the model assigns to that
1:05:02character.
1:05:03And then we of course want the
1:05:05probability to be very high.
1:05:07And that is another way of saying that
1:05:08the loss is low.
1:05:10So, we're going to use gradient-based
1:05:12optimization then to tune the parameters
1:05:14of this network because we have the loss
1:05:16function and we're going to minimize it.
1:05:18So, we're going to tune the weights so
1:05:20that the neural net is correctly
1:05:21predicting the probabilities for the
1:05:23next character.
1:05:24So, let's get started. The first thing I
1:05:26want to do is I want to compile the
1:05:27training set of this neural network,
1:05:29right? So, create
1:05:31the training set
1:05:33of all the bigrams.
1:05:36Okay?
1:05:37And
1:05:39here
1:05:40I'm going to copy-paste this code
1:05:43because this code iterates over all the
1:05:45bigrams.
1:05:47So, here we start with the words. We
1:05:49iterate over all the bigrams. And
1:05:50previously, as you recall, we did the
1:05:52counts. But now we're not going to do
1:05:54counts. We're just creating a training
1:05:55set.
1:05:56Now, this training set will be made up
1:05:58of two lists.
1:06:02We have the
1:06:04inputs
1:06:06and the targets, the the labels.
1:06:09And these bigrams will denote XY. Those
1:06:11are the characters, right?
1:06:13And so, we're given the first character
1:06:14of the bigram and then we're trying to
1:06:16predict the next one.
1:06:17Both of these are going to be integers.
1:06:19So, here we'll take X's. data.append is
1:06:22just
1:06:23X1. Y's.data.append IX2.
1:06:27And then here
1:06:29we actually don't want lists of
1:06:30integers. We will create uh tensors out
1:06:33of these. So, X's is torch.tensor
1:06:36X's and Y's is torch.tensor of Y's.
1:06:41And then we don't actually want to take
1:06:43all the words just yet because I want
1:06:45everything to be manageable. Uh so,
1:06:47let's just do the first word, which is
1:06:48Emma.
1:06:51And then it's clear what these X's and
1:06:52Y's would be.
1:06:55Here, let me print
1:06:57character one, character two, just so
1:06:59you see what's going on here.
1:07:01So, the bigrams of these characters is
1:07:04.e, em, mm, ma, a. So, so this single
1:07:09word, as I mentioned, has 1 2 3 4 5
1:07:12examples for our neural network.
1:07:14There are five separate examples in
1:07:16Emma.
1:07:17And those examples are summarized here.
1:07:19When the input to the neural neural
1:07:20network is integer zero,
1:07:23the desired label is integer five, which
1:07:26corresponds to e.
1:07:28When the input to the neural network is
1:07:29five, we want its weights to be arranged
1:07:32so that 13 gets a very high probability.
1:07:35When 13 is put in, we want 13 to have a
1:07:37high probability.
1:07:39When 13 is put in, we also want one to
1:07:41have a high probability.
1:07:43When one is input, we want zero to have
1:07:45a very high probability. So, there are
1:07:47five separate input examples to a neural
1:07:50net
1:07:51in this data set.
1:07:55I wanted to add a tangent of a note of
1:07:57caution to be careful with a lot of the
1:07:59APIs of some of these frameworks.
1:08:01You saw me silently use torch.Tensor
1:08:04with a lowercase t, and the output
1:08:06looked right.
1:08:07But, you should be aware that there's
1:08:09actually two ways of constructing a
1:08:10tensor. There's a torch.lowercase
1:08:13tensor, and there's also a torch.capital
1:08:15tensor class, which you can also
1:08:17construct. Uh so, you can actually call
1:08:19both. You can also do torch.capital
1:08:21tensor,
1:08:22and you get an x's and y's as well.
1:08:25So, that's not confusing at all.
1:08:27Um
1:08:29There are threads on what is the
1:08:29difference between these two.
1:08:31And um
1:08:33unfortunately, the docs are just like
1:08:34not clear on the difference. And when
1:08:36you look at the the docs of lowercase
1:08:38tensor, constructs tensor with no
1:08:40autograd history by copying data.
1:08:43It's just like it doesn't
1:08:45it doesn't make sense. So, the actual
1:08:47difference, as far as I can tell, is
1:08:48explained eventually in this random
1:08:50thread that you can Google.
1:08:51And really, it comes down to, I believe,
1:08:55that um
1:08:56where is this?
1:08:58torch.Tensor infers the dtype, the data
1:09:00type, automatically, while torch.Tensor
1:09:02just returns a float tensor.
1:09:04I would recommend stick to torch.lower
1:09:06case tensor.
1:09:07So, um
1:09:09indeed, we see that when I construct
1:09:12this with a capital T, the data type
1:09:14here of X's is float 32.
1:09:18But, torch.lower case tensor
1:09:21you see how it's now X.dtype is now
1:09:24integer.
1:09:26So, um
1:09:28it's advised that you use lower case T
1:09:30and you can read more about it if you
1:09:32like in some of these threads.
1:09:34Uh but basically
1:09:35um
1:09:36I'm pointing out some of these things
1:09:37beca- because I want to caution you and
1:09:39I want you to re- get used to reading a
1:09:41lot of documentation and reading through
1:09:43a lot of uh Q&A's and threads like this.
1:09:46And um
1:09:48you know, some of this stuff is
1:09:49unfortunately not easy and not very well
1:09:50documented and you have to be careful
1:09:51out there. What we want here is integers
1:09:54because that's what makes uh sense. Um
1:09:58and so uh lower case tensor is what we
1:10:00are using. Okay, now we want to think
1:10:02through how we're going to feed in these
1:10:03examples into a neural network.
1:10:06Now, it's not quite as straightforward
1:10:07as n-
1:10:09plugging it in because these examples
1:10:11right now are integers. So, there's like
1:10:12a 0, 5, or 13. It gives us the index of
1:10:15the character and you can't just plug an
1:10:17integer index into a neural net.
1:10:20These neural nets uh right are sort of
1:10:22made up of these neurons.
1:10:24And uh these neurons have weights. And
1:10:27as you saw in micrograd, these weights
1:10:29act multiplicatively on the inputs. WX
1:10:31plus B, there's tanh's and so on. And
1:10:34so, it doesn't really make sense to make
1:10:35an input neuron take on integer values
1:10:37that you feed in and then multiply on
1:10:40with weights.
1:10:41So, instead, a common way of encoding
1:10:44integers is what's called one-hot
1:10:45encoding.
1:10:47In one-hot encoding, uh we take an
1:10:49integer like 13 and we create a vector
1:10:52that is all zeros except for the 13th
1:10:54dimension, which we turn to a one.
1:10:57And then that vector can feed into a
1:10:59neural net.
1:11:01Now, conveniently,
1:11:03uh PyTorch actually has something called
1:11:04the one-hot uh uh
1:11:07um
1:11:07function inside torch.nn.functional.
1:11:10It takes a tensor made up of integers.
1:11:13Um
1:11:14long is a is a is an integer.
1:11:18Um
1:11:19and it also takes a number of classes,
1:11:21um
1:11:22which is how large you want your uh
1:11:24tensor uh your vector to be.
1:11:27So here, let's import is a common way of
1:11:30importing it.
1:11:34And then let's do F.one_hot.
1:11:36And we feed in uh the integers that we
1:11:38want to encode.
1:11:40So we can actually feed in the entire
1:11:41array of X's.
1:11:44And we can tell it that num_classes is
1:11:4627.
1:11:47So it doesn't have to try to guess it.
1:11:49It may have guessed that it's only 13
1:11:51and would give us an incorrect result.
1:11:54So this is the one-hot. Let's call this
1:11:56X_enc for X encoded.
1:12:02And then we see that X_encoded.shape is
1:12:045 by 27.
1:12:07And uh
1:12:08we can also visualize it, plt.imshow of
1:12:10X_enc,
1:12:12to make it a little bit more clear
1:12:13because this is a little messy.
1:12:15So we see that we've encoded all the
1:12:17five examples uh into vectors.
1:12:20We have five examples, so we have five
1:12:22rows, and each row here is now an
1:12:24example into a neural net.
1:12:26And we see that the appropriate bit is
1:12:28turned on as a one, and everything else
1:12:30is zero.
1:12:31So um
1:12:33here for example, the the zeroth bit is
1:12:35turned on, the fifth bit is turned on,
1:12:3813th bits are turned on for both of
1:12:40these examples, and then the first bit
1:12:42here is turned on.
1:12:44So that's how we can encode um integers
1:12:47into vectors.
1:12:49And then these vectors can feed in to
1:12:51neural nets. One more issue to be
1:12:52careful with here, by the way, is
1:12:55let's look at the data type of X
1:12:56encoding. We always want to be careful
1:12:58with data types.
1:12:59What would you expect X encoding's data
1:13:01type to be? When we're plugging numbers
1:13:03into neural nets, we don't want them to
1:13:05be integers. We want them to be floating
1:13:07point numbers that can take on various
1:13:09values. But the D type here is actually
1:13:1264-bit integer.
1:13:14And the reason for that, I suspect, is
1:13:15that one hot received a 64-bit integer
1:13:18here and it returned to the same data
1:13:21type.
1:13:21And when you look at the signature of
1:13:23one hot, it doesn't even take a D type,
1:13:25a desired data type of the output
1:13:27tensor.
1:13:28And so we can't In a lot of functions in
1:13:30torch, we'd be able to do something like
1:13:32D type equals torch.float32,
1:13:34which is what we want, but one hot does
1:13:36not support that.
1:13:37So instead, we're going to want to cast
1:13:39this to float like this.
1:13:43So that these
1:13:44everything is the same.
1:13:46Everything looks the same, but the D
1:13:48type is float 32. And floats can feed
1:13:51into um neural nets. So now let's
1:13:53construct our first neuron.
1:13:56This neuron will look at these input
1:13:58vectors.
1:14:00And as you remember from micrograd,
1:14:02these neurons basically perform a very
1:14:03simple function, wx plus b, where wx is
1:14:06a dot product, right?
1:14:09So we can achieve the same thing here.
1:14:12Let's first define the weights of this
1:14:14neuron. Basically, where are the initial
1:14:15weights at initialization for this
1:14:17neuron?
1:14:18Let's initialize them with torch.randn.
1:14:21torch.randn
1:14:23is um
1:14:24fills a tensor with random numbers
1:14:27drawn from a normal distribution.
1:14:29And a normal distribution has a
1:14:32probability uh density function like
1:14:33this. And so most of the numbers drawn
1:14:35from this distribution will be around
1:14:37zero,
1:14:38uh but some of them will be as high as
1:14:40almost three and so on. And very few
1:14:42numbers will be above three in
1:14:44magnitude.
1:14:46So we need to take a size as an input
1:14:49here.
1:14:50And I'm going to use size as 27 by 1.
1:14:54So, 27 by 1, and then let's visualize W.
1:14:58So, W is a column vector of 27 numbers.
1:15:03And uh these weights are then multiplied
1:15:06by the inputs.
1:15:08So, now to perform this multiplication,
1:15:10we can take X encoding
1:15:12and we can multiply it with W.
1:15:15This is a matrix multiplication operator
1:15:17in PyTorch.
1:15:20And the output of this operation is 5 by
1:15:221.
1:15:23The reason it's 5 by 1 is the following.
1:15:25We took X encoding, which is 5 by 27,
1:15:29and we multiplied it by 27 by 1.
1:15:33And
1:15:34in matrix multiplication,
1:15:36you see that the output will become 5 by
1:15:391 because these 27 will multiply and
1:15:43add.
1:15:44So, basically what we're seeing here,
1:15:46out out of this operation,
1:15:48is we are seeing the five um
1:15:51activations
1:15:53of this neuron
1:15:56on these five inputs. And we've
1:15:58evaluated all of them in parallel. We
1:16:00didn't feed in just a single input to
1:16:02this single neuron. We fed in
1:16:04simultaneously all the five inputs into
1:16:06the same neuron,
1:16:08and in parallel, PyTorch has evaluated
1:16:11the WX plus B, but here it's just WX.
1:16:14There's no bias.
1:16:15It has valued W W times X for all of
1:16:18them uh independently. Now, instead of a
1:16:21single neuron though, I would like to
1:16:22have 27 neurons, and I'll show you in a
1:16:24second why I want 27 neurons.
1:16:27So, instead of having just a one here,
1:16:29which is indicating this presence of one
1:16:31single neuron,
1:16:32we can use 27.
1:16:34And then when W is 27 by 27,
1:16:38this will in parallel evaluate all the
1:16:4127 neurons on all the five inputs.
1:16:46Giving us a much better, much much
1:16:48bigger result.
1:16:49So, now what we've done is 5 by 27
1:16:51multiplied 27 by 27.
1:16:54And the output of this is now 5 by 27.
1:16:57So, we can see that the shape of this
1:17:01is 5 by 27.
1:17:04So, what is every element here telling
1:17:05us, right?
1:17:07It's telling us for every one of 27
1:17:09neurons that we created,
1:17:13what is the firing rate of those neurons
1:17:16on every one of those five examples?
1:17:19So,
1:17:20the element, for example, 3 {comma} 13
1:17:25is giving us the firing rate of the 13th
1:17:28neuron looking at the third input.
1:17:31And the way this was achieved is by a
1:17:34dot product
1:17:36between the third input
1:17:38and the 13th column
1:17:41of this W matrix here.
1:17:44Okay? So, using matrix multiplication,
1:17:47we can very efficiently evaluate
1:17:50the dot product between lots of input
1:17:52examples in a batch.
1:17:55And lots of neurons, where all of those
1:17:57neurons have weights in the columns of
1:17:59those Ws.
1:18:01And in matrix multiplication, we're just
1:18:02doing those dot products and
1:18:04in parallel. Just to show you that this
1:18:06is the case, we can take X and we can
1:18:08take the third
1:18:10row.
1:18:12And we can take the W and take its 13th
1:18:14column.
1:18:17And then we can do X and get three.
1:18:21Element-wise multiply with W at 13.
1:18:26And sum that up. That's WX plus B.
1:18:29Uh well, there's no plus B. It's just WX
1:18:31dot product. And that's
1:18:34this number.
1:18:35So, you see that this is just being done
1:18:36efficiently by the matrix multiplication
1:18:39operation for all the input examples and
1:18:42for all the output neurons of this first
1:18:45layer.
1:18:46Okay, so we fed our 27-dimensional
1:18:48inputs into a first layer of a neural
1:18:50net that has 27 neurons, right? So, we
1:18:53have 27 inputs and now we have 27
1:18:56neurons. These neurons perform W * X.
1:18:59They don't have a bias and they don't
1:19:01have a non-linearity like tanh. We're
1:19:03going to leave them to be a linear
1:19:05layer.
1:19:06In addition to that, we're not going to
1:19:08have any other layers. This is going to
1:19:09be it. It's just going to be
1:19:11the dumbest, smallest, simplest neural
1:19:13net, which is just a single linear
1:19:14layer.
1:19:16And now I'd like to explain what I want
1:19:18those 27 outputs to be.
1:19:21Intuitively, what we're trying to
1:19:22produce here for every single input
1:19:23example is we're trying to produce some
1:19:25kind of a probability distribution for
1:19:27the next character in a sequence. And
1:19:29there's 27 of them.
1:19:31But we have to come up with like precise
1:19:33semantics for exactly how we're going to
1:19:34interpret these 27 numbers that these
1:19:37neurons take on.
1:19:39Now, intuitively,
1:19:41you see here that these numbers are
1:19:42negative and some of them are positive,
1:19:44etc.
1:19:45And that's because these are coming out
1:19:46of a neural net layer initialized with
1:19:48these um
1:19:50uh normal distribution
1:19:52uh parameters.
1:19:54But what we want is we want something
1:19:55like we had here. Like each row here
1:19:59told us the counts and then we
1:20:01normalized the counts to get
1:20:02probabilities. And we want something
1:20:04similar to come out of a neural net.
1:20:06But what we just have right now is just
1:20:07some negative and positive numbers.
1:20:10Now, we want those numbers to somehow
1:20:12represent the probabilities for the next
1:20:14character.
1:20:15But you see that probabilities, they
1:20:17they have a special structure. They um
1:20:20they're positive numbers and they sum to
1:20:21one.
1:20:22And so, that doesn't just come out of a
1:20:24neural net.
1:20:25And then, they can't be counts because
1:20:28uh these counts are positive and counts
1:20:31are integers.
1:20:32So, counts are also not really a good
1:20:34thing to output from a neural net.
1:20:36So, instead what the neural net is going
1:20:38to output and how we are going to
1:20:39interpret the
1:20:42the 27 numbers is that these 27 numbers
1:20:45are giving us log counts
1:20:48basically.
1:20:49Um so, instead of giving us counts
1:20:52directly like in this table, they're
1:20:54giving us log counts.
1:20:56And to get the counts, we're going to
1:20:57take the log counts and we're going to
1:20:59exponentiate them.
1:21:01Now,
1:21:02exponentiation
1:21:04takes the following form.
1:21:06Um it takes numbers
1:21:08that are negative or they are positive.
1:21:10It takes the entire real line. And then
1:21:13if you plug in negative numbers, you're
1:21:14going to get e to the x, which is
1:21:18uh always below one.
1:21:20So, you're getting numbers lower than
1:21:21one.
1:21:23And if you plug in numbers greater than
1:21:25zero, you're getting numbers greater
1:21:27than one
1:21:28all the way growing to the infinity.
1:21:30And this here grows to zero.
1:21:33So, basically we're going to take these
1:21:36numbers
1:21:37here.
1:21:40And
1:21:43instead of them being positive and
1:21:44negative in all of the place, we're
1:21:46going to interpret them as log counts
1:21:48and then we're going to element-wise
1:21:50exponentiate these numbers.
1:21:52Exponentiating them now gives us
1:21:54something like this.
1:21:56And you see that these numbers now
1:21:57because of they went through an
1:21:58exponent, all the negative numbers
1:22:00turned into numbers below one like
1:22:020.338.
1:22:04And all the positive numbers originally
1:22:06turned into even more positive numbers
1:22:08sort of greater than one.
1:22:10Um so, like for example, seven
1:22:13um is some positive number over here. Um
1:22:18that is greater than zero.
1:22:21But, exponentiated outputs here
1:22:24um basically give us something that we
1:22:26can use and interpret as the equivalent
1:22:28of counts or originally. So, you see
1:22:31these counts here, 1 12 7 51 1 etc.
1:22:36The neural net is kind of now predicting
1:22:39uh
1:22:40counts.
1:22:41And these counts are positive numbers.
1:22:44They can never be below zero, so that
1:22:45makes sense. And uh they can now take on
1:22:48various values
1:22:49depending on the settings of W.
1:22:54So, let me break this down.
1:22:56We're going to interpret these to be the
1:22:58log counts.
1:23:01Another word for this that is often used
1:23:03is so-called logits.
1:23:05These are logits, log counts.
1:23:08And these will be sort of the counts.
1:23:11Logits exponentiated.
1:23:13And this is equivalent to the N matrix,
1:23:16sort of, the N
1:23:18array that we used previously. Remember,
1:23:20this was the N.
1:23:21This is the the array of counts. And
1:23:24each row here are the counts for the
1:23:27for the um
1:23:28next character, sort of.
1:23:32So, those are the counts. And now the
1:23:34probabilities are just the counts um
1:23:38normalized.
1:23:39And so, um
1:23:41I'm not going to find the same, but
1:23:43basically, I'm not going to scroll all
1:23:44over the place.
1:23:46We've already done this. We want to
1:23:48counts that sum along the first
1:23:50dimension, and we want to keep dims as
1:23:53true.
1:23:54We went over this, and this is how we
1:23:56normalize the rows of our counts matrix
1:24:00to get our probabilities.
1:24:03probs
1:24:04So, now these are the probabilities.
1:24:07And these are the counts that we have
1:24:10currently. And now when I show the
1:24:11probabilities,
1:24:13you see that um
1:24:15every row here,
1:24:17of course,
1:24:19will sum to one
1:24:21because they're normalized.
1:24:23And the shape of this
1:24:25is 5 by 27.
1:24:27And so really what we've achieved is for
1:24:29every one of our five examples, we now
1:24:32have a row that came out of a neural
1:24:34net.
1:24:35And because of the transformations here,
1:24:37we made sure that this output of this
1:24:39neural net now are probabilities or we
1:24:41can interpret to be probabilities.
1:24:44So,
1:24:45our WX here gave us logits
1:24:48and then we interpret those to be log
1:24:49counts.
1:24:50We exponentiate to get something that
1:24:52looks like counts.
1:24:54And then we normalize those counts to
1:24:55get a probability distribution.
1:24:57And all of these are differentiable
1:24:59operations.
1:25:00So, what we've done now is we are taking
1:25:02inputs.
1:25:03We have differentiable operations that
1:25:04we can back propagate through
1:25:07and we're getting out probability
1:25:08distributions.
1:25:09So, um for example, for the zeroth
1:25:12example that fed in,
1:25:15right, which was um
1:25:17the zeroth example here was a one-hot
1:25:18vector of zero.
1:25:20And um
1:25:22it basically corresponded to feeding in
1:25:26uh
1:25:26this example here. So, we're feeding in
1:25:28a dot into a neural net. And the way we
1:25:30fed the dot into a neural net is that we
1:25:32first got its index.
1:25:34Then we one-hot encoded it.
1:25:36Then it went into the neural net and out
1:25:39came
1:25:40this distribution of probabilities.
1:25:43And its shape
1:25:46is 27. There's 27 numbers and we're
1:25:49going to interpret this as the neural
1:25:51net's assignment for how likely
1:25:54every one of these characters um
1:25:56the 27 characters are to come next.
1:25:59And as we tune the weights W,
1:26:02we're going to be of course getting
1:26:03different probabilities out for any
1:26:05character that you input.
1:26:07And so now the question is just can we
1:26:08optimize and find a good W
1:26:11such that the probabilities coming out
1:26:13are pretty good. And the way we measure
1:26:15pretty good is by the loss function.
1:26:17Okay, so I organized everything into a
1:26:18single summary so that hopefully it's a
1:26:20bit more clear. So it starts here.
1:26:22We have an input data set.
1:26:24We have some inputs to the neural net
1:26:26and we have some labels for the correct
1:26:28next character in a sequence. And these
1:26:30are integers.
1:26:32Here I'm using uh torch generators now
1:26:35so that you see the same numbers that I
1:26:37see.
1:26:38And I'm generating
1:26:40um
1:26:4027 neurons weights
1:26:42and each neuron here receives 27 inputs.
1:26:48Then here we're going to plug in all the
1:26:50input examples x's into a neural net. So
1:26:52here, this is a forward pass.
1:26:55First, we have to encode all of the
1:26:57inputs into one-hot representations.
1:27:00So we have 27 classes, we pass in these
1:27:02integers and x inc becomes a array that
1:27:07is 5 by 27.
1:27:09Zeros except for a few ones.
1:27:12We then multiply this in the first layer
1:27:14of a neural net to get logits.
1:27:16Exponentiate the logits to get fake
1:27:18counts, sort of.
1:27:20And normalize these counts to get
1:27:22probabilities.
1:27:24So the la- these last two lines by the
1:27:26way here are called the softmax.
1:27:29Uh which I pulled up here.
1:27:32Softmax is a very often used layer in a
1:27:34neural net that takes these z's which
1:27:37are logits,
1:27:38exponentiates them,
1:27:40and uh divides and normalizes. It's a
1:27:43way of taking outputs of a neural net
1:27:45layer and these uh these outputs can be
1:27:47positive or negative.
1:27:49And it outputs probability
1:27:51distributions. It outputs something that
1:27:53is always sums to one and are positive
1:27:56numbers, just like probabilities.
1:27:58Um so it's kind of like a normalization
1:28:00function if you want to think of it that
1:28:01way. And you can put it on top of any
1:28:03other linear layer inside a neural net
1:28:05and it basically makes a neural net
1:28:07output probabilities. That's very often
1:28:09used and we used it as well here.
1:28:13So this is the forward pass and that's
1:28:14how we made a neural net output
1:28:16probability.
1:28:17Now
1:28:19you'll notice that
1:28:20um
1:28:23all of these
1:28:24this entire forward pass is made up of
1:28:26differentiable
1:28:27layers. Everything here we can back
1:28:29propagate through. And we saw some of
1:28:30the back propagation in micrograd.
1:28:33This is just
1:28:34multiplication and addition. All that's
1:28:36happening here is just multiply and then
1:28:38add. And we know how to back propagate
1:28:39through them.
1:28:40Exponentiation we know how to back
1:28:42propagate through.
1:28:43And then here we are summing and sum is
1:28:47is easily back propagatable as well.
1:28:50And division as well. So everything here
1:28:52is differentiable operation
1:28:54and we can back propagate through.
1:28:57Now we achieve these probabilities which
1:28:59are 5 by 27.
1:29:01For every single example we have a
1:29:03vector of probabilities that sum to one.
1:29:06And then here I wrote a bunch of stuff
1:29:08uh to sort of like break down uh the
1:29:10examples.
1:29:11So we have five examples making up Emma,
1:29:14right?
1:29:16And there are five bigrams inside Emma.
1:29:20So bigram example a bigram example one
1:29:23is that E is the beginning character
1:29:26right after dot.
1:29:28And the indexes for these are zero and
1:29:30five.
1:29:31So then we feed in a zero.
1:29:34That's the input to the neural net.
1:29:36We get probabilities from the neural net
1:29:38that are 27 numbers.
1:29:41And then the label is five because E
1:29:44actually comes after dot.
1:29:45So that's the label.
1:29:47And then
1:29:49we use this label five to index into the
1:29:52probability distribution here.
1:29:54So this index five here is 0 1 2 3 4 5.
1:29:59It's this number here.
1:30:01Which is here.
1:30:04So that's basically the probability
1:30:05assigned by the neural net to the actual
1:30:07correct character.
1:30:08You see that the network currently
1:30:10thinks that this next character that E
1:30:12following dot is only 1% likely. Which
1:30:15is of course not very good, right?
1:30:17Because this actually is a training
1:30:18example and the network thinks that this
1:30:20is currently very very unlikely. But
1:30:22that's just because we didn't get very
1:30:24lucky in generating a good setting of W.
1:30:27So right now this network thinks this is
1:30:28unlikely and 0.01 is not a good outcome.
1:30:32So the log likelihood then
1:30:34is very negative.
1:30:36And the negative log likelihood is very
1:30:38positive.
1:30:39And so four is a very high negative log
1:30:42likelihood and that means we're going to
1:30:44have a high loss.
1:30:45Because what is the loss? The loss is
1:30:47just the average negative log
1:30:49likelihood.
1:30:51So the second character is E M.
1:30:53And you see here that also the network
1:30:55thought that M following E is very
1:30:57unlikely, 1%.
1:31:00Uh the for M following M it thought it
1:31:02was 2%.
1:31:04And for A following M it actually
1:31:06thought it was 7% likely. So just by
1:31:09chance this one actually has a pretty
1:31:11good probability and therefore a pretty
1:31:12low negative log likelihood.
1:31:15And finally here it thought this was 1%
1:31:17likely.
1:31:18So overall our average negative log
1:31:20likelihood, which is the loss, the total
1:31:22loss that summarizes basically the how
1:31:25well this network currently works at
1:31:27least on this one word, not on the full
1:31:29data set, just the one word, is 3.76.
1:31:32Which is actually very fairly high loss.
1:31:34This is not a very good setting of Ws.
1:31:36Now here's what we can do.
1:31:38We're currently getting 3.76.
1:31:41We can actually come here and we can
1:31:42change our W. We can resample it. So let
1:31:45me just add one to have a different
1:31:47seed.
1:31:48And then we get a different W.
1:31:50And then we can rerun this.
1:31:52And with this different seed with this
1:31:54different setting of Ws we now get 3.37.
1:31:58So this is a much better W, right? And
1:32:00that and it's better because the
1:32:02probability just happened to come out
1:32:04higher for the for the characters that
1:32:07actually are next.
1:32:08And so you can imagine actually just
1:32:10resampling this, you know, we can try
1:32:12two.
1:32:14So
1:32:15Okay, this was not very good.
1:32:17Let's try one more.
1:32:18We can try three.
1:32:20Okay, this was terrible setting because
1:32:22we have a very high loss.
1:32:24So
1:32:26anyway, I'm going to erase this.
1:32:29What what I'm doing here, which is just
1:32:31guess and check of randomly assigning
1:32:33parameters and seeing if the network is
1:32:34good, that is amateur hour. That's not
1:32:37how you optimize a neural net. The way
1:32:39you optimize a neural net is you start
1:32:41with some random guess, and we're going
1:32:42to commit to this one even though it's
1:32:43not very good.
1:32:45But now the big deal is we have a loss
1:32:46function.
1:32:48So this loss
1:32:50is made up only of differentiable
1:32:52operations.
1:32:54And we can minimize the loss by tuning
1:32:57W's by computing the gradients of the
1:33:00loss with respect to these W matrices.
1:33:05And so then we can tune W to minimize
1:33:07the loss and find a good setting of W
1:33:09using gradient-based optimization. So
1:33:11let's see how that will work. Now things
1:33:13are actually going to look almost
1:33:14identical to what we had with micrograd.
1:33:17So here I pulled up the lecture from
1:33:20micrograd, the notebook. It's from this
1:33:22repository.
1:33:23And when I scroll all the way to the end
1:33:25where we left off with micrograd, we had
1:33:26something very very similar.
1:33:28We had a number of input examples. In
1:33:31this case, we had four input examples
1:33:32inside X's.
1:33:34And we had their targets. These are
1:33:36targets.
1:33:37Just like here, we have our X's now, but
1:33:39we have five of them, and they're now
1:33:41integers instead of vectors.
1:33:44But we're going to convert our integers
1:33:46to vectors, except our vectors will be
1:33:4727 large instead of three large.
1:33:51And then here what we did is first we
1:33:53did a forward pass where where ran a
1:33:55neural net on all of the inputs
1:33:58to get predictions.
1:34:00Our neural net at the time, this NFX,
1:34:02was a net a multi-layer perceptron.
1:34:05Our neural net is going to look
1:34:06different because our neural net is just
1:34:08a single layer.
1:34:10Single linear layer followed by a
1:34:12softmax.
1:34:13So, that's our neural net.
1:34:15And the loss here was the mean squared
1:34:17error. So, we simply subtracted the
1:34:19prediction from the ground truth and
1:34:21squared it and summed it all up. And
1:34:23that was the loss. And loss was the
1:34:24single number that summarized the
1:34:26quality of the neural net. And when loss
1:34:29is low, like almost zero, that means the
1:34:32neural net is um
1:34:33predicting correctly.
1:34:36So, we had a single number that uh that
1:34:38summarized the
1:34:40uh the performance of the neural net.
1:34:42And everything here was differentiable
1:34:43and was stored in massive compute graph.
1:34:46And then we iterated over all the
1:34:48parameters. We made sure that the
1:34:50gradients are set to zero.
1:34:51And we called loss.backward.
1:34:54And loss.backward initiated
1:34:55backpropagation at the final output node
1:34:58of loss. Right? So,
1:35:00yeah, remember these expressions? We had
1:35:02loss all the way at the end. We start
1:35:03backpropagation and we went all the way
1:35:05back.
1:35:06And we made sure that we populated all
1:35:08the parameters.grad.
1:35:10So, that grad started at zero, but
1:35:12backpropagation filled it in.
1:35:14And then in the update, we iterated over
1:35:16all the parameters and we simply did a
1:35:18parameter update where every single uh
1:35:21element of our parameters was nudged in
1:35:24the opposite direction of the gradient.
1:35:27And so, we're going to do the exact same
1:35:30thing here.
1:35:31Uh so, I'm going to pull this up
1:35:34on the side here
1:35:38so that we have it available. And we're
1:35:40actually going to do the exact same
1:35:41thing.
1:35:42So, this was the forward pass. So, where
1:35:44we did this.
1:35:46And probs is our Ypred.
1:35:49So, now we have to evaluate the loss,
1:35:50but we're not using the mean squared
1:35:51error. we're using the negative log
1:35:53likelihood because we are doing
1:35:54classification, we're not doing
1:35:56regression, as it's called.
1:35:59So, here we want to calculate loss.
1:36:02Now, the way we calculate it is is just
1:36:04this average negative log likelihood.
1:36:07Now, this probs here
1:36:10has a shape of 5 by 27.
1:36:13And so, to get all the we basically want
1:36:15to pluck out the probabilities at the
1:36:18correct indices here.
1:36:20So, in particular, because the labels
1:36:21are stored here in the array wise,
1:36:24basically what we're after is for the
1:36:26first example, we're looking at
1:36:27probability of five, right? At index
1:36:30five.
1:36:31For the second example,
1:36:32at the the second row or row index one,
1:36:36we are interested in the probability
1:36:37assigned to index 13.
1:36:40At the second example, we also have 13.
1:36:43At the third row, we want one.
1:36:47And at the last row, which is four, we
1:36:49want zero. So, these are the
1:36:51probabilities we're interested in,
1:36:53right?
1:36:54And you can see that they're not amazing
1:36:56as we saw above.
1:36:58So, these are the probabilities we want,
1:37:00but we want like a more efficient way to
1:37:02access these probabilities. Uh not just
1:37:05listing them out in a tuple like this.
1:37:07So, it turns out that the way to do this
1:37:08in PyTorch, uh one of the ways at least,
1:37:10is we can basically pass in all of these
1:37:16Sorry about that. All of these um
1:37:19integers in a vectors.
1:37:22So, the
1:37:23these ones, you see how they're just 0 1
1:37:252 3 4,
1:37:27we can actually create that using MP not
1:37:29MP, sorry, torch.arange of five.
1:37:320 1 2 3 4.
1:37:34So, we can index here with torch.arange
1:37:36of five.
1:37:38And here, we index with wise.
1:37:41And you see that that gives us
1:37:43exactly these numbers.
1:37:49So, that plugs out the probabilities of
1:37:51that the neural network assigns to the
1:37:54correct next character.
1:37:56Now, we take those probabilities and we
1:37:58don't we actually look at the log
1:37:59probability. So, we want to dot log.
1:38:03And then, we want to just average that
1:38:06up. So, take the mean of all of that.
1:38:08And then, it's the negative average log
1:38:11likelihood that is the loss.
1:38:14So, the loss here is
1:38:163.7 something. And you see that this
1:38:18loss, 3.76, 3.76 is exactly as we've
1:38:22obtained before, but this is a
1:38:23vectorized form of that expression.
1:38:26So, we get the same loss.
1:38:29And this same loss we can consider sort
1:38:31of as part of this forward pass.
1:38:34And we've achieved here now loss.
1:38:36Okay, so we made our way all the way to
1:38:37loss. We defined the forward pass. We
1:38:40forwarded the network and the loss. Now,
1:38:42we're ready to do backward pass.
1:38:44So, backward pass.
1:38:48We want to first make sure that all the
1:38:49gradients are reset. So, they're at
1:38:51zero.
1:38:52Now, in PyTorch, you can set the
1:38:55gradients to be zero, but you can also
1:38:56just set it to none. And setting it to
1:38:58none is more efficient. And PyTorch will
1:39:00interpret none as like a lack of a
1:39:03gradient and is the same as zeros.
1:39:05So, this is a way to set to zero the
1:39:07gradient.
1:39:10And now, we do loss.backward.
1:39:14Before we do loss.backward, we need one
1:39:16more thing. If you remember from
1:39:17micrograd,
1:39:19PyTorch actually requires
1:39:21that we pass in requires_grad is true.
1:39:25Uh so that we tell
1:39:27PyTorch that we are interested in
1:39:28calculating gradients for this leaf
1:39:30tensor. By default, this is false.
1:39:33So, let me recalculate with that.
1:39:36And then, set to none and loss.backward.
1:39:40Now, something magical happened when
1:39:42loss of backward was run.
1:39:44Because PyTorch, just like micrograd,
1:39:47when we did the forward pass here,
1:39:49it keeps track of all the operations
1:39:51under the hood. It builds a full
1:39:53computational graph.
1:39:54Just like the graphs we could produce in
1:39:57micrograd, those graphs exist inside
1:39:59PyTorch.
1:40:00And so, it knows all the dependencies
1:40:02and all the mathematical operations of
1:40:04everything.
1:40:05And when you then calculate the loss, we
1:40:07can call a dot backward on it.
1:40:09And dot backward then fills in the
1:40:11gradients of all the intermediates all
1:40:15the way back to W's, which are the
1:40:18parameters of our neural net. So, now we
1:40:20can do W.grad,
1:40:22and we see that it has structure.
1:40:23There's stuff inside it.
1:40:29And these gradients, every single
1:40:31element here,
1:40:33so W.shape is 27 by 27.
1:40:36W.grad's shape is the same, 27 by 27.
1:40:40And every element of W.grad
1:40:43is telling us
1:40:44the influence of that weight on the loss
1:40:47function.
1:40:48So, for example, this number all the way
1:40:50here,
1:40:52if this element, the 0 0 element of W,
1:40:55because the gradient is positive, it's
1:40:57telling us that this has a positive
1:41:00influence on the loss, slightly nudging
1:41:03W
1:41:04slightly taking W00
1:41:06and adding a small H to it
1:41:10would increase the loss mildly because
1:41:13this gradient is positive.
1:41:15Some of these gradients are also
1:41:16negative.
1:41:18So, that's telling us about the gradient
1:41:20information, and we can use this
1:41:22gradient information to update the
1:41:24weights of this neural network. So,
1:41:26let's now do the update. It's going to
1:41:28be very similar to what we had in
1:41:29micrograd. We need no uh loop over all
1:41:32the parameters because we only have one
1:41:34parameter uh tensor, and that is W. So,
1:41:37we simply do W.data plus equals uh, the
1:41:42We can actually copy this almost
1:41:43exactly. -0.1 *
1:41:46uh, W.grad.
1:41:48Um,
1:41:49and that would be the update to the
1:41:52tensor.
1:41:54So, that updates the tensor.
1:41:58And because the tensor is updated, we
1:42:01would expect that now the loss should
1:42:03decrease.
1:42:04So, here, if I print loss
1:42:09.item,
1:42:11it was 3.76, right? So, we've updated
1:42:14the W here. So, if I recalculate forward
1:42:17pass,
1:42:18loss now should be slightly lower. So,
1:42:213.76 goes to
1:42:233.74.
1:42:25And then, we can again set to set grad
1:42:28to none and backward update.
1:42:32And now the parameters changed again.
1:42:34So, if we recalculate the forward pass,
1:42:37we expect a lower loss again, 3.72.
1:42:42Okay, and this is again doing the We're
1:42:44now doing gradient descent.
1:42:48And when we achieve a low loss, that
1:42:50will mean that the network is assigning
1:42:52high probabilities to the correct next
1:42:54characters. Okay, so I rearranged
1:42:56everything and I put it all together
1:42:58from scratch.
1:42:59So, here is where we construct our data
1:43:01set of bigrams.
1:43:03You see that we are still iterating only
1:43:04on the first word, Emma.
1:43:06I'm going to change that in a second. I
1:43:09added a number that counts the number of
1:43:11elements in X's, so that we explicitly
1:43:14see the number of examples is five.
1:43:16Because currently we're just working
1:43:18with Emma, and there's five bigrams
1:43:19there.
1:43:20And here I added a loop of exactly what
1:43:22we had before. So, we had 10 iterations
1:43:25of gradient descent, of forward pass,
1:43:27backward pass, and an update. And so,
1:43:29running these two cells, initialization
1:43:31and gradient descent
1:43:32gives us some improvement on uh the loss
1:43:36function.
1:43:38But now, I want to use all the words.
1:43:41And there's not five, but 228,000
1:43:44bigrams now.
1:43:46However, this should require no
1:43:48modification whatsoever. Everything
1:43:49should just run because all the code we
1:43:51wrote doesn't care if there's five
1:43:53bigrams or 228,000 bigrams. And with
1:43:56everything, we should just work. So,
1:43:58you see that this will just run.
1:44:00But now we are optimizing over the
1:44:01entire training set of all the bigrams.
1:44:04And you see now that we are decreasing
1:44:06very slightly. So, actually, we can
1:44:08probably afford a larger learning rate.
1:44:12We can probably afford even larger
1:44:13learning rate.
1:44:20Even 50 seems to work on this very, very
1:44:22simple example, right? So, let me
1:44:24re-initialize, and let's run 100
1:44:26iterations.
1:44:29See what happens.
1:44:33Okay.
1:44:36We seem to be
1:44:39coming up to some pretty good losses
1:44:40here. 2.47.
1:44:42Let me run 100 more.
1:44:44What is the number that we expect, by
1:44:46the way, in the loss? We expect to get
1:44:48something around what we had originally,
1:44:50actually.
1:44:52So, all the way back, if you remember,
1:44:53in the beginning of this video, when we
1:44:55optimized uh just by counting,
1:44:58our loss was roughly 2.47
1:45:01after we added smoothing.
1:45:03But before smoothing, we had roughly
1:45:042.45
1:45:06uh likelihood.
1:45:08Um sorry, loss.
1:45:09And so, that's actually roughly the
1:45:11vicinity of what we expect to achieve.
1:45:13But before we achieved it by counting,
1:45:15and here we are achieving the roughly
1:45:17the same result, but with gradient-based
1:45:19optimization.
1:45:21So, we come to about 2.4
1:45:236, 2.45, etc.
1:45:26And that makes sense because
1:45:27fundamentally, we're not taking any
1:45:28additional information. We're still just
1:45:30taking in the previous character and
1:45:31trying to predict the next one. But
1:45:33instead of doing it explicitly by
1:45:35counting and normalizing,
1:45:38we are doing it with gradient-based
1:45:39learning. And it just so happens that
1:45:41the explicit approach happens to very
1:45:43well optimize the loss function without
1:45:46any need for gradient-based optimization
1:45:48because the setup for bigram language
1:45:50models are is is so straightforward and
1:45:52so simple. We can just afford to
1:45:54estimate those probabilities directly
1:45:55and maintain them in a table.
1:45:58But the gradient-based approach is
1:46:00significantly more flexible.
1:46:03So we've actually gained a lot because
1:46:06what we can do now is um
1:46:09we can expand this approach and
1:46:11complexify the neural net.
1:46:12So currently we're just taking a single
1:46:14character and feeding into a neural net
1:46:15and the neural net is extremely simple.
1:46:17But we're about to iterate on this
1:46:19substantially. We're going to be taking
1:46:21multiple previous characters and we're
1:46:23going to be feed them in feeding them
1:46:25into increasingly more complex neural
1:46:26nets. But fundamentally uh the output of
1:46:29the neural net will always just be
1:46:30logits.
1:46:32And those logits will go through the
1:46:34exact same transformation. We are going
1:46:36to take them through a softmax,
1:46:38calculate the loss function and the
1:46:39negative log likelihood,
1:46:41and do gradient-based optimization.
1:46:43And so actually as we complexify the
1:46:46neural nets and work all the way up to
1:46:48transformers,
1:46:49none of this will really fundamentally
1:46:51change. None of this will fundamentally
1:46:52change. The only thing that will change
1:46:54is
1:46:55the way we do the forward pass where we
1:46:57take in some previous characters and
1:46:59calculate the logits for the next
1:47:01character in a sequence. That will
1:47:03become more complex
1:47:05and uh but we'll use the same machinery
1:47:07to optimize it.
1:47:08And um
1:47:10it's not obvious how we would have
1:47:12extended this bigram approach into the
1:47:15case where there are many more
1:47:17characters at the input because
1:47:19eventually these tables would get way
1:47:21too large because there's way too many
1:47:23combinations of what previous characters
1:47:26uh could be.
1:47:27If you only have one previous character,
1:47:29we can just keep everything in a table,
1:47:31the counts. But if you have the last 10
1:47:33characters that are input, we can't
1:47:35actually keep everything in a table
1:47:36anymore. So this is fundamentally an
1:47:38unscalable approach, and the neural
1:47:40network approach is significantly more
1:47:42scalable, and it's something that
1:47:44actually we can improve on over time. So
1:47:46that's where we will be digging next. I
1:47:48wanted to point out two more things.
1:47:51Number one,
1:47:52I want you to notice that this
1:47:55X inc here,
1:47:56this is made up of one-hot vectors, and
1:47:59then those one-hot vectors are
1:48:00multiplied by this W matrix.
1:48:03And we think of this as uh multiple
1:48:05neurons being forwarded in a fully
1:48:07connected manner.
1:48:08But actually what's happening here is
1:48:10that, for example,
1:48:12if you have a one-hot vector here that
1:48:14has a one at, say, the fifth dimension,
1:48:17then because of the way the matrix
1:48:19multiplication works,
1:48:21multiplying that one-hot vector with W
1:48:23actually ends up plucking out the fifth
1:48:25row of W.
1:48:27Logits would become just a fifth row of
1:48:30W.
1:48:31And that's because of the way the matrix
1:48:33multiplication works.
1:48:35Um
1:48:36so that's actually what ends up
1:48:38happening.
1:48:40So but that's actually exactly what
1:48:42happened before.
1:48:43Because remember all the way up here,
1:48:46we have a bigram. We took the first
1:48:48character, and then that first character
1:48:50indexed into a row of this array here.
1:48:55And that row gave us the probability
1:48:56distribution for the next character.
1:48:58So the first character was used as a
1:49:00lookup into a
1:49:02uh
1:49:03matrix here to get the probability
1:49:05distribution.
1:49:06Well, that's actually exactly what's
1:49:07happening here.
1:49:08Because we're taking the index, we're
1:49:10encoding it as one-hot, and multiplying
1:49:12it by W.
1:49:13So logits literally becomes the
1:49:16uh the
1:49:18the appropriate row of W.
1:49:20And that gets just as before
1:49:22exponentiated to create the counts
1:49:25and then normalized and becomes
1:49:26probability.
1:49:27So this W here
1:49:29is literally
1:49:31the same as this array here.
1:49:35But W, remember, is the log counts, not
1:49:38the counts. So it's more precise to say
1:49:40that W exponentiated
1:49:42W.exp is this array.
1:49:46But this array was filled in by counting
1:49:49and by basically
1:49:52populating the counts of bigrams,
1:49:53whereas in the gradient-based framework,
1:49:55we initialize it randomly and then we
1:49:57let the loss guide us
1:50:00to arrive at the exact same array.
1:50:03So this array exactly here
1:50:05is
1:50:06basically the array W at the end of
1:50:09optimization, except we arrived at it
1:50:12piece by piece by following the loss.
1:50:15And that's why we also obtain the same
1:50:16loss function at the end. And the second
1:50:18note is if I come here,
1:50:20remember the smoothing where we added
1:50:22fake counts to our counts in order to
1:50:26smooth out and make more uniform the
1:50:28distributions of these probabilities.
1:50:31And that prevented us from assigning
1:50:32zero probability to
1:50:34um
1:50:35to any one bigram.
1:50:37Now, if I increase the count here,
1:50:40what's happening to the probability?
1:50:42As I increase the count, probability
1:50:45becomes more and more uniform.
1:50:48Right? Because these counts go only up
1:50:50to like 900 or whatever. So if I'm
1:50:51adding plus a million to every single
1:50:54number here, you can see how uh the row
1:50:57and its probability then when we divide
1:50:59is just going to become more and more
1:51:00close to exactly even probability,
1:51:03uniform distribution.
1:51:05It turns out that the gradient-based
1:51:06framework has an equivalent to
1:51:09smoothing.
1:51:10In particular,
1:51:13think through these W's here,
1:51:15which we initialized randomly.
1:51:18We could also think about initializing
1:51:20W's to be zero.
1:51:22If all the entries of W are zero,
1:51:26then you'll see that logits will become
1:51:27all zero.
1:51:28And then exponentiating those logits
1:51:30becomes all one.
1:51:32And then the probabilities turn out to
1:51:33be exactly uniform.
1:51:35So, basically when W's are all equal to
1:51:38each other, or say especially zero,
1:51:41then the probabilities come out
1:51:42completely uniform.
1:51:44So,
1:51:45trying to incentivize W to be near zero
1:51:49is basically equivalent to label
1:51:52smoothing. And the more you incentivize
1:51:54that in the loss function, the more
1:51:56smooth distribution you're going to
1:51:57achieve.
1:51:58So, this brings us to something that's
1:52:00called regularization,
1:52:02where we can actually augment the loss
1:52:03function to have a small component that
1:52:06we call a regularization loss.
1:52:09In particular, what we're going to do is
1:52:10we can take W,
1:52:11and we can for example square all of its
1:52:13entries,
1:52:14and then we can uh oops.
1:52:17Sorry about that.
1:52:19We can take all the entries of W, and we
1:52:20can sum them.
1:52:23And because we're squaring, uh there
1:52:25will be no signs anymore. Um
1:52:28negatives and positives all get squashed
1:52:30to be positive numbers.
1:52:31And then the way this works is you
1:52:33achieve zero loss if W is exactly or
1:52:36zero.
1:52:37But if W has non-zero numbers, you
1:52:39accumulate loss.
1:52:41And so, we can actually take this, and
1:52:42we can add it on here.
1:52:44So, we can do something like loss plus
1:52:48W squared
1:52:50dot sum.
1:52:52Or let's actually, instead of sum, let's
1:52:53take a mean, cuz otherwise the sum gets
1:52:55too large.
1:52:57So, mean is like a little bit more
1:52:58manageable.
1:53:01And then we have a regularization loss
1:53:02here, let's say 0.01 times,
1:53:05or something like that. You can choose
1:53:06the regularization strength.
1:53:09And then we can just optimize this.
1:53:12And now this optimization actually has
1:53:14two components. Not only is it trying to
1:53:16make all the probabilities work out, but
1:53:18in addition to that, there's an
1:53:19additional component that simultaneously
1:53:21tries to make all W's be zero. Because
1:53:24if W's are non-zero, you feel a loss.
1:53:26And so minimizing this, the only way to
1:53:28do achieve that is for W to be zero.
1:53:30And so you can think of this as adding
1:53:32like a spring force or like a gravity
1:53:34force that that pushes W to be zero.
1:53:37So W wants to be zero, and the
1:53:39probabilities want to be uniform, but
1:53:41they also simultaneously want to match
1:53:43up your your probabilities as indicated
1:53:46by the data.
1:53:47And so the strength of this
1:53:49regularization is exactly controlling
1:53:52the amount of counts
1:53:54that you add here.
1:53:57Adding a lot more counts here
1:54:00corresponds to
1:54:02increasing this number.
1:54:04Because the more you increase it, the
1:54:06more this part of the loss function
1:54:08dominates this part, and the more these
1:54:10these weights will be unable to grow
1:54:13because as they grow,
1:54:15they accumulate way too much loss.
1:54:18And so if this is strong enough,
1:54:21then we are not able to overcome the
1:54:23force of this loss, and we will never
1:54:26and basically everything will be uniform
1:54:28predictions.
1:54:29So I thought that's kind of cool.
1:54:30Okay, and lastly, before we wrap up,
1:54:33I wanted to show you how you would
1:54:34sample from this neural net model.
1:54:36And I copy-pasted the sampling code from
1:54:39before.
1:54:40Where remember that we sampled five
1:54:43times.
1:54:44And all we did is we started zero, we
1:54:46grabbed the current IX row of P.
1:54:50And that was our probability row
1:54:52from which we sampled the next index and
1:54:55just accumulated that and break when
1:54:57zero.
1:54:58And running this gave us these results.
1:55:03I still have the
1:55:05P in memory, so this is fine.
1:55:07Now,
1:55:09this P doesn't come from the row of P.
1:55:12Instead, it comes from this neural net.
1:55:14First, we take IX
1:55:17and we encode it into a one-hot row of X
1:55:21inc.
1:55:22This X inc multiplies our W,
1:55:25which really just plucks out the row of
1:55:26W corresponding to IX. Really, that's
1:55:29what's happening.
1:55:30And that gets our logits, and then we
1:55:33normalize those logits, exponentiate to
1:55:35get counts, and then normalize to get uh
1:55:37the distribution, and then we can sample
1:55:39from the distribution.
1:55:41So, if I run this,
1:55:45kind of anticlimactic or climatic,
1:55:47depending how you look at it, but we get
1:55:48the exact same result.
1:55:50Um
1:55:52and that's because this is in the
1:55:53identical model. Not only does it
1:55:55achieve the same loss, but um as I
1:55:58mentioned, these are identical models,
1:55:59and this W is the log counts of what
1:56:02we've estimated before. But, we came to
1:56:05this answer in a very different way and
1:56:07it's got a very different
1:56:08interpretation. But, fundamentally, this
1:56:10is basically the same model and gives
1:56:11the same samples here. And so,
1:56:14that's kind of cool. Okay, so we've
1:56:16actually covered a lot of ground. We
1:56:18introduced the bigram character-level
1:56:20language model.
1:56:22We saw how we can train the model, how
1:56:24we can sample from the model, and how we
1:56:25can evaluate the quality of the model
1:56:27using the negative log likelihood loss.
1:56:30And then we actually trained the model
1:56:31in two completely different ways that
1:56:33actually give the same result and the
1:56:35same model.
1:56:36In the first way, we just counted up the
1:56:38frequency of all the bigrams and
1:56:40normalized.
1:56:41In the second way, we used the uh
1:56:44negative log likelihood loss as a guide
1:56:47to optimizing the counts matrix
1:56:50uh or the counts array so that the loss
1:56:52is minimized in the in a gradient-based
1:56:55framework. And we saw that both of them
1:56:56give the same result.
1:56:58And um
1:57:00that's it.
1:57:01Now, the second one of these, the
1:57:02gradient base framework, is much more
1:57:03flexible. And right now, our neural
1:57:06network is super simple. We're taking a
1:57:08single previous character, and we're
1:57:10taking it through a single linear layer
1:57:12to calculate the logits.
1:57:14This is about to complexify. So, in the
1:57:16follow-up videos, we're going to be
1:57:17taking more and more of these
1:57:19characters,
1:57:20and we're going to be feeding them into
1:57:21a neural net.
1:57:23But, this neural net will still output
1:57:24the exact same thing. The neural net
1:57:25will output logits.
1:57:28And these logits will still be
1:57:29normalized in the exact same way, and
1:57:30all the loss and everything else in the
1:57:32gradient gradient base framework,
1:57:33everything stays identical.
1:57:35It's just that this neural net will now
1:57:37complexify all the way to transformers.
1:57:40So, that's going to be pretty awesome,
1:57:42and I'm looking forward to it. For now,
1:57:44bye.

About this transcript

This page contains the full transcript of The spelled-out intro to language modeling: building makemore by Andrej Karpathy, generated from the public captions YouTube serves with the video. The transcript has 19,445 words across 3,332 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.