YouTube transcript (kuYAsz7zspQ) — Transcript
Full transcript
- 0:05I hope everyone is staying dry. Um
- 0:08I'm not.
- 0:09So, as I mentioned last time, the Marine
- 0:11project had a
- 0:14which was running and it finished and it
- 0:17actually matched the the forecast. So,
- 0:19remember we were running these each of
- 0:21these curves is
- 0:23essentially a iso flops curve, which is
- 0:26a bunch of smaller model runs
- 0:28and you try to find the computer optimal
- 0:30point. You fit a scaling law and this
- 0:34was the point where
- 0:36we predicted the loss and we ran the
- 0:39loss the model and it got lost within
- 0:410.05. So, I thought that was pretty cool
- 0:44and if you extrapolate out to GPT-5
- 0:47level performance, this is the loss you
- 0:49get. Of course, your mileage might
- 0:52vary depending on how these scaling laws
- 0:55are.
- 0:56Okay, just wanted to share share that
- 0:58news.
- 1:00Um so, last lecture
- 1:02I gave an overview of the entire class
- 1:05and we are talking about tokenization,
- 1:07which is going to be on the first
- 1:08assignment. Um today I want to talk
- 1:11about um
- 1:13um resource accounting, which is going
- 1:16to be more on the systems side of
- 1:18things.
- 1:19Um so,
- 1:21uh to recall,
- 1:24the the main thing we're trying to do is
- 1:26um train the best model we can given a
- 1:29finite set of resources, which could be
- 1:31a compute, memory, sometimes data, but
- 1:34that's not really going to be a limiting
- 1:36factor for us in this class.
- 1:39And our goal is to simply to maximize
- 1:42the computational efficiency of our
- 1:45training.
- 1:46So, before you can optimize the
- 1:48computational efficiency, we need to
- 1:50understand the efficiency of a given
- 1:52computation and for that we need to
- 1:55understand the compute and memory
- 1:56characteristics.
- 1:58Um just to give you a taste of the type
- 2:00of questions you will hopefully be able
- 2:02to answer by the end of the class.
- 2:04Um so, here's a question. How long would
- 2:06it take to train a 70 billion parameter
- 2:08model on 15 trillion tokens on
- 2:101024
- 2:12Actually, this should be H100 cell.
- 2:15Um
- 2:16So, how do you answer that? Well,
- 2:18there's a formula which we'll talk about
- 2:19how you can get the number of flops to
- 2:22be be six times the number of parameters
- 2:24times the number of tokens. Um we can
- 2:27look up the spec sheet to see how fast
- 2:29the H100 is.
- 2:31Um we have this thing which called MFU,
- 2:33which we'll talk about 0.5.
- 2:36Um then you can estimate the number of
- 2:38flops
- 2:39um that you need
- 2:41that you the hardware gives you per day
- 2:44um and then you can compute the number
- 2:46of days. So, the number of days is 143.
- 2:49Okay.
- 2:50Um here's another question. What's the
- 2:51largest model you can train on H1 eight
- 2:54H100s using Adam W?
- 2:58Um so,
- 3:00uh you can look at Well,
- 3:03H100s have 80 GB of HBM
- 3:08memory.
- 3:09Um the number of bytes a per parameter,
- 3:13which is 2 + 2 + 4 + 4.
- 3:17We'll explain where that comes from.
- 3:19And then the number of parameters is um
- 3:23that you can get is going to be about 53
- 3:26billion. So, there's some caveats here
- 3:28that we don't count the activations,
- 3:31which depends on the batch length and
- 3:33batch size and the sequence length.
- 3:35So, this is all very rough back of the
- 3:37envelope calculations, but hopefully by
- 3:39the end of this class you'll understand
- 3:41where these come from and the point is
- 3:43not to
- 3:45precisely calculate every single thing,
- 3:48but just get the rough shape of things.
- 3:53Um okay. So, last time I talked about
- 3:57knowledge and what you can take away
- 3:59from this class.
- 4:00Um mechanics, which are how things work.
- 4:03So, today that will be pretty
- 4:04straightforward. The mechanics are just
- 4:06how PyTorch work, how tensors work. It
- 4:09should be fairly
- 4:11There's no magic here.
- 4:13Um the mindset I want you to impart on
- 4:16you is that resource accounting is going
- 4:18to be very crucial and while everyone to
- 4:22get in habit whenever you write you
- 4:24know, a piece of line of code, think
- 4:26about you know, the performance
- 4:28characteristics.
- 4:30And then finally, intuitions. Here we're
- 4:32just going to get a sense of the the
- 4:35resources, how they're spent. There's
- 4:37going to be no ML magic today. I'll
- 4:39leave that to Tatsu for the next
- 4:41lecture.
- 4:43Okay.
- 4:44So, let's get into things. So, let's
- 4:46start bottom up and start building up.
- 4:49So,
- 4:50what is at the bottom?
- 4:51The the bottom are tensors. So, tensors
- 4:53are the building block of storing
- 4:55everything.
- 4:56If you have parameters, gradients,
- 4:58optimizer states, data, activation,
- 5:01everything essentially is a tensor.
- 5:03Um so, you know, for example, you can
- 5:06take a look at the DeepSeek
- 5:083.2 model
- 5:10and you see that the model itself is a
- 5:13bunch of different tensors. Each tensor
- 5:14has some shapes and also some precision,
- 5:17which I'll talk about
- 5:18later.
- 5:21And
- 5:22so, as as you know,
- 5:24tensors subsume vectors, matrices
- 5:28and
- 5:29and can be generalize to any number of
- 5:32entities.
- 5:33Okay.
- 5:35So, let's talk about how
- 5:37much tensors take
- 5:39to store.
- 5:40Um
- 5:42So, it depends on the the type of
- 5:45tensor. So, in general, we're going to
- 5:47be dealing with tensors that store
- 5:49floating point, but tensors can also
- 5:51store
- 5:52you know, integers
- 5:54and other types.
- 5:56Um so, for floating point,
- 5:58um
- 5:59uh typically whenever you talk about
- 6:02float, I think the standard people refer
- 6:05to as float what is called float 32.
- 6:08Um so, a float 32, if you break it down,
- 6:12has 32 bits. One of the bits is a sign,
- 6:15eight of bits are the exponent, which
- 6:17gives you dynamic range and
- 6:20the rest is the the mantissa or the
- 6:22fraction, which gives you kind of
- 6:25you know, variation.
- 6:26Okay, this is also known as FP32
- 6:30or
- 6:31you know, single precision.
- 6:33Um so, and the term single precision
- 6:36comes from the fact that back in the day
- 6:37when you're doing scientific computing,
- 6:39float 32 was sort of like a baseline. It
- 6:42was just like you expect if you someone
- 6:44gave you a float, you would kind of
- 6:45expect it to be at least single
- 6:46precision and if you want
- 6:49um
- 6:50more precision, then you can get double
- 6:53precision as that's you know, float 64.
- 6:56But in deep learning, we're kind of
- 6:57going the other way because even 32 is a
- 7:01lot and the types of computations that
- 7:03we want to do don't demand the high
- 7:06precision that some kind of numeric
- 7:08simulations do.
- 7:10Um okay. So, before we get to other
- 7:12types, let's just look at a float 32.
- 7:15Okay, so let's construct a 4 by 8
- 7:17matrix.
- 7:18By default, the type of a tensor you
- 7:21create is float 32. So, if you want
- 7:23something else, you should declare it.
- 7:26Um and the memory usage is just the
- 7:28number of elements times the element
- 7:30size here, which is 4 bytes
- 7:33for a 32-bit
- 7:35number
- 7:36and that's going to be 128 bytes.
- 7:39Okay.
- 7:40Um
- 7:42just to give you kind of perspective.
- 7:44So, in GP3, which is a fairly old model,
- 7:46one of the matrix in the feed forward
- 7:49layer is
- 7:51um
- 7:51about
- 7:53you know, 2.3 GB.
- 7:56Okay, so these tensors can be you know,
- 7:58get quite big and this is by far even
- 8:00the biggest one that one can imagine.
- 8:04Okay, so
- 8:06So, since we're interested in
- 8:08efficiency,
- 8:10we want to generally reduce the amount
- 8:12of storage um
- 8:14and and we'll see that as you reduce the
- 8:17precision, um you actually save my
- 8:20memory and you also save time because
- 8:22operating on uh
- 8:24you know,
- 8:2516 bits is going to be faster.
- 8:28Um let's say
- 8:29twice as fast, but not always it sort of
- 8:32depends, but um
- 8:34and and then memory
- 8:36by reducing memory, we'll see later that
- 8:39actually reducing memory can save time
- 8:40as well,
- 8:42which is maybe less obvious, but it will
- 8:44hopefully become clear.
- 8:46So, the the obvious thing is you say,
- 8:49okay, let's take away half the bits. Now
- 8:51you have float 16.
- 8:52So, float 16 says you have a sign, you
- 8:54have only five bits of exponent and then
- 8:56the rest is the the mantissa.
- 8:59Um so,
- 9:01fine, fifth float 16 is is is good,
- 9:05except for
- 9:07you know, its dynamic range is
- 9:09kind of poor. Right? So, even if you
- 9:11have, let's say, try to construct a 1E
- 9:14minus 8 tensor, then that is actually
- 9:17just zero.
- 9:19So, you can't really represent very big
- 9:20numbers and you can't represent very
- 9:22small numbers.
- 9:24Um and the reason is that this exponent
- 9:26is you know, it's only
- 9:29you know,
- 9:30you know, five bits of of exponent from
- 9:33compared to eight.
- 9:35So,
- 9:36if you train with FP16, which people did
- 9:40back back in the you know, day,
- 9:44you will get instability. You will you
- 9:46get underflow, you get overflow, you'll
- 9:48get NaNs. It's it was it's pretty
- 9:51challenging.
- 9:53So, then
- 9:55so, BFloat16
- 9:57was invented. So this was developed in
- 10:00actually 2018 to address this issue.
- 10:03And the observation was that well, let's
- 10:07not compromise on
- 10:09the number of bits. It's it's it's the
- 10:10number of bits is going to be the same
- 10:11as FP16, but we're going to shift some
- 10:14of the bits from the mantissa to the
- 10:16exponent.
- 10:18So that means it has more dynamic range
- 10:20than float you know uh
- 10:2416.
- 10:25Um
- 10:26and it actually has the same dynamic
- 10:28range as float 32.
- 10:30But of course the the resolution is
- 10:32worse because there's no free lunch
- 10:34here.
- 10:35But it turns out that in a lot of deep
- 10:37learning applications,
- 10:38this is well worth the trade-off. You
- 10:40want the dynamic range to not overflow
- 10:43and underflow and because things are
- 10:46kind of sloppy and stochastic anyway,
- 10:48you don't need that much resolution.
- 10:52Okay?
- 10:55So
- 10:56let me actually skip over this this
- 10:58part.
- 10:59So
- 11:00So okay, so to summarize what are the
- 11:03what are the implications for training?
- 11:05So you can absolutely train with float
- 11:0732.
- 11:08And if you're training a small model,
- 11:11you probably just and you don't want to
- 11:12worry about it just you know float 32 is
- 11:14fine. But it requires four bytes of
- 11:18um
- 11:19uh of sorry, four bytes of memory per
- 11:24you know float. And that can take up a
- 11:27lot of memory. Um and if you
- 11:30train with float 16 then that's going to
- 11:34be too risky. So float 30 16 a BF 16 is
- 11:39sort of a sweet spot. Um even BF16 can
- 11:43be kind of risky as well maybe see in a
- 11:46little bit.
- 11:48Um one thing that people have has become
- 11:52kind of common practice is to use mixed
- 11:54precision training.
- 11:56Um and
- 11:58actually let me
- 12:00I think these
- 12:02Okay, let me compile this again.
- 12:05I think these are stale. So mixed
- 12:07precision training is uh where some of
- 12:11the computations use
- 12:14some precision and for some other
- 12:15computations use other precision. So in
- 12:18general as a general rule,
- 12:20BF
- 12:22you know 16 is
- 12:24what you would use for parameters
- 12:27activations and and gradients. And for
- 12:30optimizer states, you would use FP32.
- 12:32We'll discuss a little bit more of that
- 12:34about that later.
- 12:36And to do invoke mixed precision
- 12:38training, so PyTorch has a AMP library.
- 12:40We're not going to talk too much about
- 12:41this, but you basically um
- 12:44wrap your code and
- 12:47PyTorch has the library takes care of it
- 12:49automatically for you in the sense that
- 12:52it tries to cast things into
- 12:54BF16 when it's safe. So for example,
- 12:56matmuls are generally safe, but if you
- 12:58try to do
- 12:59exponentiation, then it will try to
- 13:02leave things as
- 13:03FP32.
- 13:06Okay, so we could do BF16 I think is
- 13:08probably where this class will end. But
- 13:11if you're feeling very adventurous, you
- 13:14can go farther. So FP8 this was
- 13:17introduced four years ago.
- 13:19Um it actually has been you know
- 13:21standardized. So if you look at FP8,
- 13:24there's actually two versions
- 13:26because depending on if you need more
- 13:29dynamic range or more resolution. Um and
- 13:33um there's there's you know two versions
- 13:35of them.
- 13:36Um and so we're not going to talk about
- 13:38that, but you know Nvidia has some you
- 13:41know the transformer engine supports
- 13:42FP8. And well most recently, you can
- 13:45actually go down to FP4. So last year
- 13:48Nvidia developed NVFP4 and there's only
- 13:51four bits per value. So if you just to
- 13:54so we're on the same page, four bits is
- 13:55not a lot. Like I can write all the
- 13:57values down on a single line here
- 14:00between minus six and and six. Um so
- 14:03that's not very much precision. Now
- 14:05there's a little bit of a cheat here,
- 14:07right? Because if you just naively only
- 14:09use these values,
- 14:10um you're you're not going to be able to
- 14:12train
- 14:13very very well. So what actually this
- 14:16means is that you every value
- 14:18you can have this four bits of freedom.
- 14:22But there are blocks and each block can
- 14:24be scaled up and down accordingly. So
- 14:26you can actually represent more values,
- 14:28but you can't
- 14:29represent the full
- 14:32dynamic range for every single value.
- 14:35And there was a model released
- 14:38actually this year, the NeMo-3 Super
- 14:40which was trained in FP4 which I think
- 14:42is a pretty pretty cool.
- 14:45Okay. So some of this you it's just good
- 14:49to know and some of this you can't even
- 14:51even touch. It's not like you create a
- 14:53tensor and you call call it FP
- 14:56you know it FP4. A lot of this is done
- 14:59under the hood by Nvidia's
- 15:02you know software stack.
- 15:05Okay, this was a bit of a kind of a long
- 15:08digression, but it's it's maybe helpful
- 15:10to appreciate that
- 15:12the intricacies of you know precision
- 15:13here.
- 15:14Okay, any questions before I move on?
- 15:17Yeah. Sorry
- 15:19there's a question like when you have a
- 15:20block you're scaling all those same
- 15:22values like like you have a tensor by
- 15:24that block something like that.
- 15:26The ratios of all of them will still be
- 15:28in FP4.
- 15:30Yeah, so the question is just like
- 15:32scaling with the maximum value is to
- 15:34have more specificity.
- 15:36Yeah, so the question is okay, just to
- 15:38explain the block a bit more. So you
- 15:40have let's say a block. Within that
- 15:42block you can vary
- 15:45up
- 15:46within the four bits. And in addition,
- 15:48all those values can be scaled up and
- 15:50down according to how many of your bits
- 15:52your scaling factor is. So if you look
- 15:55at an individual value,
- 15:58you actually get more than four bits of
- 16:00dynamic range, but it's just like you
- 16:02can't have this value be way over here
- 16:04and the next neighboring value way down
- 16:08Okay, so
- 16:09uh one question about like
- 16:11what about like one bit?
- 16:12Uh
- 16:13couldn't BF16 be pushed to a bit one bit
- 16:15and all of that? Yeah, so there's a
- 16:17problem with training obviously.
- 16:19Right, right. So so maybe this is a good
- 16:20point. So the question is about one bit
- 16:23because you can't really go lower than
- 16:24that.
- 16:26So
- 16:28there's a difference between training
- 16:29and inference. So a lot of the low bit
- 16:31stuff
- 16:33if we talk about quan- I think we'll
- 16:34talk about quantization later. Is that
- 16:36you train a model at
- 16:38on and you know maybe even BF16 and then
- 16:42you quantize into like let's say one or
- 16:44two bits. Um and that is much easier
- 16:47than training a one bit
- 16:50language model which I don't think
- 16:51anyone has
- 16:53you know done. Maybe it's possible, but
- 16:55I don't think anyone has trained
- 16:56anything credible there.
- 16:59Okay.
- 17:00Let us go on then. So we talked about
- 17:03tensors and the memory calculus is
- 17:06pretty simple just the number of
- 17:07elements times however much memory each
- 17:09element takes up.
- 17:11Um so
- 17:13you know by default the tensors you
- 17:15create in PyTorch are going to be on CPU
- 17:18and
- 17:20um and of course you have to if you want
- 17:24things to go fast, you want to move them
- 17:25to GPU. And actually so there's a bit of
- 17:30a I have a slight issue with the slides
- 17:32with that. They were executed on my
- 17:33laptop which means that I don't have
- 17:35GPU. So some of the code I'll just show
- 17:37but not um
- 17:39execute. Um
- 17:41so
- 17:42to okay, let's see.
- 17:45This is maybe not that the interesting,
- 17:47but I think everyone knows how to move
- 17:50um
- 17:51tensors to GPU. But just remember to do
- 17:53that otherwise you won't get your
- 17:55speed ups.
- 17:58Okay, so we talked about memory of
- 18:00tensors which is very straightforward.
- 18:02Now let's talk about computing with
- 18:04tensors.
- 18:05So before talking about flops accounting
- 18:07for um
- 18:09the the tensor operations,
- 18:11let's take a little bit of a digression
- 18:13to talk about einsums. So how many of
- 18:16you have familiar used einsums before?
- 18:19Okay, so like maybe two thirds. Okay,
- 18:22good. So um
- 18:24the motivation behind einsums for those
- 18:26of you who might not be indoctrinated is
- 18:30it's very easy to mess up or I I find it
- 18:33very confusing to
- 18:35look at code such as this and you you
- 18:39have X and Y and then you have like
- 18:42transpose minus two and minus one and
- 18:45you're trying to figure out like what
- 18:47what minus two and minus one is. Um
- 18:50And so this is kind of maybe
- 18:54the motivation
- 18:56for um
- 18:58you know using variable and names rather
- 19:00than indices.
- 19:02So einsums is a library for manipulating
- 19:04you know tensors where the dimensions
- 19:05are named. And this is inspired by
- 19:07Einstein summation notation. There's a
- 19:10nice tutorial which you can go through.
- 19:12I'm just going to cover some of the you
- 19:15know basics.
- 19:16Um
- 19:17So basically the way to think about
- 19:19einsum is a generalized matrix
- 19:21multiplication with
- 19:23you know good bookkeeping. So here's an
- 19:26example. So I have
- 19:28um
- 19:29a matrix 3 by 4. I have a 4 by 3 matrix
- 19:33and if I do the matmul,
- 19:35this is you know this is actually you
- 19:38know pretty nice. It's pretty easy to
- 19:40understand.
- 19:41Uh in einsums, basically you say
- 19:44X has two dimensions, the row and the
- 19:47column which I'm going to name seq1 and
- 19:49hidden. I'm going to have Y that's a
- 19:51matrix which has also rows and columns,
- 19:54which I'm going to name um
- 19:56hidden and seek two. And um I'm going to
- 20:00produce um a um tensor or here a matrix
- 20:04where um the dimensions are indexed by
- 20:07seek one and seek two. And anything that
- 20:09is not mentioned here, the hidden, uh
- 20:12gets summed out.
- 20:13Okay? So, the way this works is I'm
- 20:15going to sum over enumerate over all
- 20:18possible values of all the variables uh
- 20:22that
- 20:23occur here, which are seek ones, hidden,
- 20:26and seek two.
- 20:28And I'm going to basically um index into
- 20:32X, index into Y, multiply them, and dump
- 20:34them accumulate them into um the result
- 20:37Z uh sub seek one seek two.
- 20:41Okay?
- 20:43All right. So, you say this is
- 20:46Come on, this is much easier than this,
- 20:48right? Um so,
- 20:51Okay, let's try a more complicated
- 20:52example. So, um here's the example I
- 20:54showed
- 20:55you know, before. Now we have a a
- 20:57tensor. It's 2 by 3 by 4 and another
- 21:01tensor 2 by 3 by 4. And if I were doing
- 21:05things um kind of the old way, basically
- 21:08I am what I'm doing is uh transposing
- 21:11these last two and then um
- 21:14because uh at implicitly sort of batches
- 21:17the the dimensions that are not the the
- 21:19matmul dimensions, um
- 21:22then you get the answer. Okay? So, you
- 21:24have to kind of reason about this a bit.
- 21:26Einops makes this very uh clear. It says
- 21:29there's a batch dimension, there's seek
- 21:31one uh hidden, seek two hidden, and I'm
- 21:34just going to produce batch uh seek one
- 21:36seek two.
- 21:37Okay?
- 21:38And notice that there's no transpose
- 21:40because I just you know, in some sense
- 21:43I've done the transpose by the naming.
- 21:46If I had hidden um and seek two, then it
- 21:48would be a no transpose. I always get
- 21:50confused by transposes and the fact I
- 21:52don't have to think about transposing
- 21:54makes me happy.
- 21:57Okay.
- 21:59So, um if you want to get fancy, uh you
- 22:03you can say, well, batch um I'm just
- 22:05going to replace with dot dot dot. And
- 22:08this means that if I had, let's say, uh
- 22:11you know, uh a you know, rank 10 tensor
- 22:13with, you know, eight different uh you
- 22:16know, batching dimensions, I can just
- 22:17write dot dot dot without enumerating
- 22:19all of them.
- 22:20Uh and this comes up in um you know,
- 22:22language modeling because you might have
- 22:24a batch dimension, you might have a
- 22:25sequence dimension, you might have a
- 22:27head dimension, and you're trying to do
- 22:29like this matrix operation for all of
- 22:31them. And you might not want to um have
- 22:34to just, you know, worry about this.
- 22:36Um and the nice thing is that you can
- 22:38write modular code where you can write
- 22:39this dot dot dot uh without worrying
- 22:42about, you know, the even the shape of
- 22:43the tensor that, you know, comes in.
- 22:47Okay?
- 22:49All right. So, that's uh ein sum, uh
- 22:52which is in the einops uh library.
- 22:55Um there's a reduce.
- 22:58So, this is a generalization of sum,
- 23:01mean, um max, and min.
- 23:03So, um for example, if you have, let's
- 23:06say, uh this this tensor,
- 23:09and you want to sum
- 23:11um according to dimension of dim minus
- 23:13one,
- 23:14okay, which means sum along the the last
- 23:17uh dimension here.
- 23:18Um so, again, I don't like the uh this
- 23:22this notation, but what you can do is
- 23:24called reduce.
- 23:25And basically what you say is that
- 23:28there's some batching dimensions. In
- 23:30this case, it would be um this first
- 23:32these first two. And then there's some,
- 23:34you know, hidden uh dimension, which
- 23:37doesn't appear on the right side, which
- 23:38means that it gets summed.
- 23:40And here I put sum, which means that the
- 23:41aggregation reduction operation is sum,
- 23:43but you can replace it with uh mean or
- 23:45max or min.
- 23:47Yeah. Is there some speed up to this?
- 23:50Is there some speed up to this? This
- 23:51basically reduces to the same type of
- 23:53primitive operations. You can think
- 23:55about it as just uh sugar.
- 23:57So, it should be the same.
- 24:03Um okay. So, the final thing I'll talk
- 24:05about is rearrange. So, this is I think
- 24:07a pretty powerful tool. Um
- 24:10I think it will come up in the
- 24:11assignment once. Um
- 24:13So, sometimes you have a dimension that
- 24:15actually represents um you know, two
- 24:17dimensions, and you want to operate on
- 24:20one of them. And the reason this happens
- 24:22is that sometimes you have a matrix and
- 24:23you flatten it.
- 24:25Um okay. And then you want to maybe
- 24:27unflatten and flatten. So, this is the
- 24:29way that works here.
- 24:31Um
- 24:32So, imagine I have a um a matrix 3 by 8,
- 24:36but where the um
- 24:39this
- 24:40uh the the this dimension eight actually
- 24:43represents like a 2 by 4 um matrix. So,
- 24:47um
- 24:48I want to multiply that 2 by 4 matrix by
- 24:51this 4 by 4 matrix.
- 24:53So, what I'm going to do,
- 24:55um actually, is to um call rearrange.
- 24:59And what I am doing here is saying
- 25:03um
- 25:04Look, this is some number of batch uh
- 25:06dimensions, which here just corresponds
- 25:08to the first uh element here. And then
- 25:11here I use parentheses to say that this
- 25:14H actually represents the product of
- 25:17heads and hidden one, um where I have to
- 25:20obviously there's multiple ways to
- 25:21decompose this. Could be 2 by 4, 4 by 2.
- 25:23So, I said the number of heads is two,
- 25:25which means that hidden one is four.
- 25:27And then
- 25:29I can um break that up into um you know,
- 25:33two dimensions, heads and hidden.
- 25:35Okay? So, uh this creates um this So,
- 25:38before X uh looked like this, and now X
- 25:43uh looks like, you know, this.
- 25:45So, um cuz this might be a little bit
- 25:47hard to see. Um
- 25:50So, then you can perform your operation
- 25:53transformation on uh W. And this is sort
- 25:56of what we've seen before, where you
- 25:58have um just a standard matmul, where um
- 26:02there's on some number of batching
- 26:04dimensions um for X here. And this is
- 26:07hidden times um
- 26:09hidden by hidden two.
- 26:11Um and then once you've done that
- 26:13transformation, you can rearrange it
- 26:16back.
- 26:17Um and this is straightforward. You
- 26:18basically look at two dimensions, and
- 26:21then you can group them into to one
- 26:24dimension.
- 26:25Okay?
- 26:30Yeah.
- 26:31Like if you take a two-dimensional thing
- 26:32and shift it into one dimension, are
- 26:34there two ways?
- 26:36Like you could do it row major or column
- 26:37major. Yeah. So, the question is that if
- 26:39you have a one-dimensional thing and you
- 26:40shift it into Sorry, you you have a
- 26:42two-dimensional thing and you shift into
- 26:44one dimensional thing, which way do you
- 26:46uh do it? Well, the order you do it is
- 26:49specified in the order uh here. Oh,
- 26:52okay. Yeah.
- 26:55Okay.
- 26:57So, I I
- 26:58Sometimes I find it takes a bit of time
- 27:00to get it used to, but it's well worth
- 27:01it because once you have einops, then
- 27:04you just think in a sort of a different
- 27:06way, and all the transposes and
- 27:08reductions all of them it's just like um
- 27:11uh it's becomes more fluid. Right? You
- 27:13just have to think through these kind of
- 27:14more um bespoke primitives.
- 27:19All right. So,
- 27:21Okay, we're going to use uh einops. Um
- 27:24we'll see it a little bit later.
- 27:26Um so, let Now let's go return back to
- 27:29resource accounting question. So, I have
- 27:31tensors. We've talked about how they
- 27:32take memory. So, how much compute do do
- 27:35they take? Okay? So, um the
- 27:38thing we're going to use to measure
- 27:41computation cost is the number of flops.
- 27:43A flop is a floating point operation,
- 27:45and it's going to we're going to assume
- 27:47it's a basic operation like addition or
- 27:49multiplication.
- 27:50So, now there's other things that you
- 27:52know, GPUs can do, but for the most part
- 27:54we're just going to uh ignore them
- 27:56because these are the the bread and
- 27:58butter and are going to eat up most of
- 28:00your um
- 28:01time.
- 28:03So, one thing that is a kind of a pet
- 28:05peeve is that if I say the word flops,
- 28:08it's actually ambiguous what I mean. Um
- 28:11so, there's flops, which is saying the
- 28:15number of floating point operations,
- 28:16usually written flops with a lowercase
- 28:19S. This is a measure amount of
- 28:21computation done. And then there's
- 28:23FLOPS, um
- 28:25which is floating point operations per
- 28:26second.
- 28:27Um sometimes it's also very confusingly
- 28:30written as FLOPS with uppercase S, which
- 28:32but I'm going to always write /s to make
- 28:34it clear that this is uh measuring the
- 28:36speed of hardware.
- 28:38So, if you go and see that uh H100s have
- 28:40800 989
- 28:43um
- 28:44um you know, teraflops, it's the second.
- 28:46And when I say that, you know, GPT-3
- 28:49took, you know, a 3e25 or whatever it
- 28:51is, uh flops, that's the the former.
- 28:54Okay?
- 28:55So, just to get that out of the way.
- 28:57Um
- 28:58so, just to give you an order of
- 28:59magnitude, so the number of FLOPS when I
- 29:02talk about 1e22
- 29:04or 23 or 25, these are kind of referring
- 29:08implicitly to amount of compute or the
- 29:09scale of some of these models.
- 29:13And uh so, if you look at H100s, um
- 29:16um you know, if you look at this uh
- 29:20glossy's uh
- 29:21uh spec sheet, um actually, it's not on
- 29:24this page. Okay, forget it. Um
- 29:27there is a spec sheet and it'll
- 29:28uh it'll tell you that for uh BF16,
- 29:32the number of uh uh FLOPS/s
- 29:35is um
- 29:37uh 1979.
- 29:39Um and then you go and you benchmark and
- 29:42it's like, wait a minute, that's not
- 29:44actually what I'm getting. And then you
- 29:46go read the the, you know, the fine
- 29:47print and there's a footnote that says
- 29:49this is with uh sparsity, so sparse
- 29:51matrix, and for dense it's uh you know,
- 29:53over two. Okay? So, all you always have
- 29:56to take these numbers divide by two. So,
- 29:57that's why you see these uh divide by
- 29:59two.
- 30:01Um okay.
- 30:03So,
- 30:04so this allows us to um
- 30:08you know, uh you know, just so our
- 30:10intuition, so if you have eight H100, so
- 30:12this one node uh for uh two weeks,
- 30:15that's um
- 30:17you know, H times the number of uh
- 30:19seconds uh per in two weeks.
- 30:22Um
- 30:23actually, this is looks like it's one
- 30:25week. Okay, fine. It's a one week uh
- 30:27times uh the number of flops you get per
- 30:30second. So, that's you know, about uh 5e
- 30:3321.
- 30:35Okay, so this is just kind of building
- 30:37intuition for the number of flops that
- 30:38certain types of hardware have and how
- 30:40many flops, you know, certain types of
- 30:42models uh require.
- 30:44Okay, it's nothing fancy, it's just math
- 30:47and uh napkin math. Um
- 30:49Okay, so so now let's do some more
- 30:52something more mechanical. Um
- 30:54So, you know, suppose you have uh a a
- 30:57linear model. Um
- 30:59it turns out that a lot of this calculus
- 31:01is uh
- 31:03of counting flops is actually going to
- 31:05be at the core it's like linear matmuls.
- 31:08So, this is actually not without too
- 31:10much loss of generality. So, you have M
- 31:12points, each point is D dimensional, and
- 31:15um we're going to map
- 31:18uh
- 31:19each of these D dimensional vectors to
- 31:21uh a K dimensional output. Okay? So, um
- 31:25so B is going to be the number of uh
- 31:27points, D is the number of dimension
- 31:30input dimensions, and K is the number of
- 31:31output dimensions.
- 31:33Okay, so we're let's construct uh some
- 31:36uh an X, which is the data matrix B by
- 31:40D, the weight matrix is D by K, and we
- 31:43do a the matmul.
- 31:45Um
- 31:45the question is, you know, how many
- 31:47flops uh that is.
- 31:49And uh it turns out that this is going
- 31:52to be two times
- 31:54um
- 31:55the
- 31:56uh basically the product of all the the
- 31:58three dimensions.
- 32:00And the way to see that is that um we
- 32:03have one, you know, multiplication
- 32:06for each uh you know, triple,
- 32:08and then also one addition.
- 32:11Okay?
- 32:12So, there's a minus one because if uh
- 32:14you don't you have to actually add like,
- 32:17you know, um
- 32:18in this case uh D minus one times, but
- 32:21let's ignore that.
- 32:23Okay, I'll come back to this um if this
- 32:25was a bit fast. Um I think there's a
- 32:27there's another way to uh you know,
- 32:28derive this.
- 32:31So, what about the flops of other
- 32:33operations? So, um element-wise
- 32:36operations are just uh you know, the
- 32:38size of the matrix. I think that's
- 32:40fairly clear, so addition also requires
- 32:42NM flops. Um so, in general, no other
- 32:45operation you encounter is expensive as
- 32:48matrix multiplication for large enough
- 32:50matrices. So, in general, we're just
- 32:53going to focus on what matmuls are doing
- 32:55with the important caveat of when we
- 32:57talk about memory. Yeah.
- 32:59I would just just like this is for
- 33:02interest, like there are some other
- 33:04algorithms for doing matrix
- 33:05multiplication. Is this only for, you
- 33:07know, just doing it
- 33:08normally?
- 33:09So, the question is that there are other
- 33:11algorithms for doing matrix
- 33:13multiplication. You mean like sub cubic
- 33:15uh cubic algorithms. Uh so, in general,
- 33:18that's
- 33:20the optimization the algorithms that
- 33:21people are going to explore for much of
- 33:23matrix multiplications are going to be
- 33:24much more about how you co-design with
- 33:27the systems rather than these more
- 33:29asymptotic uh algorithms.
- 33:32Yeah.
- 33:33Yeah. Um we're considering addition and
- 33:36multiplication in the same way. It's not
- 33:38possible to do addition more efficiently
- 33:40than multiplication.
- 33:42Yeah, so I think the way uh the hardware
- 33:45is built, the two are basically the, you
- 33:47know, the same. But yeah, intuitively it
- 33:49seems like addition I can do addition
- 33:51faster than do multiplication, but um
- 33:53the hardware the way hardware built is
- 33:55built they're kind of the same.
- 33:59Okay, so
- 34:00you can think about this matmul as um
- 34:03B is the number of data points,
- 34:05um and DK is the number of parameters,
- 34:09right? So, remember X is B by D and W is
- 34:12D by K.
- 34:14So, another way to think about this um
- 34:17formula is that the number of flops uh
- 34:21in order to do a forward pass of this
- 34:24linear uh matrix is actually two times
- 34:28the number of tokens or data points
- 34:30times the number of parameters.
- 34:33Okay?
- 34:34So, this is a you know, it turns out
- 34:36that this actually generalizes to
- 34:37transformers, which if you remember the
- 34:39six times N times D formula, we can kind
- 34:41of see the shape of that uh you know,
- 34:43forming.
- 34:46Okay, so um
- 34:48unfortunately, these calculations are
- 34:49not going to be very meaningful because
- 34:51I'm doing this on on on CPU, but I'll
- 34:54just walk through the
- 34:55uh the code here. So, um
- 34:58so the what we've so far done is uh
- 35:03measured flops. So, this is independent
- 35:06of of hardware, it's just like the
- 35:08number of calculations you need to do
- 35:10for your model.
- 35:12So, now the question is like how long
- 35:13does it actually take on hardware?
- 35:15So, one way to find out is you just time
- 35:17it.
- 35:18Okay? So, in this class, I think in uh a
- 35:20few lectures we're going to talk more
- 35:22about uh benchmarking, um but here's a
- 35:25little preview.
- 35:27So,
- 35:28um in general,
- 35:30um when you time, especially on on GPU,
- 35:33you um have to call uh CUDA synchronize
- 35:37to to make sure that because, you know,
- 35:39the GPU is like kind of running
- 35:40asynchronously, you want to make sure
- 35:41that you have this kind of
- 35:43synchronization point.
- 35:45Um and then you perform the operation,
- 35:47and after the
- 35:48uh operation,
- 35:50then you
- 35:52uh you also have to have the
- 35:53synchronization barrier. If you omit
- 35:55this, you're going to find that wow,
- 35:57your timings are really fast, and that's
- 35:59because these this is a non-blocking
- 36:00call, it just returns.
- 36:02Um and often it's general good practice
- 36:04to try this multiple times and take the
- 36:06the average.
- 36:08>> [clears throat]
- 36:09>> Okay.
- 36:10So,
- 36:12so um the actual flops per second
- 36:15is um
- 36:17basically the number of flops you did
- 36:21times the time that you recorded on your
- 36:23hardware.
- 36:24Right?
- 36:26And um so, remember there's a also a
- 36:28number number, which is the GPU has a
- 36:30spec sheet.
- 36:32Uh let's see if I can pull it up on this
- 36:35link.
- 36:36Um
- 36:39I feel like maybe I have the wrong link.
- 36:40I'll have to fix that after class.
- 36:43Um
- 36:44that's uh it gives you
- 36:47um some number of flops, which was 989
- 36:52um
- 36:53teraflops.
- 36:54And so, in general, the number of actual
- 36:58flops per second is going to be
- 36:59different from the promised uh flops per
- 37:02second.
- 37:03Okay?
- 37:04Um and the
- 37:07Okay, um oops.
- 37:09Sorry.
- 37:12And the uh the way to direct think about
- 37:14the discrepancy is something called um
- 37:17model flops utilization or MFU.
- 37:20And the definition of uh MFU is the
- 37:23actual flops per second divided by the
- 37:25promised uh flops per second. And here
- 37:28this is, you know, ignoring the the um
- 37:31you know, the communication and other
- 37:33overhead.
- 37:34Okay? So, you basically take the actual
- 37:36and divide by the promised.
- 37:39So, in general,
- 37:40um
- 37:41you know, it's rare that you get more
- 37:42than what's uh you know, uh
- 37:45I mean, it's
- 37:46yeah, you you just never get anything
- 37:48more than what you were promised. Often
- 37:51you get less, and in general, if you get
- 37:54about MFU of uh 0.5 for modern models,
- 37:57um you should be pretty happy with
- 37:59yourself. Um if you have only just like
- 38:02a straight-up matmul, you can get maybe
- 38:04like, you know, potentially like 80 uh
- 38:070.8 um even. Uh but you you usually
- 38:10can't get that high. And sometimes if
- 38:13you're um have really,
- 38:16you know, something's really wrong,
- 38:17you'll get something like 0.1, which
- 38:19means that you should do something.
- 38:21Okay.
- 38:22So, whenever you're you write your
- 38:24model, you can calculate uh you now know
- 38:27how to calculate MFU through a
- 38:29combination of counting the number of
- 38:31logical flops that you
- 38:33uh your model is needs to, you know, do,
- 38:37um and then looking at the wall clock
- 38:39time and essentially dividing.
- 38:41Yeah.
- 38:42Was there a question over there? Oh.
- 38:49Yeah, so is the promised What is the
- 38:51promised flops number? That is um in the
- 38:54spec sheet, it is
- 38:56um
- 38:57already divided by two
- 38:59um of the 1989 number.
- 39:03And then on top of that you only get
- 39:04like 0.5 of that in in general,
- 39:06depending on your um computation.
- 39:13You're getting 50%
- 39:15Yeah, so why are you getting only 50%
- 39:17MFU?
- 39:18Um actually, that's a good question.
- 39:19I'll come back to that when we talk
- 39:21about memory bottlenecks.
- 39:27Okay, so to to summarize here,
- 39:30um matrix multiplications dominate um
- 39:33the
- 39:34you know, generally the computations,
- 39:35and that's sort of by, you know, design.
- 39:38Um
- 39:39so
- 39:40um
- 39:42and the number of flops per second
- 39:44depends on the hardware.
- 39:46So, better hardware um leads to better
- 39:49num more flops, and also depends on the
- 39:52data type. Um which means that if you
- 39:55look at the the spec sheet, um different
- 39:58uh data types will have different flops.
- 40:00If you try to do float 32, nowadays it's
- 40:02going to be really really slow because
- 40:04um they're not really optimizing for
- 40:06that workload, whereas uh now BF16 or um
- 40:11FP8 are going to be much uh faster.
- 40:16Okay? And MFU, now you know what MFU is.
- 40:18It's uh the you know, actual flops
- 40:21divided by promised flops.
- 40:27All right. So, to go back to the
- 40:29question of why is MFU 0.5?
- 40:32And to understand that, I'm going to
- 40:34have to introduce this idea of
- 40:35arithmetic intensity.
- 40:37And and the re- reason is that, well,
- 40:40it's not just um
- 40:43you know, doing a bunch of
- 40:45matmuls and then you're done and looking
- 40:47at how long it the matmuls take.
- 40:49Um this is my very
- 40:51uh cartoon version of what uh hardware
- 40:54looks like.
- 40:55Um you have high bandwidth memory
- 40:58uh and then you have where the the
- 41:00compute uh cores or the cell
- 41:02accelerators chips are.
- 41:04And then um what how do you compute?
- 41:08Well, you have to
- 41:10send your inputs, your matrices there's
- 41:13there's uh tensors are sitting down
- 41:14here, from the memory to the
- 41:16accelerator. You do the the computation,
- 41:19and then you uh send it back.
- 41:22Okay? So, if you want to measure how
- 41:24long this takes, this sort of depends on
- 41:26two things. One is the accelerator
- 41:28speed,
- 41:30which is what we've talked about uh just
- 41:32you know, just now.
- 41:34But the other thing that matters is the
- 41:35memory bandwidth of your hardware, which
- 41:37we haven't talked about.
- 41:39And uh if you look at the spec sheet, we
- 41:42talked about how the flops per second
- 41:44was
- 41:45um
- 41:461979 E12
- 41:48/ 2.
- 41:50And the bytes per second, which is the
- 41:51memory bandwidth, this is 3.3 uh
- 41:53terabytes per per second.
- 41:57Okay? And remember we uh why we were
- 42:00looking at memory
- 42:02um
- 42:03uh
- 42:04how much things took to store,
- 42:07it's going to it's
- 42:08most obviously that, well, you just if
- 42:11you have a model that's too big it
- 42:12doesn't fit in your memory, that's not
- 42:14going to be fun. But also, it turns out
- 42:17that memory you need to move this
- 42:19memory, which takes time. So, actually
- 42:22the size of how large things actually
- 42:24influences uh speed as well.
- 42:27Okay, so um I'm going to talk through
- 42:30some operations, and I'm going to
- 42:32basically compute how long things are
- 42:34going to take, and introduce this idea
- 42:36of arithmetic intensity.
- 42:38Okay, so suppose I have a million uh
- 42:40dimensional uh vector of BF uh 16,
- 42:44and um and I'm going to just compute a
- 42:47value
- 42:49on this. So, re- remember ReLU is just
- 42:51uh max of X and zero uh done
- 42:53element-wise on the entire vector here.
- 42:58So, um
- 43:00I count two things. One is the number of
- 43:02bytes that were moved.
- 43:05So, um I have to read X,
- 43:08um in you know, copy it into the the um
- 43:13the accelerators,
- 43:14and each of this is going to be two
- 43:18because a BF16 is two bytes per float
- 43:20times N floats. So, that's two N, and
- 43:23then I'm going to write Y back. So,
- 43:25that's another two N.
- 43:27Okay? So, that's the number of bytes
- 43:28that have to be uh moved. And then, how
- 43:30many flops were done?
- 43:32Well, um each of these elements, I'm
- 43:35just comparing with zero, and that's it.
- 43:38So, that's N comparisons.
- 43:40Um
- 43:41Okay?
- 43:43So, now I look at the communication
- 43:46time, which is the number of bytes
- 43:48that were I needed to move,
- 43:50um you know, divided by the the speed of
- 43:54that movement,
- 43:55and that gives me the time, which is,
- 43:57you know, 1e-6
- 43:59uh you know, seconds.
- 44:01And what about the computation time?
- 44:02That's the flops divided by flops per
- 44:04second.
- 44:05So, that's um 1e-9.
- 44:09Okay.
- 44:10So, there's also going to be another
- 44:12important uh assumption, which generally
- 44:15we try to hold is that we overlap
- 44:17communication and computation. We're
- 44:20going to talk more about that when we
- 44:21talk more deeply about GPUs, but the
- 44:23idea is that in in this case, as as we
- 44:27don't sit here waiting for the the you
- 44:30know, the things to move, as soon as
- 44:32they're there, we start computing them,
- 44:34and then
- 44:35um we move them back. So, that this this
- 44:38movement and also the compute is
- 44:41happening at the same time.
- 44:43So, mathematically uh
- 44:46we're just going to assume that the
- 44:47total time is the max of the two,
- 44:49because we're going to assume that we
- 44:51can perfectly overlap them. In practice,
- 44:52it's not going to be perfectly
- 44:53overlapped, there's going to be some
- 44:54overhead, but this is uh um a good
- 44:56enough for now.
- 44:58Okay, so the total time,
- 45:00as we see here, is 1e-6.
- 45:04Okay.
- 45:05So, if you ask, you know, what is the
- 45:07bottleneck here?
- 45:08So, when the communication time is
- 45:10greater than the computation time, then
- 45:13we call it uh the algorithm memory
- 45:15bound,
- 45:16because you're spending most of your
- 45:17time just waiting for bits to show up.
- 45:19And when the com- computation time is
- 45:21greater than communication time, that's
- 45:22compute bound,
- 45:24because then you're actually, you know,
- 45:26your bottleneck is actually doing the
- 45:27compute.
- 45:29And so, in this case, uh what is ReLU?
- 45:34Rectified linear units. Oh, sorry. Is it
- 45:36memory bound or uh you know,
- 45:38compute bound? Memory bound.
- 45:39>> [laughter]
- 45:40>> A memory bound. Yeah.
- 45:43Um So, and it it's clear, right?
- 45:46Because, you know, the compute is way
- 45:48less than the communication.
- 45:51Okay.
- 45:53So,
- 45:54here's another way to see it, and this
- 45:56is where I'm going to define the
- 45:59intensity. So, what is intensity of an
- 46:01accelerator is essentially how much work
- 46:04can the accelerator do
- 46:06do per byte transferred?
- 46:08And for any given accelerator based on
- 46:11the spec sheet, you basically have um
- 46:14the the flops per second divided by the
- 46:18um the bytes per second.
- 46:20Okay? So, how much useful uh work can
- 46:22you do per byte that's moved? For H100s,
- 46:25that's uh 295.
- 46:28Okay? So, that means for every byte
- 46:31of
- 46:32um you can do 295
- 46:36um floating point operations.
- 46:38Okay? So, that's an intuitive number to
- 46:40uh kind of have in your head, about 300.
- 46:43Um okay, so now the arithmetic intensity
- 46:46of uh algorithm is how much actual work
- 46:49was done per byte for this workload.
- 46:51And if you look at it for the ReLU
- 46:53computation, it's flops over bytes,
- 46:56and it's uh actually it's so this is
- 46:58actually a
- 46:59a quarter, I guess. Um
- 47:01not half. So, the point is that it's
- 47:03very small,
- 47:04um 0.25.
- 47:07Okay, so now we can talk about
- 47:09bottlenecks through the language of
- 47:11intensity. So, if something is memory
- 47:14bound if the arithmetic intensity is
- 47:16smaller than uh the accelerator
- 47:19intensity, and compute bound if it's
- 47:21greater than accelerator intensity.
- 47:25All right, so these are equivalent, and
- 47:26if you look at the algebra, it's
- 47:28basically you have two fractions, and
- 47:29then you just kind of multiply and
- 47:31divide to you basically switch the terms
- 47:34around.
- 47:36Okay, so in this case,
- 47:38um
- 47:39you know, we're uh memory bound.
- 47:42So, in general, we're going to find
- 47:44ourselves in a situation where we're
- 47:45memory bound because data movement is
- 47:48expensive, and so if you can get higher
- 47:51arithmetic intensity, that's good.
- 47:54Okay.
- 47:55So, 0.25. If you see that number, if
- 47:57someone tells you your arithmetic
- 47:58intensity is 0.25, you should uh say,
- 48:01"Oh, this is this is really bad."
- 48:03Yeah. What's some typical arithmetic
- 48:05intensity for
- 48:08Yeah, so we'll get to that. Okay.
- 48:12All right. So, so okay, so one way to
- 48:16think about increasing arithmetic
- 48:17intensity is that let's just try to do
- 48:20more stuff uh per unit of byte moved.
- 48:23Okay, so the GELU is another activation.
- 48:26Um it kind of looks like this. It's
- 48:27some, you know, formula that is uh more
- 48:30uh um
- 48:31um doesn't
- 48:33uh you know, have zeros.
- 48:34Um
- 48:35and uh if you do this calculation, the
- 48:38number of bytes are moved back and
- 48:41forth, it's still 2N + 2N.
- 48:44Um
- 48:45and the flops
- 48:47here, it's about 20 um flops per
- 48:51element-wise GELU operation. So, it's
- 48:5320N. Right? So, the arithmetic intensity
- 48:55is, let's say, five. This is a kind of a
- 48:58crude estimate. And in this case, are we
- 49:01memory bound or compute bound?
- 49:10Memory bound. Memory bound.
- 49:12Right? Because five is still smaller
- 49:13than 295.
- 49:15Way smaller.
- 49:17So, even though GELU does a lot of uh a
- 49:20lot of work,
- 49:21um more work than ReLU, in the way that
- 49:24things are structured, it's still um
- 49:27it's still memory bound.
- 49:30Which means that if you were just
- 49:31computing ReLU and GELU, you think
- 49:34that, well, GELU is like so complicated,
- 49:37it must be really expensive, but
- 49:38actually it's exactly the same. Right?
- 49:40Because that's not where the bottleneck
- 49:42is.
- 49:46Okay, so now let's uh look at some uh
- 49:48linear operations. So, dot product.
- 49:50Okay, so dot product, you have a vector
- 49:52X, you have a vector W, size N, and you
- 49:55take the dot product. So, how many bytes
- 49:57are moved? So, you read X, which is 2N,
- 50:00read W, which is 2N, and then you write
- 50:02Y, which is a scalar, which is two.
- 50:05Okay, and the number of flops is you do
- 50:08the N multiplications and
- 50:11uh and
- 50:12N minus one additions, so that's 2N
- 50:13minus one.
- 50:16Okay, so um what's the arithmetic
- 50:18intensity? Oh, for this one it's uh
- 50:20about half.
- 50:22Okay, so which is also pretty bad, which
- 50:24means that hopefully by you get the
- 50:27idea, it's memory bound.
- 50:30Okay?
- 50:33So, what about a matrix vector product?
- 50:36Okay, so what I have this um X is a
- 50:38vector, W is a M by M matrix, um and you
- 50:41perform this product.
- 50:43Um how many of you think this will be
- 50:44compute bound?
- 50:46How many of you think memory bound?
- 50:48Okay.
- 50:49So, let's see.
- 50:51So, I'm going to read X, uh read, which
- 50:53is 2N, read W, which is 2N squared, and
- 50:56write Y, which is uh 2N. And there's uh
- 51:00the number of flops is basically you're
- 51:02you know uh doing N dot products, so
- 51:05that's N times the cost of doing a dot
- 51:07product.
- 51:09And uh the arithmetic intensity is, you
- 51:11know, barely barely higher.
- 51:14Um
- 51:16So,
- 51:17uh so this is,
- 51:18you know, um
- 51:20also memory bound.
- 51:21Okay.
- 51:22So, now let's talk about matrix
- 51:24multiplication. So, this is where things
- 51:26get um interesting. So, I have a M by M
- 51:28matrix, another M by M matrix, and I
- 51:30multiply them.
- 51:31And uh the number of bytes that were
- 51:33moved was um
- 51:36you know, 2N squared plus 2N squared,
- 51:39and then I have to write a 2N squared um
- 51:41bytes uh back to Y.
- 51:43And the number of flops here is N
- 51:46squared dot products.
- 51:49And then what's arithmetic intensity?
- 51:52It's 300, woohoo. Okay, so 340, and in
- 51:56general it's roughly N over three.
- 52:00Um and intuitively this makes sense
- 52:03because you're sending N squared things,
- 52:07but you're computing N cubed things.
- 52:10Right? So, the number of things you're
- 52:11computing over number of things you're
- 52:13sending is order N.
- 52:16And this gets better the larger you make
- 52:18the matrices.
- 52:19So, and in general,
- 52:21this is why when you hear people talk
- 52:24about, "Oh, we need to make large batch
- 52:26sizes or have large matrices." It's
- 52:28exactly this. If you if you're under uh
- 52:32the accelerator intensity,
- 52:34um
- 52:35making things smaller doesn't actually
- 52:37speed things up, right? It's all kind of
- 52:39the same. Whereas, if you get to the
- 52:41point where you're over uh this, then
- 52:43you're actually saturating your um you
- 52:45know,
- 52:46uh GPUs.
- 52:48Okay, so finally this is uh you know,
- 52:50compute bound.
- 52:52Okay.
- 52:53So, as long we have large matrices,
- 52:55we're actually pretty good. We're in uh
- 52:57compute bound, saturating the
- 52:58accelerator. And the question about
- 53:00earlier about, you know, what about
- 53:02transformers? Well, it turns out we'll
- 53:04see um
- 53:05in both your assignment, but also in the
- 53:07next lecture, that transformers are
- 53:09essentially big, you know, matrix
- 53:11multiplications with some things
- 53:13sprinkled in between. So, um
- 53:16the so that's good news from arithmetic
- 53:18intensity, and this is by design.
- 53:20Transformers are designed in a certain
- 53:21way to have high um arithmetic uh
- 53:24intensity.
- 53:26And one comment here, just to foreshadow
- 53:29the inference lecture, is that matrix
- 53:31vector products is essentially kind of
- 53:33what goes on when you're doing
- 53:34transformer inference. Right? Because
- 53:36inference, you're generating one token
- 53:38at a time,
- 53:39and so you only get to it's like a
- 53:41vector that you're trying to dot product
- 53:43with a with a matrix.
- 53:45Um and that's and that's where as we saw
- 53:47was uh memory bound.
- 53:50Whereas in training time, you get this
- 53:51whole sequence, and you're processing
- 53:53all at once.
- 53:56Okay.
- 53:58So, um note also that the intensity
- 54:00depends on the the precision. So, by
- 54:03default everything we're doing here is
- 54:05FP uh you know, 16.
- 54:08Um
- 54:09Okay, so also the to tie it to the other
- 54:11question about MFU, and the reason you
- 54:13might be getting
- 54:14um low MFU is that while you
- 54:18um might be doing
- 54:20>> [clears throat]
- 54:21>> um so, MFU is a a promised flops uh
- 54:25sorry, actual
- 54:26over promised. Right? So, if you have
- 54:29really large memory band uh bottlenecks,
- 54:32then you're not actually going to get
- 54:34very good you know, throughput even
- 54:35though um
- 54:37you know, you
- 54:38you know, the the promise is like if you
- 54:40didn't have memory bottlenecks, you were
- 54:41just like going through and doing all
- 54:43the computations of your model.
- 54:47Okay, uh final thing on arithmetic
- 54:49intensity, um
- 54:51roofline plots.
- 54:53So, there's a nice way to Yeah?
- 54:56Yeah.
- 54:58Generally speaking, these models tend to
- 55:00be memory bound, as they say, for most
- 55:02of the computation.
- 55:04But yet, the accelerators are 50% is not
- 55:08as good, and then maybe you get 70,
- 55:10maybe you get 80, that's really good.
- 55:12Mea- meaning that the accelerator is
- 55:14outsized compared to memory. Uh
- 55:17bandwidth.
- 55:18Um why is it like that? Why do these
- 55:21accelerators
- 55:22big just big Well, they're just idling
- 55:25or waiting for memory.
- 55:27So, so um the question is uh could you
- 55:30design maybe accelerators that had
- 55:32better um you know, characteristics?
- 55:35Um yeah, maybe when we talk about GPUs
- 55:37and understand a bit more how they work,
- 55:39we can talk about uh why this is.
- 55:42Um but if you have uh an answer, you
- 55:43should tell Jensen, and maybe he can um
- 55:46design a better um better hardware.
- 55:49>> [snorts]
- 55:49>> Um
- 55:49okay, so
- 55:51let's visualize the relationship between
- 55:53arithmetic intensity and performance.
- 55:55So, this plot basically plots the
- 55:57arithmetic intensity on the X axis. So,
- 56:00every
- 56:01um vertical uh slice here is corresponds
- 56:04to a particular, let's say, algorithm.
- 56:07And then we have these lines here, and
- 56:09each of these lines corresponds to,
- 56:11let's say, a particular accelerator.
- 56:13Maybe
- 56:14um you know, H100s or
- 56:16uh B200s or so.
- 56:19And so, uh what this shows is and then
- 56:21on the Y axis is um the flops per second
- 56:24that are realized.
- 56:26So, if your algorithm has low arithmetic
- 56:30intensity, like um value or dot
- 56:32products, you're going to be over here,
- 56:35which means that the flops per second
- 56:36realized is going to be um
- 56:39you know, not as high as your peak
- 56:41accelerator.
- 56:43And as you increase the memory
- 56:45uh arithmetic intensity, that's going to
- 56:48things are going to be better. You're
- 56:50going to be able to saturate your um
- 56:52your hardware up until a certain point.
- 56:55And after a certain point, you're
- 56:56compute bound, and uh you're
- 56:59obviously you can't exceed the peak
- 57:00flops um
- 57:03Okay?
- 57:04All right, so
- 57:06um let's go on here. Okay.
- 57:10So, um now I'm going to go back and talk
- 57:15about memory and compute for
- 57:18um the operations that we need in um in
- 57:22training.
- 57:23Right? So, so far we've done tensor
- 57:24operations, basically matmuls, and um
- 57:28you know, we saw basically how much
- 57:29memory it took and how much compute it
- 57:31took.
- 57:32Um and the interaction between them.
- 57:34So, now let's actually think about what
- 57:36it takes to train. So, here is uh our
- 57:39running example here. Um actually it's
- 57:42not a linear network, uh just a deep
- 57:44network. Um
- 57:45so, I'm going to consider a case where I
- 57:47have um a input, which is a B by D uh
- 57:51input. Um and a number of layers, where
- 57:54each layer is a D by D uh matmul.
- 57:58Um pa- and then that produces some
- 58:01um set of pre-activations, and then
- 58:04element-wise value, that produces some
- 58:06activations of the first layer, and then
- 58:08this is repeated again and again.
- 58:11Okay, and the output is just going to be
- 58:13the same size.
- 58:17Okay, so the deep network um I mean,
- 58:20just to see what it looks like in
- 58:21PyTorch, um just so it's basically a set
- 58:24of blocks.
- 58:25Each block uh
- 58:27is
- 58:29has a weight vector associ- uh sorry, a
- 58:31weight matrix associated with it.
- 58:33Okay?
- 58:34And um
- 58:38So, when you the number of parameters
- 58:40here is D squared times the number of
- 58:43layers, L.
- 58:45And when you run the the model on a
- 58:48batch of data,
- 58:49um what we're doing is we're going
- 58:51through the layers, and each layer um
- 58:54we basically apply the linear
- 58:57transformation, and then a point-wise uh
- 58:59value activation.
- 59:01Um and then we do that for all the
- 59:03layers. Okay? So, this is hopefully
- 59:05straightforward.
- 59:06Um
- 59:07very simple model.
- 59:09>> [clears throat]
- 59:09>> Um okay, so now let's talk about, you
- 59:11know, uh
- 59:13you know, gradients.
- 59:15So, in uh let's use an actually a sim-
- 59:17even simpler example. So, this is just
- 59:20um a simple linear model regression.
- 59:23Um
- 59:24so, we have a a vector 1 2 3, a weight
- 59:28uh a vec- weight vector um 1 1 1.
- 59:32Um
- 59:33and we take a dot product uh and we form
- 59:36this uh MSE loss.
- 59:38Okay? And what happens when we uh you
- 59:40know, take the the do the backward pass
- 59:43um
- 59:43is that
- 59:45um each of the the variables involved in
- 59:47this computation graph has a gradient
- 59:50that is um either set or not set. So,
- 59:53um
- 59:54the
- 59:56w.grad gets set to, you know, 1 2 3.
- 1:00:00Okay?
- 1:00:01So, this is just basic uh mechanics of
- 1:00:03uh you know, PyTorch. I think it should
- 1:00:05be familiar to um all of you.
- 1:00:08So now, the question is like how much uh
- 1:00:11compute does
- 1:00:13uh gradients you know, take?
- 1:00:16Okay. So, let's count flops for
- 1:00:18computing gradients.
- 1:00:19>> [clears throat]
- 1:00:21>> So, let's take a you know, simplified
- 1:00:23model
- 1:00:24um where you have um an X which is BID
- 1:00:29and um times a W1 matrix.
- 1:00:32Um well, we're going to it's sort of
- 1:00:34ignore this uh value for now uh just for
- 1:00:36simplicity times W2 which is the same D
- 1:00:38by D may same sa- shape D by D matrix.
- 1:00:42Okay, I'm going to use einsum here um
- 1:00:45uh for reasons that will become clear.
- 1:00:47Um so, the first thing you do in this
- 1:00:49linear network as you take X and you
- 1:00:50take W1. X is batch by input dimension.
- 1:00:55Uh W1 uh is in by out and you get a
- 1:00:58batch by out.
- 1:01:00Okay? And H2 takes H1 and W2 and it's
- 1:01:04this kind of the same story. These are
- 1:01:05just matmuls.
- 1:01:09So, and then you form a loss. I'm just
- 1:01:10setting it to some arbitrary just to get
- 1:01:12a number out.
- 1:01:14Okay.
- 1:01:15So, what happens in the in the backward
- 1:01:17pass? Um
- 1:01:18I'm going to call these retain gradients
- 1:01:20for debugging purposes which you'll see
- 1:01:22later. Uh you take the backward pass on
- 1:01:24the loss and the question is how much
- 1:01:27work did that uh how many floating point
- 1:01:29operations was that?
- 1:01:31Okay, so let's zoom in on one layer.
- 1:01:33Let's focus on the second layer here.
- 1:01:35So, this this uh layer.
- 1:01:37Um
- 1:01:39and so, the second layer takes H1 and
- 1:01:42just multiplies by a matrix W2 to get
- 1:01:45H2. And that's it.
- 1:01:47So, if you look at the num- uh the flops
- 1:01:50in the forward pass, this is just uh a
- 1:01:52matmul.
- 1:01:53And we just uh take the three dimensions
- 1:01:56and we multiply them together and BF16
- 1:01:58that's uh two bytes. Uh sorry, that's
- 1:02:01that's irrelevant. It's um this is just
- 1:02:03two because it's a addition and a
- 1:02:05multiplication.
- 1:02:06Okay.
- 1:02:07Um
- 1:02:08so, in the backward pass, you know, what
- 1:02:10does this look like? Okay, so in the
- 1:02:12backward pass, um if you remember your
- 1:02:14chain rule and your backprop algorithm,
- 1:02:17you have to compute two things. You have
- 1:02:19to compute um
- 1:02:21the the kind of backward message the the
- 1:02:23gradient of the loss with respect to
- 1:02:27your your kind of input
- 1:02:29um
- 1:02:30and you also have to compute the
- 1:02:32gradient with respect to your
- 1:02:33parameters.
- 1:02:36Okay? And what do those gradients look
- 1:02:38like? Um this is just by
- 1:02:40you know, the definition of uh these um
- 1:02:43I guess it's just the chain rule here
- 1:02:46um written in einsum notation. You take
- 1:02:48the
- 1:02:49the D loss D H2 which is H2.grad.
- 1:02:54That's a a batch by out uh you know,
- 1:02:57matrix.
- 1:02:58And you multiply it by um W2
- 1:03:02which is in by out.
- 1:03:04And
- 1:03:06and then um
- 1:03:07that gives you uh
- 1:03:09H1.grad which is D loss uh D H1.
- 1:03:12And and which is batch in.
- 1:03:15Okay? So, I always uh get this uh you
- 1:03:17know, confused whenever you see if you
- 1:03:19see, you know, learn calculus there's
- 1:03:21like, you know, one of them has a
- 1:03:22transpose and like I always forget which
- 1:03:24order to put them in. And einsum I think
- 1:03:26makes it very clear because if you we
- 1:03:29you can remember even just like looking
- 1:03:31at the scalar case that it has to look
- 1:03:33like H1.grad is H2.grad times W2 and the
- 1:03:37only question is, you know, what uh how
- 1:03:39do you index things? And you index
- 1:03:41things by basically looking at the shape
- 1:03:43of these uh in the named dimensions. And
- 1:03:46in this case, it's just a matrix
- 1:03:48multiplication where you are summing
- 1:03:51over the output, you know, dimension.
- 1:03:54Okay? And when you do that, we can check
- 1:03:57that so H1.grad was the thing that was
- 1:03:59computed by loss.backward and this is
- 1:04:02actually indeed the same thing that I
- 1:04:04kind of wrote out here just as a sanity
- 1:04:05check.
- 1:04:07Okay, so the other thing you have to
- 1:04:08compute is
- 1:04:09uh
- 1:04:10W the gradient with respect to the
- 1:04:12parameters. And that's uh
- 1:04:15D loss D H2 times H1.
- 1:04:18So, it's the same kind of backward
- 1:04:20message that's coming back but
- 1:04:21multiplying with other thing that you're
- 1:04:23not taking the
- 1:04:25you know, the gradient you know, with
- 1:04:26respect to.
- 1:04:27And you just write out the the
- 1:04:29dimensions and in this case, you're
- 1:04:32summing over um the the batch dimension.
- 1:04:37Okay.
- 1:04:38So,
- 1:04:40but the you know, the nice thing is that
- 1:04:42if you just look at this, we know how
- 1:04:44expensive uh you know, this is. The
- 1:04:47number of flops is essentially the
- 1:04:50product of all the dimensions. It
- 1:04:52doesn't matter which ones you're you're
- 1:04:54batching or not. It's it's basically you
- 1:04:55enumerate over the IJK and you you
- 1:04:58aggregate the way you aggregate is
- 1:05:00different but um
- 1:05:02the the flops is the you know, the same.
- 1:05:06Okay, so notice that the backward pass
- 1:05:08is exactly twice as expensive as the
- 1:05:10forward pass and this is because you
- 1:05:14have to take you know, compute two
- 1:05:16uh gradients. One with respect to the
- 1:05:18parameters the for each parameter and
- 1:05:20then one with respect to the input the
- 1:05:23other thing that's not the parameter.
- 1:05:27Okay?
- 1:05:29Maybe I'll pause any questions about
- 1:05:31that.
- 1:05:38>> [clears throat]
- 1:05:40>> Okay, so now that was just for W2.
- 1:05:43And you just need to apply this to all
- 1:05:45the parameters in network and if you put
- 1:05:47it all together,
- 1:05:48you will see that for this network,
- 1:05:52the forward pass is two times
- 1:05:55the number of data points which is B
- 1:05:56times the number of parameters flops.
- 1:05:59And the backward pass is twice of that
- 1:06:01which is four times the number of data
- 1:06:02points times number of parameters flops.
- 1:06:05And so, the grand total is 2 + 4 is 6
- 1:06:09and uh 6 times number of data points and
- 1:06:11parameters. So, this is where the 6 uh
- 1:06:14ND uh comes from that you might have
- 1:06:16seen in various places. It's just like
- 1:06:19uh counting forward and you know,
- 1:06:21backward.
- 1:06:24So, we did this for just these uh deep
- 1:06:27networks. Um but it turns out that this
- 1:06:29is actually a good approximation for uh
- 1:06:32transformers as well as soon as long as
- 1:06:34the uh context length isn't too large.
- 1:06:36If the context length is too large then
- 1:06:38you get the context length squared and
- 1:06:39that's
- 1:06:40uh more flops that isn't in this kind of
- 1:06:42accounting.
- 1:06:48Okay, so uh let's talk a bit about So,
- 1:06:51now we have gradients.
- 1:06:52Now we have to the other piece when you
- 1:06:54do training is we have to do
- 1:06:56optimization.
- 1:06:57So, uh like here's our deep network. Um
- 1:07:00I'm going to just so that we're I'm not
- 1:07:02just giving you uh what's in the
- 1:07:04assignment one, I'm going to use um the
- 1:07:07Adagrad optimizer which is um uh from
- 1:07:11this is from 2011. This is predates
- 1:07:13Adam. It's where you can think about it
- 1:07:15somewhere in between SGD and um Adam.
- 1:07:18And uh basically it's SGD where you um
- 1:07:23you know, look at the the you know, the
- 1:07:25second moments of the gradient.
- 1:07:28Um momentum is where you look at the
- 1:07:29first moment of gradients and um Adam is
- 1:07:32where you, you know, uh combine the the
- 1:07:34two of them. Um I'm not going to have
- 1:07:36time to go into the optimizer details
- 1:07:38here since we're focusing mostly on the
- 1:07:41uh the usage.
- 1:07:42So, we can define an um
- 1:07:45you know, optimizer.
- 1:07:46And
- 1:07:48and when you compute the the you know,
- 1:07:50the gradients, um
- 1:07:52uh
- 1:07:54and you compute gradients and then when
- 1:07:57you take an optimizer step, um
- 1:08:00if you're defining your new optimizer,
- 1:08:02so in the assignment one, you're going
- 1:08:03to uh uh implement Adam.
- 1:08:06Um for each of the parameter uh
- 1:08:10you know, groups which are, you know, in
- 1:08:11this case W1 or W2,
- 1:08:14um
- 1:08:14we look at there's something called the
- 1:08:17optimizer state which is uh of you know,
- 1:08:21storage that you can the optimizer uses
- 1:08:23while it's it's it's running.
- 1:08:26Um and what we're computing in Adagrad
- 1:08:29is uh something is the squared
- 1:08:32gradients. Um the sum of the squared of
- 1:08:34gradients.
- 1:08:35Um so,
- 1:08:37here we're getting it from the optimizer
- 1:08:39state. We're updating it with the
- 1:08:41current gradient and we're storing it
- 1:08:43back.
- 1:08:44And then after you update the the G2,
- 1:08:47then you update the parameters.
- 1:08:49>> [clears throat]
- 1:08:50>> Okay? So, in Adagrad it's you basically
- 1:08:53divide by the square root of the
- 1:08:54gradient you know, average gradient
- 1:08:56squared or sum of gradient squared
- 1:08:58rather.
- 1:09:01Okay, so um
- 1:09:05let's see. So, I'm actually going to uh
- 1:09:08maybe skip the
- 1:09:10you know, the training
- 1:09:12loop. Actually,
- 1:09:14wait.
- 1:09:15Hold on.
- 1:09:17I think I actually skipped something I
- 1:09:18didn't mean to.
- 1:09:19Okay, so that was uh the optimizer
- 1:09:23state. And now let's look at how much
- 1:09:25memory the optimizer is using or in
- 1:09:27general what is the memory usage. So,
- 1:09:30for parameters in this network, there's
- 1:09:33D squared parameters
- 1:09:35for each of the L layers, and each
- 1:09:38parameter takes two bytes if we're
- 1:09:39storing an FP16, so that's number of
- 1:09:41parameters.
- 1:09:42Activations,
- 1:09:44um
- 1:09:45this is
- 1:09:46uh you know
- 1:09:47two times
- 1:09:49um the
- 1:09:51which is for BF16
- 1:09:53you know, batch times D times the number
- 1:09:54of layers for every layer you have an
- 1:09:56activation. You have gradients,
- 1:09:59uh which is basically a copy of all the
- 1:10:01you know, the parameters. And this is
- 1:10:03all the BF16, and the optimizer state is
- 1:10:06for every
- 1:10:08actually this should be the
- 1:10:10I think this should be sorry, not this
- 1:10:12I'll fix this as a typo. This should be
- 1:10:13the number of parameters, not parameter
- 1:10:14memory.
- 1:10:15Um
- 1:10:16and this should be number of parameters.
- 1:10:17So, number of parameters
- 1:10:19uh times two, and then this is four
- 1:10:21times the number of parameters.
- 1:10:24And the reason for this is that it's
- 1:10:25customary to use FP32 for the optimizer
- 1:10:29states for stability reasons. Um
- 1:10:32obviously people have tried using uh
- 1:10:34BF16, and what tends up happening is
- 1:10:37that now you're taking squares and
- 1:10:38you're averaging all those steps, and it
- 1:10:40doesn't uh you know, it's not very
- 1:10:42stable.
- 1:10:43Um
- 1:10:45and
- 1:10:46for AdaGrad it's here we have four bytes
- 1:10:48per parameter for storing the optimizer
- 1:10:50state. Adam, you store the first order
- 1:10:52bit
- 1:10:52uh and the second order moments, so
- 1:10:54that's eight bytes.
- 1:10:56So, if you think about the optimizer
- 1:10:57state is actually a lot of the memory um
- 1:11:01uh
- 1:11:01you know, used.
- 1:11:03Um one note is that
- 1:11:06you know, remember memory serves kind of
- 1:11:08two purposes. One is like how much you
- 1:11:10have to store this thing in your HBM.
- 1:11:13But the other thing is that it has to be
- 1:11:14shipped to the um
- 1:11:17uh to
- 1:11:19the [clears throat] um
- 1:11:20to to the accelerators, and in general
- 1:11:24um the optimizer state is not really the
- 1:11:26bottleneck for for compute. So, the
- 1:11:28amount of memory here is not really uh
- 1:11:31so important for um you know,
- 1:11:33performance in terms of speed, but it is
- 1:11:36uh means that you can't fit large models
- 1:11:38in your your memory.
- 1:11:41>> [clears throat]
- 1:11:42>> Okay. So, to put it um you know,
- 1:11:44together
- 1:11:45uh as we mentioned, the number of
- 1:11:47parameters here is D times D times L,
- 1:11:49and the number of flops is D times
- 1:11:51um number of tokens or number of data
- 1:11:53points times number of parameters.
- 1:11:55So, for transformers, this is going to
- 1:11:57be uh a bit more complicated, but you're
- 1:11:59going to do that in assignment one, and
- 1:12:01you do it more um carefully.
- 1:12:04Okay, I'm going to skip the training
- 1:12:05loop. This is just a sort of a general
- 1:12:07um
- 1:12:08um
- 1:12:08kind of review.
- 1:12:09Two things I want to quickly touch on
- 1:12:11before we conclude. One is that
- 1:12:14as we see, memory does uh
- 1:12:17have an important effect both on the
- 1:12:19ability to store large models, but also
- 1:12:21sometimes um your speed. And so, in
- 1:12:24general you want to reduce your memory
- 1:12:26usage. So, there's two
- 1:12:27um things that people you know,
- 1:12:29typically do. One is uh gradient
- 1:12:31accumulation.
- 1:12:33Um so, in general you want to use batch
- 1:12:36sizes that are large enough uh to
- 1:12:39improve stability up to a a critical
- 1:12:41batch size which Tatsu will talk about
- 1:12:43later.
- 1:12:44Um
- 1:12:45and as we saw that activation memory
- 1:12:48scales with batch size. So, you might
- 1:12:50going to run out of memory at some point
- 1:12:52if you have too large batch sizes. So,
- 1:12:54gradient accumulation says, well
- 1:12:57you compute, you have these micro
- 1:12:59batches, and you compute the gradient on
- 1:13:01the micro batches, and then you just
- 1:13:03accumulate the gradients. You don't zero
- 1:13:04out the gradients, and every
- 1:13:06um batch size over micro batch size
- 1:13:08steps, you update the parameters and a
- 1:13:10zero out the gradients.
- 1:13:12Okay? So, this is actually a very simple
- 1:13:14code change um which allows you to save
- 1:13:17um you know, on compute.
- 1:13:19So, the other thing I'll quickly mention
- 1:13:21is activation checkpointing. So, in
- 1:13:23training
- 1:13:24we need to
- 1:13:26uh uh
- 1:13:27store
- 1:13:28in general the activations of all the
- 1:13:30layers. This is done by you know,
- 1:13:32default.
- 1:13:33Um actually interestingly for inference,
- 1:13:35we don't need to compute uh the
- 1:13:37gradients, so we only need to store the
- 1:13:38current layers activations.
- 1:13:40Um but for training
- 1:13:43the memory usage is
- 1:13:45um
- 1:13:46um B times D times um
- 1:13:49two [clears throat] times L.
- 1:13:52And
- 1:13:53um the question is can you reduce this?
- 1:13:56So, the activation checkpointing also
- 1:13:58known as gradient checkpointing or
- 1:13:59rematerialization. The key idea here is
- 1:14:01that um in the forward pass, you just
- 1:14:04keep the activations for only a subset
- 1:14:06of the layers.
- 1:14:07And in the backward pass, you recompute
- 1:14:09the missing activations from the last
- 1:14:11checkpoint. That's why it's called
- 1:14:13checkpointing. And this is a general
- 1:14:15trick that you see in you know, systems
- 1:14:17which is trading off if you want to
- 1:14:19reduce memory, you can just recompute
- 1:14:21things.
- 1:14:22Um so, here we're going to define this
- 1:14:25uh to operationalize this. This is
- 1:14:27actually fairly
- 1:14:28um
- 1:14:29you know,
- 1:14:30uh
- 1:14:31uh
- 1:14:32easy actually. I didn't
- 1:14:35Okay, maybe I didn't see the Okay, so
- 1:14:37basically what you do here is you have
- 1:14:41the same model except for
- 1:14:43um
- 1:14:44you just add torch.utils.checkpoint
- 1:14:46on a layer.
- 1:14:48So, that means do this computation, but
- 1:14:50don't store any of the intermediate um
- 1:14:52um
- 1:14:53you know, activations. Only store what
- 1:14:56is uh needed. So, this means that um
- 1:15:01if you store all the activations,
- 1:15:03remember in your deep network, you have
- 1:15:06the thing that's pre the ReLU, and the
- 1:15:07thing that's after the ReLU.
- 1:15:09Right? So, that's a lot of of storage,
- 1:15:12and instead if you do activation
- 1:15:13checkpointing on those blocks where each
- 1:15:16block has a linear and a ReLU, you don't
- 1:15:18store the pre ReLU, and you save
- 1:15:20basically half. Uh you can get away with
- 1:15:23half the memory.
- 1:15:24And then at the when you're doing the
- 1:15:26backward pass, um
- 1:15:28you need G3, but you can compute G3
- 1:15:30easily from H2.
- 1:15:33Okay, you can go farther than this.
- 1:15:35Um you can say
- 1:15:37well, I can store all the layers, but
- 1:15:40you know, in extreme case, I can just
- 1:15:43store no layers.
- 1:15:44Um and so, that will be memory
- 1:15:47maximally memory efficient. The only
- 1:15:49thing is that the compute is going to be
- 1:15:51L squared because every for every one of
- 1:15:53these layers you have to start from the
- 1:15:55beginning.
- 1:15:56Um and maybe a sweet spot is if you
- 1:15:59store the checkpoints at square root L
- 1:16:01layers, that means your activation
- 1:16:03memory is square root of L, and your
- 1:16:06your com recomputation overhead is also
- 1:16:09um you know, uh square root of L. So,
- 1:16:12that's uh that's balanced.
- 1:16:15Okay, so um to summarize this lecture
- 1:16:18um everything is operating on on
- 1:16:20tensors, parameters, gradients,
- 1:16:21activations, optimizer states, data.
- 1:16:25Um we introduce this Einops, which
- 1:16:26hopefully you guys can embrace as a way
- 1:16:28to think about tensor operations.
- 1:16:31Um six times number of data points times
- 1:16:32number of parameters is a formula which
- 1:16:35now we have demystified as number of
- 1:16:37flops per
- 1:16:39um
- 1:16:39for training step.
- 1:16:41And uh
- 1:16:43actually if this is training step, this
- 1:16:44should be batch size. Um
- 1:16:46and then we talked about arithmetic
- 1:16:48intensity and roofline analysis, which
- 1:16:50allows us to diagnose whether a
- 1:16:52computation is memory bound or uh
- 1:16:54compute bound. Um matrix multiplications
- 1:16:57are compute bound. Basically everything
- 1:16:59else is memory bound.
- 1:17:01Um and then finally, gradient
- 1:17:03accumulation activation checkpointing
- 1:17:05are ways to reduce the memory
- 1:17:07and by reducing the memory, uh that
- 1:17:09allows you to use um bigger uh batch
- 1:17:12sizes.
- 1:17:14Okay, so that's it for today's lecture.
- 1:17:16So, next week uh Tatsu will talk about
- 1:17:18architectures.
About this transcript
This page contains the full transcript of YouTube transcript (kuYAsz7zspQ) , generated from the public captions YouTube serves with the video. The transcript has 12,028 words across 2,082 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.