YouTube2Text

YouTube transcript (kuYAsz7zspQ) — Transcript

12,028 words · 2,082 segments · language en · Watch on YouTube

Full transcript

  1. 0:05I hope everyone is staying dry. Um
  2. 0:08I'm not.
  3. 0:09So, as I mentioned last time, the Marine
  4. 0:11project had a
  5. 0:14which was running and it finished and it
  6. 0:17actually matched the the forecast. So,
  7. 0:19remember we were running these each of
  8. 0:21these curves is
  9. 0:23essentially a iso flops curve, which is
  10. 0:26a bunch of smaller model runs
  11. 0:28and you try to find the computer optimal
  12. 0:30point. You fit a scaling law and this
  13. 0:34was the point where
  14. 0:36we predicted the loss and we ran the
  15. 0:39loss the model and it got lost within
  16. 0:410.05. So, I thought that was pretty cool
  17. 0:44and if you extrapolate out to GPT-5
  18. 0:47level performance, this is the loss you
  19. 0:49get. Of course, your mileage might
  20. 0:52vary depending on how these scaling laws
  21. 0:55are.
  22. 0:56Okay, just wanted to share share that
  23. 0:58news.
  24. 1:00Um so, last lecture
  25. 1:02I gave an overview of the entire class
  26. 1:05and we are talking about tokenization,
  27. 1:07which is going to be on the first
  28. 1:08assignment. Um today I want to talk
  29. 1:11about um
  30. 1:13um resource accounting, which is going
  31. 1:16to be more on the systems side of
  32. 1:18things.
  33. 1:19Um so,
  34. 1:21uh to recall,
  35. 1:24the the main thing we're trying to do is
  36. 1:26um train the best model we can given a
  37. 1:29finite set of resources, which could be
  38. 1:31a compute, memory, sometimes data, but
  39. 1:34that's not really going to be a limiting
  40. 1:36factor for us in this class.
  41. 1:39And our goal is to simply to maximize
  42. 1:42the computational efficiency of our
  43. 1:45training.
  44. 1:46So, before you can optimize the
  45. 1:48computational efficiency, we need to
  46. 1:50understand the efficiency of a given
  47. 1:52computation and for that we need to
  48. 1:55understand the compute and memory
  49. 1:56characteristics.
  50. 1:58Um just to give you a taste of the type
  51. 2:00of questions you will hopefully be able
  52. 2:02to answer by the end of the class.
  53. 2:04Um so, here's a question. How long would
  54. 2:06it take to train a 70 billion parameter
  55. 2:08model on 15 trillion tokens on
  56. 2:101024
  57. 2:12Actually, this should be H100 cell.
  58. 2:15Um
  59. 2:16So, how do you answer that? Well,
  60. 2:18there's a formula which we'll talk about
  61. 2:19how you can get the number of flops to
  62. 2:22be be six times the number of parameters
  63. 2:24times the number of tokens. Um we can
  64. 2:27look up the spec sheet to see how fast
  65. 2:29the H100 is.
  66. 2:31Um we have this thing which called MFU,
  67. 2:33which we'll talk about 0.5.
  68. 2:36Um then you can estimate the number of
  69. 2:38flops
  70. 2:39um that you need
  71. 2:41that you the hardware gives you per day
  72. 2:44um and then you can compute the number
  73. 2:46of days. So, the number of days is 143.
  74. 2:49Okay.
  75. 2:50Um here's another question. What's the
  76. 2:51largest model you can train on H1 eight
  77. 2:54H100s using Adam W?
  78. 2:58Um so,
  79. 3:00uh you can look at Well,
  80. 3:03H100s have 80 GB of HBM
  81. 3:08memory.
  82. 3:09Um the number of bytes a per parameter,
  83. 3:13which is 2 + 2 + 4 + 4.
  84. 3:17We'll explain where that comes from.
  85. 3:19And then the number of parameters is um
  86. 3:23that you can get is going to be about 53
  87. 3:26billion. So, there's some caveats here
  88. 3:28that we don't count the activations,
  89. 3:31which depends on the batch length and
  90. 3:33batch size and the sequence length.
  91. 3:35So, this is all very rough back of the
  92. 3:37envelope calculations, but hopefully by
  93. 3:39the end of this class you'll understand
  94. 3:41where these come from and the point is
  95. 3:43not to
  96. 3:45precisely calculate every single thing,
  97. 3:48but just get the rough shape of things.
  98. 3:53Um okay. So, last time I talked about
  99. 3:57knowledge and what you can take away
  100. 3:59from this class.
  101. 4:00Um mechanics, which are how things work.
  102. 4:03So, today that will be pretty
  103. 4:04straightforward. The mechanics are just
  104. 4:06how PyTorch work, how tensors work. It
  105. 4:09should be fairly
  106. 4:11There's no magic here.
  107. 4:13Um the mindset I want you to impart on
  108. 4:16you is that resource accounting is going
  109. 4:18to be very crucial and while everyone to
  110. 4:22get in habit whenever you write you
  111. 4:24know, a piece of line of code, think
  112. 4:26about you know, the performance
  113. 4:28characteristics.
  114. 4:30And then finally, intuitions. Here we're
  115. 4:32just going to get a sense of the the
  116. 4:35resources, how they're spent. There's
  117. 4:37going to be no ML magic today. I'll
  118. 4:39leave that to Tatsu for the next
  119. 4:41lecture.
  120. 4:43Okay.
  121. 4:44So, let's get into things. So, let's
  122. 4:46start bottom up and start building up.
  123. 4:49So,
  124. 4:50what is at the bottom?
  125. 4:51The the bottom are tensors. So, tensors
  126. 4:53are the building block of storing
  127. 4:55everything.
  128. 4:56If you have parameters, gradients,
  129. 4:58optimizer states, data, activation,
  130. 5:01everything essentially is a tensor.
  131. 5:03Um so, you know, for example, you can
  132. 5:06take a look at the DeepSeek
  133. 5:083.2 model
  134. 5:10and you see that the model itself is a
  135. 5:13bunch of different tensors. Each tensor
  136. 5:14has some shapes and also some precision,
  137. 5:17which I'll talk about
  138. 5:18later.
  139. 5:21And
  140. 5:22so, as as you know,
  141. 5:24tensors subsume vectors, matrices
  142. 5:28and
  143. 5:29and can be generalize to any number of
  144. 5:32entities.
  145. 5:33Okay.
  146. 5:35So, let's talk about how
  147. 5:37much tensors take
  148. 5:39to store.
  149. 5:40Um
  150. 5:42So, it depends on the the type of
  151. 5:45tensor. So, in general, we're going to
  152. 5:47be dealing with tensors that store
  153. 5:49floating point, but tensors can also
  154. 5:51store
  155. 5:52you know, integers
  156. 5:54and other types.
  157. 5:56Um so, for floating point,
  158. 5:58um
  159. 5:59uh typically whenever you talk about
  160. 6:02float, I think the standard people refer
  161. 6:05to as float what is called float 32.
  162. 6:08Um so, a float 32, if you break it down,
  163. 6:12has 32 bits. One of the bits is a sign,
  164. 6:15eight of bits are the exponent, which
  165. 6:17gives you dynamic range and
  166. 6:20the rest is the the mantissa or the
  167. 6:22fraction, which gives you kind of
  168. 6:25you know, variation.
  169. 6:26Okay, this is also known as FP32
  170. 6:30or
  171. 6:31you know, single precision.
  172. 6:33Um so, and the term single precision
  173. 6:36comes from the fact that back in the day
  174. 6:37when you're doing scientific computing,
  175. 6:39float 32 was sort of like a baseline. It
  176. 6:42was just like you expect if you someone
  177. 6:44gave you a float, you would kind of
  178. 6:45expect it to be at least single
  179. 6:46precision and if you want
  180. 6:49um
  181. 6:50more precision, then you can get double
  182. 6:53precision as that's you know, float 64.
  183. 6:56But in deep learning, we're kind of
  184. 6:57going the other way because even 32 is a
  185. 7:01lot and the types of computations that
  186. 7:03we want to do don't demand the high
  187. 7:06precision that some kind of numeric
  188. 7:08simulations do.
  189. 7:10Um okay. So, before we get to other
  190. 7:12types, let's just look at a float 32.
  191. 7:15Okay, so let's construct a 4 by 8
  192. 7:17matrix.
  193. 7:18By default, the type of a tensor you
  194. 7:21create is float 32. So, if you want
  195. 7:23something else, you should declare it.
  196. 7:26Um and the memory usage is just the
  197. 7:28number of elements times the element
  198. 7:30size here, which is 4 bytes
  199. 7:33for a 32-bit
  200. 7:35number
  201. 7:36and that's going to be 128 bytes.
  202. 7:39Okay.
  203. 7:40Um
  204. 7:42just to give you kind of perspective.
  205. 7:44So, in GP3, which is a fairly old model,
  206. 7:46one of the matrix in the feed forward
  207. 7:49layer is
  208. 7:51um
  209. 7:51about
  210. 7:53you know, 2.3 GB.
  211. 7:56Okay, so these tensors can be you know,
  212. 7:58get quite big and this is by far even
  213. 8:00the biggest one that one can imagine.
  214. 8:04Okay, so
  215. 8:06So, since we're interested in
  216. 8:08efficiency,
  217. 8:10we want to generally reduce the amount
  218. 8:12of storage um
  219. 8:14and and we'll see that as you reduce the
  220. 8:17precision, um you actually save my
  221. 8:20memory and you also save time because
  222. 8:22operating on uh
  223. 8:24you know,
  224. 8:2516 bits is going to be faster.
  225. 8:28Um let's say
  226. 8:29twice as fast, but not always it sort of
  227. 8:32depends, but um
  228. 8:34and and then memory
  229. 8:36by reducing memory, we'll see later that
  230. 8:39actually reducing memory can save time
  231. 8:40as well,
  232. 8:42which is maybe less obvious, but it will
  233. 8:44hopefully become clear.
  234. 8:46So, the the obvious thing is you say,
  235. 8:49okay, let's take away half the bits. Now
  236. 8:51you have float 16.
  237. 8:52So, float 16 says you have a sign, you
  238. 8:54have only five bits of exponent and then
  239. 8:56the rest is the the mantissa.
  240. 8:59Um so,
  241. 9:01fine, fifth float 16 is is is good,
  242. 9:05except for
  243. 9:07you know, its dynamic range is
  244. 9:09kind of poor. Right? So, even if you
  245. 9:11have, let's say, try to construct a 1E
  246. 9:14minus 8 tensor, then that is actually
  247. 9:17just zero.
  248. 9:19So, you can't really represent very big
  249. 9:20numbers and you can't represent very
  250. 9:22small numbers.
  251. 9:24Um and the reason is that this exponent
  252. 9:26is you know, it's only
  253. 9:29you know,
  254. 9:30you know, five bits of of exponent from
  255. 9:33compared to eight.
  256. 9:35So,
  257. 9:36if you train with FP16, which people did
  258. 9:40back back in the you know, day,
  259. 9:44you will get instability. You will you
  260. 9:46get underflow, you get overflow, you'll
  261. 9:48get NaNs. It's it was it's pretty
  262. 9:51challenging.
  263. 9:53So, then
  264. 9:55so, BFloat16
  265. 9:57was invented. So this was developed in
  266. 10:00actually 2018 to address this issue.
  267. 10:03And the observation was that well, let's
  268. 10:07not compromise on
  269. 10:09the number of bits. It's it's it's the
  270. 10:10number of bits is going to be the same
  271. 10:11as FP16, but we're going to shift some
  272. 10:14of the bits from the mantissa to the
  273. 10:16exponent.
  274. 10:18So that means it has more dynamic range
  275. 10:20than float you know uh
  276. 10:2416.
  277. 10:25Um
  278. 10:26and it actually has the same dynamic
  279. 10:28range as float 32.
  280. 10:30But of course the the resolution is
  281. 10:32worse because there's no free lunch
  282. 10:34here.
  283. 10:35But it turns out that in a lot of deep
  284. 10:37learning applications,
  285. 10:38this is well worth the trade-off. You
  286. 10:40want the dynamic range to not overflow
  287. 10:43and underflow and because things are
  288. 10:46kind of sloppy and stochastic anyway,
  289. 10:48you don't need that much resolution.
  290. 10:52Okay?
  291. 10:55So
  292. 10:56let me actually skip over this this
  293. 10:58part.
  294. 10:59So
  295. 11:00So okay, so to summarize what are the
  296. 11:03what are the implications for training?
  297. 11:05So you can absolutely train with float
  298. 11:0732.
  299. 11:08And if you're training a small model,
  300. 11:11you probably just and you don't want to
  301. 11:12worry about it just you know float 32 is
  302. 11:14fine. But it requires four bytes of
  303. 11:18um
  304. 11:19uh of sorry, four bytes of memory per
  305. 11:24you know float. And that can take up a
  306. 11:27lot of memory. Um and if you
  307. 11:30train with float 16 then that's going to
  308. 11:34be too risky. So float 30 16 a BF 16 is
  309. 11:39sort of a sweet spot. Um even BF16 can
  310. 11:43be kind of risky as well maybe see in a
  311. 11:46little bit.
  312. 11:48Um one thing that people have has become
  313. 11:52kind of common practice is to use mixed
  314. 11:54precision training.
  315. 11:56Um and
  316. 11:58actually let me
  317. 12:00I think these
  318. 12:02Okay, let me compile this again.
  319. 12:05I think these are stale. So mixed
  320. 12:07precision training is uh where some of
  321. 12:11the computations use
  322. 12:14some precision and for some other
  323. 12:15computations use other precision. So in
  324. 12:18general as a general rule,
  325. 12:20BF
  326. 12:22you know 16 is
  327. 12:24what you would use for parameters
  328. 12:27activations and and gradients. And for
  329. 12:30optimizer states, you would use FP32.
  330. 12:32We'll discuss a little bit more of that
  331. 12:34about that later.
  332. 12:36And to do invoke mixed precision
  333. 12:38training, so PyTorch has a AMP library.
  334. 12:40We're not going to talk too much about
  335. 12:41this, but you basically um
  336. 12:44wrap your code and
  337. 12:47PyTorch has the library takes care of it
  338. 12:49automatically for you in the sense that
  339. 12:52it tries to cast things into
  340. 12:54BF16 when it's safe. So for example,
  341. 12:56matmuls are generally safe, but if you
  342. 12:58try to do
  343. 12:59exponentiation, then it will try to
  344. 13:02leave things as
  345. 13:03FP32.
  346. 13:06Okay, so we could do BF16 I think is
  347. 13:08probably where this class will end. But
  348. 13:11if you're feeling very adventurous, you
  349. 13:14can go farther. So FP8 this was
  350. 13:17introduced four years ago.
  351. 13:19Um it actually has been you know
  352. 13:21standardized. So if you look at FP8,
  353. 13:24there's actually two versions
  354. 13:26because depending on if you need more
  355. 13:29dynamic range or more resolution. Um and
  356. 13:33um there's there's you know two versions
  357. 13:35of them.
  358. 13:36Um and so we're not going to talk about
  359. 13:38that, but you know Nvidia has some you
  360. 13:41know the transformer engine supports
  361. 13:42FP8. And well most recently, you can
  362. 13:45actually go down to FP4. So last year
  363. 13:48Nvidia developed NVFP4 and there's only
  364. 13:51four bits per value. So if you just to
  365. 13:54so we're on the same page, four bits is
  366. 13:55not a lot. Like I can write all the
  367. 13:57values down on a single line here
  368. 14:00between minus six and and six. Um so
  369. 14:03that's not very much precision. Now
  370. 14:05there's a little bit of a cheat here,
  371. 14:07right? Because if you just naively only
  372. 14:09use these values,
  373. 14:10um you're you're not going to be able to
  374. 14:12train
  375. 14:13very very well. So what actually this
  376. 14:16means is that you every value
  377. 14:18you can have this four bits of freedom.
  378. 14:22But there are blocks and each block can
  379. 14:24be scaled up and down accordingly. So
  380. 14:26you can actually represent more values,
  381. 14:28but you can't
  382. 14:29represent the full
  383. 14:32dynamic range for every single value.
  384. 14:35And there was a model released
  385. 14:38actually this year, the NeMo-3 Super
  386. 14:40which was trained in FP4 which I think
  387. 14:42is a pretty pretty cool.
  388. 14:45Okay. So some of this you it's just good
  389. 14:49to know and some of this you can't even
  390. 14:51even touch. It's not like you create a
  391. 14:53tensor and you call call it FP
  392. 14:56you know it FP4. A lot of this is done
  393. 14:59under the hood by Nvidia's
  394. 15:02you know software stack.
  395. 15:05Okay, this was a bit of a kind of a long
  396. 15:08digression, but it's it's maybe helpful
  397. 15:10to appreciate that
  398. 15:12the intricacies of you know precision
  399. 15:13here.
  400. 15:14Okay, any questions before I move on?
  401. 15:17Yeah. Sorry
  402. 15:19there's a question like when you have a
  403. 15:20block you're scaling all those same
  404. 15:22values like like you have a tensor by
  405. 15:24that block something like that.
  406. 15:26The ratios of all of them will still be
  407. 15:28in FP4.
  408. 15:30Yeah, so the question is just like
  409. 15:32scaling with the maximum value is to
  410. 15:34have more specificity.
  411. 15:36Yeah, so the question is okay, just to
  412. 15:38explain the block a bit more. So you
  413. 15:40have let's say a block. Within that
  414. 15:42block you can vary
  415. 15:45up
  416. 15:46within the four bits. And in addition,
  417. 15:48all those values can be scaled up and
  418. 15:50down according to how many of your bits
  419. 15:52your scaling factor is. So if you look
  420. 15:55at an individual value,
  421. 15:58you actually get more than four bits of
  422. 16:00dynamic range, but it's just like you
  423. 16:02can't have this value be way over here
  424. 16:04and the next neighboring value way down
  425. 16:08Okay, so
  426. 16:09uh one question about like
  427. 16:11what about like one bit?
  428. 16:12Uh
  429. 16:13couldn't BF16 be pushed to a bit one bit
  430. 16:15and all of that? Yeah, so there's a
  431. 16:17problem with training obviously.
  432. 16:19Right, right. So so maybe this is a good
  433. 16:20point. So the question is about one bit
  434. 16:23because you can't really go lower than
  435. 16:24that.
  436. 16:26So
  437. 16:28there's a difference between training
  438. 16:29and inference. So a lot of the low bit
  439. 16:31stuff
  440. 16:33if we talk about quan- I think we'll
  441. 16:34talk about quantization later. Is that
  442. 16:36you train a model at
  443. 16:38on and you know maybe even BF16 and then
  444. 16:42you quantize into like let's say one or
  445. 16:44two bits. Um and that is much easier
  446. 16:47than training a one bit
  447. 16:50language model which I don't think
  448. 16:51anyone has
  449. 16:53you know done. Maybe it's possible, but
  450. 16:55I don't think anyone has trained
  451. 16:56anything credible there.
  452. 16:59Okay.
  453. 17:00Let us go on then. So we talked about
  454. 17:03tensors and the memory calculus is
  455. 17:06pretty simple just the number of
  456. 17:07elements times however much memory each
  457. 17:09element takes up.
  458. 17:11Um so
  459. 17:13you know by default the tensors you
  460. 17:15create in PyTorch are going to be on CPU
  461. 17:18and
  462. 17:20um and of course you have to if you want
  463. 17:24things to go fast, you want to move them
  464. 17:25to GPU. And actually so there's a bit of
  465. 17:30a I have a slight issue with the slides
  466. 17:32with that. They were executed on my
  467. 17:33laptop which means that I don't have
  468. 17:35GPU. So some of the code I'll just show
  469. 17:37but not um
  470. 17:39execute. Um
  471. 17:41so
  472. 17:42to okay, let's see.
  473. 17:45This is maybe not that the interesting,
  474. 17:47but I think everyone knows how to move
  475. 17:50um
  476. 17:51tensors to GPU. But just remember to do
  477. 17:53that otherwise you won't get your
  478. 17:55speed ups.
  479. 17:58Okay, so we talked about memory of
  480. 18:00tensors which is very straightforward.
  481. 18:02Now let's talk about computing with
  482. 18:04tensors.
  483. 18:05So before talking about flops accounting
  484. 18:07for um
  485. 18:09the the tensor operations,
  486. 18:11let's take a little bit of a digression
  487. 18:13to talk about einsums. So how many of
  488. 18:16you have familiar used einsums before?
  489. 18:19Okay, so like maybe two thirds. Okay,
  490. 18:22good. So um
  491. 18:24the motivation behind einsums for those
  492. 18:26of you who might not be indoctrinated is
  493. 18:30it's very easy to mess up or I I find it
  494. 18:33very confusing to
  495. 18:35look at code such as this and you you
  496. 18:39have X and Y and then you have like
  497. 18:42transpose minus two and minus one and
  498. 18:45you're trying to figure out like what
  499. 18:47what minus two and minus one is. Um
  500. 18:50And so this is kind of maybe
  501. 18:54the motivation
  502. 18:56for um
  503. 18:58you know using variable and names rather
  504. 19:00than indices.
  505. 19:02So einsums is a library for manipulating
  506. 19:04you know tensors where the dimensions
  507. 19:05are named. And this is inspired by
  508. 19:07Einstein summation notation. There's a
  509. 19:10nice tutorial which you can go through.
  510. 19:12I'm just going to cover some of the you
  511. 19:15know basics.
  512. 19:16Um
  513. 19:17So basically the way to think about
  514. 19:19einsum is a generalized matrix
  515. 19:21multiplication with
  516. 19:23you know good bookkeeping. So here's an
  517. 19:26example. So I have
  518. 19:28um
  519. 19:29a matrix 3 by 4. I have a 4 by 3 matrix
  520. 19:33and if I do the matmul,
  521. 19:35this is you know this is actually you
  522. 19:38know pretty nice. It's pretty easy to
  523. 19:40understand.
  524. 19:41Uh in einsums, basically you say
  525. 19:44X has two dimensions, the row and the
  526. 19:47column which I'm going to name seq1 and
  527. 19:49hidden. I'm going to have Y that's a
  528. 19:51matrix which has also rows and columns,
  529. 19:54which I'm going to name um
  530. 19:56hidden and seek two. And um I'm going to
  531. 20:00produce um a um tensor or here a matrix
  532. 20:04where um the dimensions are indexed by
  533. 20:07seek one and seek two. And anything that
  534. 20:09is not mentioned here, the hidden, uh
  535. 20:12gets summed out.
  536. 20:13Okay? So, the way this works is I'm
  537. 20:15going to sum over enumerate over all
  538. 20:18possible values of all the variables uh
  539. 20:22that
  540. 20:23occur here, which are seek ones, hidden,
  541. 20:26and seek two.
  542. 20:28And I'm going to basically um index into
  543. 20:32X, index into Y, multiply them, and dump
  544. 20:34them accumulate them into um the result
  545. 20:37Z uh sub seek one seek two.
  546. 20:41Okay?
  547. 20:43All right. So, you say this is
  548. 20:46Come on, this is much easier than this,
  549. 20:48right? Um so,
  550. 20:51Okay, let's try a more complicated
  551. 20:52example. So, um here's the example I
  552. 20:54showed
  553. 20:55you know, before. Now we have a a
  554. 20:57tensor. It's 2 by 3 by 4 and another
  555. 21:01tensor 2 by 3 by 4. And if I were doing
  556. 21:05things um kind of the old way, basically
  557. 21:08I am what I'm doing is uh transposing
  558. 21:11these last two and then um
  559. 21:14because uh at implicitly sort of batches
  560. 21:17the the dimensions that are not the the
  561. 21:19matmul dimensions, um
  562. 21:22then you get the answer. Okay? So, you
  563. 21:24have to kind of reason about this a bit.
  564. 21:26Einops makes this very uh clear. It says
  565. 21:29there's a batch dimension, there's seek
  566. 21:31one uh hidden, seek two hidden, and I'm
  567. 21:34just going to produce batch uh seek one
  568. 21:36seek two.
  569. 21:37Okay?
  570. 21:38And notice that there's no transpose
  571. 21:40because I just you know, in some sense
  572. 21:43I've done the transpose by the naming.
  573. 21:46If I had hidden um and seek two, then it
  574. 21:48would be a no transpose. I always get
  575. 21:50confused by transposes and the fact I
  576. 21:52don't have to think about transposing
  577. 21:54makes me happy.
  578. 21:57Okay.
  579. 21:59So, um if you want to get fancy, uh you
  580. 22:03you can say, well, batch um I'm just
  581. 22:05going to replace with dot dot dot. And
  582. 22:08this means that if I had, let's say, uh
  583. 22:11you know, uh a you know, rank 10 tensor
  584. 22:13with, you know, eight different uh you
  585. 22:16know, batching dimensions, I can just
  586. 22:17write dot dot dot without enumerating
  587. 22:19all of them.
  588. 22:20Uh and this comes up in um you know,
  589. 22:22language modeling because you might have
  590. 22:24a batch dimension, you might have a
  591. 22:25sequence dimension, you might have a
  592. 22:27head dimension, and you're trying to do
  593. 22:29like this matrix operation for all of
  594. 22:31them. And you might not want to um have
  595. 22:34to just, you know, worry about this.
  596. 22:36Um and the nice thing is that you can
  597. 22:38write modular code where you can write
  598. 22:39this dot dot dot uh without worrying
  599. 22:42about, you know, the even the shape of
  600. 22:43the tensor that, you know, comes in.
  601. 22:47Okay?
  602. 22:49All right. So, that's uh ein sum, uh
  603. 22:52which is in the einops uh library.
  604. 22:55Um there's a reduce.
  605. 22:58So, this is a generalization of sum,
  606. 23:01mean, um max, and min.
  607. 23:03So, um for example, if you have, let's
  608. 23:06say, uh this this tensor,
  609. 23:09and you want to sum
  610. 23:11um according to dimension of dim minus
  611. 23:13one,
  612. 23:14okay, which means sum along the the last
  613. 23:17uh dimension here.
  614. 23:18Um so, again, I don't like the uh this
  615. 23:22this notation, but what you can do is
  616. 23:24called reduce.
  617. 23:25And basically what you say is that
  618. 23:28there's some batching dimensions. In
  619. 23:30this case, it would be um this first
  620. 23:32these first two. And then there's some,
  621. 23:34you know, hidden uh dimension, which
  622. 23:37doesn't appear on the right side, which
  623. 23:38means that it gets summed.
  624. 23:40And here I put sum, which means that the
  625. 23:41aggregation reduction operation is sum,
  626. 23:43but you can replace it with uh mean or
  627. 23:45max or min.
  628. 23:47Yeah. Is there some speed up to this?
  629. 23:50Is there some speed up to this? This
  630. 23:51basically reduces to the same type of
  631. 23:53primitive operations. You can think
  632. 23:55about it as just uh sugar.
  633. 23:57So, it should be the same.
  634. 24:03Um okay. So, the final thing I'll talk
  635. 24:05about is rearrange. So, this is I think
  636. 24:07a pretty powerful tool. Um
  637. 24:10I think it will come up in the
  638. 24:11assignment once. Um
  639. 24:13So, sometimes you have a dimension that
  640. 24:15actually represents um you know, two
  641. 24:17dimensions, and you want to operate on
  642. 24:20one of them. And the reason this happens
  643. 24:22is that sometimes you have a matrix and
  644. 24:23you flatten it.
  645. 24:25Um okay. And then you want to maybe
  646. 24:27unflatten and flatten. So, this is the
  647. 24:29way that works here.
  648. 24:31Um
  649. 24:32So, imagine I have a um a matrix 3 by 8,
  650. 24:36but where the um
  651. 24:39this
  652. 24:40uh the the this dimension eight actually
  653. 24:43represents like a 2 by 4 um matrix. So,
  654. 24:47um
  655. 24:48I want to multiply that 2 by 4 matrix by
  656. 24:51this 4 by 4 matrix.
  657. 24:53So, what I'm going to do,
  658. 24:55um actually, is to um call rearrange.
  659. 24:59And what I am doing here is saying
  660. 25:03um
  661. 25:04Look, this is some number of batch uh
  662. 25:06dimensions, which here just corresponds
  663. 25:08to the first uh element here. And then
  664. 25:11here I use parentheses to say that this
  665. 25:14H actually represents the product of
  666. 25:17heads and hidden one, um where I have to
  667. 25:20obviously there's multiple ways to
  668. 25:21decompose this. Could be 2 by 4, 4 by 2.
  669. 25:23So, I said the number of heads is two,
  670. 25:25which means that hidden one is four.
  671. 25:27And then
  672. 25:29I can um break that up into um you know,
  673. 25:33two dimensions, heads and hidden.
  674. 25:35Okay? So, uh this creates um this So,
  675. 25:38before X uh looked like this, and now X
  676. 25:43uh looks like, you know, this.
  677. 25:45So, um cuz this might be a little bit
  678. 25:47hard to see. Um
  679. 25:50So, then you can perform your operation
  680. 25:53transformation on uh W. And this is sort
  681. 25:56of what we've seen before, where you
  682. 25:58have um just a standard matmul, where um
  683. 26:02there's on some number of batching
  684. 26:04dimensions um for X here. And this is
  685. 26:07hidden times um
  686. 26:09hidden by hidden two.
  687. 26:11Um and then once you've done that
  688. 26:13transformation, you can rearrange it
  689. 26:16back.
  690. 26:17Um and this is straightforward. You
  691. 26:18basically look at two dimensions, and
  692. 26:21then you can group them into to one
  693. 26:24dimension.
  694. 26:25Okay?
  695. 26:30Yeah.
  696. 26:31Like if you take a two-dimensional thing
  697. 26:32and shift it into one dimension, are
  698. 26:34there two ways?
  699. 26:36Like you could do it row major or column
  700. 26:37major. Yeah. So, the question is that if
  701. 26:39you have a one-dimensional thing and you
  702. 26:40shift it into Sorry, you you have a
  703. 26:42two-dimensional thing and you shift into
  704. 26:44one dimensional thing, which way do you
  705. 26:46uh do it? Well, the order you do it is
  706. 26:49specified in the order uh here. Oh,
  707. 26:52okay. Yeah.
  708. 26:55Okay.
  709. 26:57So, I I
  710. 26:58Sometimes I find it takes a bit of time
  711. 27:00to get it used to, but it's well worth
  712. 27:01it because once you have einops, then
  713. 27:04you just think in a sort of a different
  714. 27:06way, and all the transposes and
  715. 27:08reductions all of them it's just like um
  716. 27:11uh it's becomes more fluid. Right? You
  717. 27:13just have to think through these kind of
  718. 27:14more um bespoke primitives.
  719. 27:19All right. So,
  720. 27:21Okay, we're going to use uh einops. Um
  721. 27:24we'll see it a little bit later.
  722. 27:26Um so, let Now let's go return back to
  723. 27:29resource accounting question. So, I have
  724. 27:31tensors. We've talked about how they
  725. 27:32take memory. So, how much compute do do
  726. 27:35they take? Okay? So, um the
  727. 27:38thing we're going to use to measure
  728. 27:41computation cost is the number of flops.
  729. 27:43A flop is a floating point operation,
  730. 27:45and it's going to we're going to assume
  731. 27:47it's a basic operation like addition or
  732. 27:49multiplication.
  733. 27:50So, now there's other things that you
  734. 27:52know, GPUs can do, but for the most part
  735. 27:54we're just going to uh ignore them
  736. 27:56because these are the the bread and
  737. 27:58butter and are going to eat up most of
  738. 28:00your um
  739. 28:01time.
  740. 28:03So, one thing that is a kind of a pet
  741. 28:05peeve is that if I say the word flops,
  742. 28:08it's actually ambiguous what I mean. Um
  743. 28:11so, there's flops, which is saying the
  744. 28:15number of floating point operations,
  745. 28:16usually written flops with a lowercase
  746. 28:19S. This is a measure amount of
  747. 28:21computation done. And then there's
  748. 28:23FLOPS, um
  749. 28:25which is floating point operations per
  750. 28:26second.
  751. 28:27Um sometimes it's also very confusingly
  752. 28:30written as FLOPS with uppercase S, which
  753. 28:32but I'm going to always write /s to make
  754. 28:34it clear that this is uh measuring the
  755. 28:36speed of hardware.
  756. 28:38So, if you go and see that uh H100s have
  757. 28:40800 989
  758. 28:43um
  759. 28:44um you know, teraflops, it's the second.
  760. 28:46And when I say that, you know, GPT-3
  761. 28:49took, you know, a 3e25 or whatever it
  762. 28:51is, uh flops, that's the the former.
  763. 28:54Okay?
  764. 28:55So, just to get that out of the way.
  765. 28:57Um
  766. 28:58so, just to give you an order of
  767. 28:59magnitude, so the number of FLOPS when I
  768. 29:02talk about 1e22
  769. 29:04or 23 or 25, these are kind of referring
  770. 29:08implicitly to amount of compute or the
  771. 29:09scale of some of these models.
  772. 29:13And uh so, if you look at H100s, um
  773. 29:16um you know, if you look at this uh
  774. 29:20glossy's uh
  775. 29:21uh spec sheet, um actually, it's not on
  776. 29:24this page. Okay, forget it. Um
  777. 29:27there is a spec sheet and it'll
  778. 29:28uh it'll tell you that for uh BF16,
  779. 29:32the number of uh uh FLOPS/s
  780. 29:35is um
  781. 29:37uh 1979.
  782. 29:39Um and then you go and you benchmark and
  783. 29:42it's like, wait a minute, that's not
  784. 29:44actually what I'm getting. And then you
  785. 29:46go read the the, you know, the fine
  786. 29:47print and there's a footnote that says
  787. 29:49this is with uh sparsity, so sparse
  788. 29:51matrix, and for dense it's uh you know,
  789. 29:53over two. Okay? So, all you always have
  790. 29:56to take these numbers divide by two. So,
  791. 29:57that's why you see these uh divide by
  792. 29:59two.
  793. 30:01Um okay.
  794. 30:03So,
  795. 30:04so this allows us to um
  796. 30:08you know, uh you know, just so our
  797. 30:10intuition, so if you have eight H100, so
  798. 30:12this one node uh for uh two weeks,
  799. 30:15that's um
  800. 30:17you know, H times the number of uh
  801. 30:19seconds uh per in two weeks.
  802. 30:22Um
  803. 30:23actually, this is looks like it's one
  804. 30:25week. Okay, fine. It's a one week uh
  805. 30:27times uh the number of flops you get per
  806. 30:30second. So, that's you know, about uh 5e
  807. 30:3321.
  808. 30:35Okay, so this is just kind of building
  809. 30:37intuition for the number of flops that
  810. 30:38certain types of hardware have and how
  811. 30:40many flops, you know, certain types of
  812. 30:42models uh require.
  813. 30:44Okay, it's nothing fancy, it's just math
  814. 30:47and uh napkin math. Um
  815. 30:49Okay, so so now let's do some more
  816. 30:52something more mechanical. Um
  817. 30:54So, you know, suppose you have uh a a
  818. 30:57linear model. Um
  819. 30:59it turns out that a lot of this calculus
  820. 31:01is uh
  821. 31:03of counting flops is actually going to
  822. 31:05be at the core it's like linear matmuls.
  823. 31:08So, this is actually not without too
  824. 31:10much loss of generality. So, you have M
  825. 31:12points, each point is D dimensional, and
  826. 31:15um we're going to map
  827. 31:18uh
  828. 31:19each of these D dimensional vectors to
  829. 31:21uh a K dimensional output. Okay? So, um
  830. 31:25so B is going to be the number of uh
  831. 31:27points, D is the number of dimension
  832. 31:30input dimensions, and K is the number of
  833. 31:31output dimensions.
  834. 31:33Okay, so we're let's construct uh some
  835. 31:36uh an X, which is the data matrix B by
  836. 31:40D, the weight matrix is D by K, and we
  837. 31:43do a the matmul.
  838. 31:45Um
  839. 31:45the question is, you know, how many
  840. 31:47flops uh that is.
  841. 31:49And uh it turns out that this is going
  842. 31:52to be two times
  843. 31:54um
  844. 31:55the
  845. 31:56uh basically the product of all the the
  846. 31:58three dimensions.
  847. 32:00And the way to see that is that um we
  848. 32:03have one, you know, multiplication
  849. 32:06for each uh you know, triple,
  850. 32:08and then also one addition.
  851. 32:11Okay?
  852. 32:12So, there's a minus one because if uh
  853. 32:14you don't you have to actually add like,
  854. 32:17you know, um
  855. 32:18in this case uh D minus one times, but
  856. 32:21let's ignore that.
  857. 32:23Okay, I'll come back to this um if this
  858. 32:25was a bit fast. Um I think there's a
  859. 32:27there's another way to uh you know,
  860. 32:28derive this.
  861. 32:31So, what about the flops of other
  862. 32:33operations? So, um element-wise
  863. 32:36operations are just uh you know, the
  864. 32:38size of the matrix. I think that's
  865. 32:40fairly clear, so addition also requires
  866. 32:42NM flops. Um so, in general, no other
  867. 32:45operation you encounter is expensive as
  868. 32:48matrix multiplication for large enough
  869. 32:50matrices. So, in general, we're just
  870. 32:53going to focus on what matmuls are doing
  871. 32:55with the important caveat of when we
  872. 32:57talk about memory. Yeah.
  873. 32:59I would just just like this is for
  874. 33:02interest, like there are some other
  875. 33:04algorithms for doing matrix
  876. 33:05multiplication. Is this only for, you
  877. 33:07know, just doing it
  878. 33:08normally?
  879. 33:09So, the question is that there are other
  880. 33:11algorithms for doing matrix
  881. 33:13multiplication. You mean like sub cubic
  882. 33:15uh cubic algorithms. Uh so, in general,
  883. 33:18that's
  884. 33:20the optimization the algorithms that
  885. 33:21people are going to explore for much of
  886. 33:23matrix multiplications are going to be
  887. 33:24much more about how you co-design with
  888. 33:27the systems rather than these more
  889. 33:29asymptotic uh algorithms.
  890. 33:32Yeah.
  891. 33:33Yeah. Um we're considering addition and
  892. 33:36multiplication in the same way. It's not
  893. 33:38possible to do addition more efficiently
  894. 33:40than multiplication.
  895. 33:42Yeah, so I think the way uh the hardware
  896. 33:45is built, the two are basically the, you
  897. 33:47know, the same. But yeah, intuitively it
  898. 33:49seems like addition I can do addition
  899. 33:51faster than do multiplication, but um
  900. 33:53the hardware the way hardware built is
  901. 33:55built they're kind of the same.
  902. 33:59Okay, so
  903. 34:00you can think about this matmul as um
  904. 34:03B is the number of data points,
  905. 34:05um and DK is the number of parameters,
  906. 34:09right? So, remember X is B by D and W is
  907. 34:12D by K.
  908. 34:14So, another way to think about this um
  909. 34:17formula is that the number of flops uh
  910. 34:21in order to do a forward pass of this
  911. 34:24linear uh matrix is actually two times
  912. 34:28the number of tokens or data points
  913. 34:30times the number of parameters.
  914. 34:33Okay?
  915. 34:34So, this is a you know, it turns out
  916. 34:36that this actually generalizes to
  917. 34:37transformers, which if you remember the
  918. 34:39six times N times D formula, we can kind
  919. 34:41of see the shape of that uh you know,
  920. 34:43forming.
  921. 34:46Okay, so um
  922. 34:48unfortunately, these calculations are
  923. 34:49not going to be very meaningful because
  924. 34:51I'm doing this on on on CPU, but I'll
  925. 34:54just walk through the
  926. 34:55uh the code here. So, um
  927. 34:58so the what we've so far done is uh
  928. 35:03measured flops. So, this is independent
  929. 35:06of of hardware, it's just like the
  930. 35:08number of calculations you need to do
  931. 35:10for your model.
  932. 35:12So, now the question is like how long
  933. 35:13does it actually take on hardware?
  934. 35:15So, one way to find out is you just time
  935. 35:17it.
  936. 35:18Okay? So, in this class, I think in uh a
  937. 35:20few lectures we're going to talk more
  938. 35:22about uh benchmarking, um but here's a
  939. 35:25little preview.
  940. 35:27So,
  941. 35:28um in general,
  942. 35:30um when you time, especially on on GPU,
  943. 35:33you um have to call uh CUDA synchronize
  944. 35:37to to make sure that because, you know,
  945. 35:39the GPU is like kind of running
  946. 35:40asynchronously, you want to make sure
  947. 35:41that you have this kind of
  948. 35:43synchronization point.
  949. 35:45Um and then you perform the operation,
  950. 35:47and after the
  951. 35:48uh operation,
  952. 35:50then you
  953. 35:52uh you also have to have the
  954. 35:53synchronization barrier. If you omit
  955. 35:55this, you're going to find that wow,
  956. 35:57your timings are really fast, and that's
  957. 35:59because these this is a non-blocking
  958. 36:00call, it just returns.
  959. 36:02Um and often it's general good practice
  960. 36:04to try this multiple times and take the
  961. 36:06the average.
  962. 36:08>> [clears throat]
  963. 36:09>> Okay.
  964. 36:10So,
  965. 36:12so um the actual flops per second
  966. 36:15is um
  967. 36:17basically the number of flops you did
  968. 36:21times the time that you recorded on your
  969. 36:23hardware.
  970. 36:24Right?
  971. 36:26And um so, remember there's a also a
  972. 36:28number number, which is the GPU has a
  973. 36:30spec sheet.
  974. 36:32Uh let's see if I can pull it up on this
  975. 36:35link.
  976. 36:36Um
  977. 36:39I feel like maybe I have the wrong link.
  978. 36:40I'll have to fix that after class.
  979. 36:43Um
  980. 36:44that's uh it gives you
  981. 36:47um some number of flops, which was 989
  982. 36:52um
  983. 36:53teraflops.
  984. 36:54And so, in general, the number of actual
  985. 36:58flops per second is going to be
  986. 36:59different from the promised uh flops per
  987. 37:02second.
  988. 37:03Okay?
  989. 37:04Um and the
  990. 37:07Okay, um oops.
  991. 37:09Sorry.
  992. 37:12And the uh the way to direct think about
  993. 37:14the discrepancy is something called um
  994. 37:17model flops utilization or MFU.
  995. 37:20And the definition of uh MFU is the
  996. 37:23actual flops per second divided by the
  997. 37:25promised uh flops per second. And here
  998. 37:28this is, you know, ignoring the the um
  999. 37:31you know, the communication and other
  1000. 37:33overhead.
  1001. 37:34Okay? So, you basically take the actual
  1002. 37:36and divide by the promised.
  1003. 37:39So, in general,
  1004. 37:40um
  1005. 37:41you know, it's rare that you get more
  1006. 37:42than what's uh you know, uh
  1007. 37:45I mean, it's
  1008. 37:46yeah, you you just never get anything
  1009. 37:48more than what you were promised. Often
  1010. 37:51you get less, and in general, if you get
  1011. 37:54about MFU of uh 0.5 for modern models,
  1012. 37:57um you should be pretty happy with
  1013. 37:59yourself. Um if you have only just like
  1014. 38:02a straight-up matmul, you can get maybe
  1015. 38:04like, you know, potentially like 80 uh
  1016. 38:070.8 um even. Uh but you you usually
  1017. 38:10can't get that high. And sometimes if
  1018. 38:13you're um have really,
  1019. 38:16you know, something's really wrong,
  1020. 38:17you'll get something like 0.1, which
  1021. 38:19means that you should do something.
  1022. 38:21Okay.
  1023. 38:22So, whenever you're you write your
  1024. 38:24model, you can calculate uh you now know
  1025. 38:27how to calculate MFU through a
  1026. 38:29combination of counting the number of
  1027. 38:31logical flops that you
  1028. 38:33uh your model is needs to, you know, do,
  1029. 38:37um and then looking at the wall clock
  1030. 38:39time and essentially dividing.
  1031. 38:41Yeah.
  1032. 38:42Was there a question over there? Oh.
  1033. 38:49Yeah, so is the promised What is the
  1034. 38:51promised flops number? That is um in the
  1035. 38:54spec sheet, it is
  1036. 38:56um
  1037. 38:57already divided by two
  1038. 38:59um of the 1989 number.
  1039. 39:03And then on top of that you only get
  1040. 39:04like 0.5 of that in in general,
  1041. 39:06depending on your um computation.
  1042. 39:13You're getting 50%
  1043. 39:15Yeah, so why are you getting only 50%
  1044. 39:17MFU?
  1045. 39:18Um actually, that's a good question.
  1046. 39:19I'll come back to that when we talk
  1047. 39:21about memory bottlenecks.
  1048. 39:27Okay, so to to summarize here,
  1049. 39:30um matrix multiplications dominate um
  1050. 39:33the
  1051. 39:34you know, generally the computations,
  1052. 39:35and that's sort of by, you know, design.
  1053. 39:38Um
  1054. 39:39so
  1055. 39:40um
  1056. 39:42and the number of flops per second
  1057. 39:44depends on the hardware.
  1058. 39:46So, better hardware um leads to better
  1059. 39:49num more flops, and also depends on the
  1060. 39:52data type. Um which means that if you
  1061. 39:55look at the the spec sheet, um different
  1062. 39:58uh data types will have different flops.
  1063. 40:00If you try to do float 32, nowadays it's
  1064. 40:02going to be really really slow because
  1065. 40:04um they're not really optimizing for
  1066. 40:06that workload, whereas uh now BF16 or um
  1067. 40:11FP8 are going to be much uh faster.
  1068. 40:16Okay? And MFU, now you know what MFU is.
  1069. 40:18It's uh the you know, actual flops
  1070. 40:21divided by promised flops.
  1071. 40:27All right. So, to go back to the
  1072. 40:29question of why is MFU 0.5?
  1073. 40:32And to understand that, I'm going to
  1074. 40:34have to introduce this idea of
  1075. 40:35arithmetic intensity.
  1076. 40:37And and the re- reason is that, well,
  1077. 40:40it's not just um
  1078. 40:43you know, doing a bunch of
  1079. 40:45matmuls and then you're done and looking
  1080. 40:47at how long it the matmuls take.
  1081. 40:49Um this is my very
  1082. 40:51uh cartoon version of what uh hardware
  1083. 40:54looks like.
  1084. 40:55Um you have high bandwidth memory
  1085. 40:58uh and then you have where the the
  1086. 41:00compute uh cores or the cell
  1087. 41:02accelerators chips are.
  1088. 41:04And then um what how do you compute?
  1089. 41:08Well, you have to
  1090. 41:10send your inputs, your matrices there's
  1091. 41:13there's uh tensors are sitting down
  1092. 41:14here, from the memory to the
  1093. 41:16accelerator. You do the the computation,
  1094. 41:19and then you uh send it back.
  1095. 41:22Okay? So, if you want to measure how
  1096. 41:24long this takes, this sort of depends on
  1097. 41:26two things. One is the accelerator
  1098. 41:28speed,
  1099. 41:30which is what we've talked about uh just
  1100. 41:32you know, just now.
  1101. 41:34But the other thing that matters is the
  1102. 41:35memory bandwidth of your hardware, which
  1103. 41:37we haven't talked about.
  1104. 41:39And uh if you look at the spec sheet, we
  1105. 41:42talked about how the flops per second
  1106. 41:44was
  1107. 41:45um
  1108. 41:461979 E12
  1109. 41:48/ 2.
  1110. 41:50And the bytes per second, which is the
  1111. 41:51memory bandwidth, this is 3.3 uh
  1112. 41:53terabytes per per second.
  1113. 41:57Okay? And remember we uh why we were
  1114. 42:00looking at memory
  1115. 42:02um
  1116. 42:03uh
  1117. 42:04how much things took to store,
  1118. 42:07it's going to it's
  1119. 42:08most obviously that, well, you just if
  1120. 42:11you have a model that's too big it
  1121. 42:12doesn't fit in your memory, that's not
  1122. 42:14going to be fun. But also, it turns out
  1123. 42:17that memory you need to move this
  1124. 42:19memory, which takes time. So, actually
  1125. 42:22the size of how large things actually
  1126. 42:24influences uh speed as well.
  1127. 42:27Okay, so um I'm going to talk through
  1128. 42:30some operations, and I'm going to
  1129. 42:32basically compute how long things are
  1130. 42:34going to take, and introduce this idea
  1131. 42:36of arithmetic intensity.
  1132. 42:38Okay, so suppose I have a million uh
  1133. 42:40dimensional uh vector of BF uh 16,
  1134. 42:44and um and I'm going to just compute a
  1135. 42:47value
  1136. 42:49on this. So, re- remember ReLU is just
  1137. 42:51uh max of X and zero uh done
  1138. 42:53element-wise on the entire vector here.
  1139. 42:58So, um
  1140. 43:00I count two things. One is the number of
  1141. 43:02bytes that were moved.
  1142. 43:05So, um I have to read X,
  1143. 43:08um in you know, copy it into the the um
  1144. 43:13the accelerators,
  1145. 43:14and each of this is going to be two
  1146. 43:18because a BF16 is two bytes per float
  1147. 43:20times N floats. So, that's two N, and
  1148. 43:23then I'm going to write Y back. So,
  1149. 43:25that's another two N.
  1150. 43:27Okay? So, that's the number of bytes
  1151. 43:28that have to be uh moved. And then, how
  1152. 43:30many flops were done?
  1153. 43:32Well, um each of these elements, I'm
  1154. 43:35just comparing with zero, and that's it.
  1155. 43:38So, that's N comparisons.
  1156. 43:40Um
  1157. 43:41Okay?
  1158. 43:43So, now I look at the communication
  1159. 43:46time, which is the number of bytes
  1160. 43:48that were I needed to move,
  1161. 43:50um you know, divided by the the speed of
  1162. 43:54that movement,
  1163. 43:55and that gives me the time, which is,
  1164. 43:57you know, 1e-6
  1165. 43:59uh you know, seconds.
  1166. 44:01And what about the computation time?
  1167. 44:02That's the flops divided by flops per
  1168. 44:04second.
  1169. 44:05So, that's um 1e-9.
  1170. 44:09Okay.
  1171. 44:10So, there's also going to be another
  1172. 44:12important uh assumption, which generally
  1173. 44:15we try to hold is that we overlap
  1174. 44:17communication and computation. We're
  1175. 44:20going to talk more about that when we
  1176. 44:21talk more deeply about GPUs, but the
  1177. 44:23idea is that in in this case, as as we
  1178. 44:27don't sit here waiting for the the you
  1179. 44:30know, the things to move, as soon as
  1180. 44:32they're there, we start computing them,
  1181. 44:34and then
  1182. 44:35um we move them back. So, that this this
  1183. 44:38movement and also the compute is
  1184. 44:41happening at the same time.
  1185. 44:43So, mathematically uh
  1186. 44:46we're just going to assume that the
  1187. 44:47total time is the max of the two,
  1188. 44:49because we're going to assume that we
  1189. 44:51can perfectly overlap them. In practice,
  1190. 44:52it's not going to be perfectly
  1191. 44:53overlapped, there's going to be some
  1192. 44:54overhead, but this is uh um a good
  1193. 44:56enough for now.
  1194. 44:58Okay, so the total time,
  1195. 45:00as we see here, is 1e-6.
  1196. 45:04Okay.
  1197. 45:05So, if you ask, you know, what is the
  1198. 45:07bottleneck here?
  1199. 45:08So, when the communication time is
  1200. 45:10greater than the computation time, then
  1201. 45:13we call it uh the algorithm memory
  1202. 45:15bound,
  1203. 45:16because you're spending most of your
  1204. 45:17time just waiting for bits to show up.
  1205. 45:19And when the com- computation time is
  1206. 45:21greater than communication time, that's
  1207. 45:22compute bound,
  1208. 45:24because then you're actually, you know,
  1209. 45:26your bottleneck is actually doing the
  1210. 45:27compute.
  1211. 45:29And so, in this case, uh what is ReLU?
  1212. 45:34Rectified linear units. Oh, sorry. Is it
  1213. 45:36memory bound or uh you know,
  1214. 45:38compute bound? Memory bound.
  1215. 45:39>> [laughter]
  1216. 45:40>> A memory bound. Yeah.
  1217. 45:43Um So, and it it's clear, right?
  1218. 45:46Because, you know, the compute is way
  1219. 45:48less than the communication.
  1220. 45:51Okay.
  1221. 45:53So,
  1222. 45:54here's another way to see it, and this
  1223. 45:56is where I'm going to define the
  1224. 45:59intensity. So, what is intensity of an
  1225. 46:01accelerator is essentially how much work
  1226. 46:04can the accelerator do
  1227. 46:06do per byte transferred?
  1228. 46:08And for any given accelerator based on
  1229. 46:11the spec sheet, you basically have um
  1230. 46:14the the flops per second divided by the
  1231. 46:18um the bytes per second.
  1232. 46:20Okay? So, how much useful uh work can
  1233. 46:22you do per byte that's moved? For H100s,
  1234. 46:25that's uh 295.
  1235. 46:28Okay? So, that means for every byte
  1236. 46:31of
  1237. 46:32um you can do 295
  1238. 46:36um floating point operations.
  1239. 46:38Okay? So, that's an intuitive number to
  1240. 46:40uh kind of have in your head, about 300.
  1241. 46:43Um okay, so now the arithmetic intensity
  1242. 46:46of uh algorithm is how much actual work
  1243. 46:49was done per byte for this workload.
  1244. 46:51And if you look at it for the ReLU
  1245. 46:53computation, it's flops over bytes,
  1246. 46:56and it's uh actually it's so this is
  1247. 46:58actually a
  1248. 46:59a quarter, I guess. Um
  1249. 47:01not half. So, the point is that it's
  1250. 47:03very small,
  1251. 47:04um 0.25.
  1252. 47:07Okay, so now we can talk about
  1253. 47:09bottlenecks through the language of
  1254. 47:11intensity. So, if something is memory
  1255. 47:14bound if the arithmetic intensity is
  1256. 47:16smaller than uh the accelerator
  1257. 47:19intensity, and compute bound if it's
  1258. 47:21greater than accelerator intensity.
  1259. 47:25All right, so these are equivalent, and
  1260. 47:26if you look at the algebra, it's
  1261. 47:28basically you have two fractions, and
  1262. 47:29then you just kind of multiply and
  1263. 47:31divide to you basically switch the terms
  1264. 47:34around.
  1265. 47:36Okay, so in this case,
  1266. 47:38um
  1267. 47:39you know, we're uh memory bound.
  1268. 47:42So, in general, we're going to find
  1269. 47:44ourselves in a situation where we're
  1270. 47:45memory bound because data movement is
  1271. 47:48expensive, and so if you can get higher
  1272. 47:51arithmetic intensity, that's good.
  1273. 47:54Okay.
  1274. 47:55So, 0.25. If you see that number, if
  1275. 47:57someone tells you your arithmetic
  1276. 47:58intensity is 0.25, you should uh say,
  1277. 48:01"Oh, this is this is really bad."
  1278. 48:03Yeah. What's some typical arithmetic
  1279. 48:05intensity for
  1280. 48:08Yeah, so we'll get to that. Okay.
  1281. 48:12All right. So, so okay, so one way to
  1282. 48:16think about increasing arithmetic
  1283. 48:17intensity is that let's just try to do
  1284. 48:20more stuff uh per unit of byte moved.
  1285. 48:23Okay, so the GELU is another activation.
  1286. 48:26Um it kind of looks like this. It's
  1287. 48:27some, you know, formula that is uh more
  1288. 48:30uh um
  1289. 48:31um doesn't
  1290. 48:33uh you know, have zeros.
  1291. 48:34Um
  1292. 48:35and uh if you do this calculation, the
  1293. 48:38number of bytes are moved back and
  1294. 48:41forth, it's still 2N + 2N.
  1295. 48:44Um
  1296. 48:45and the flops
  1297. 48:47here, it's about 20 um flops per
  1298. 48:51element-wise GELU operation. So, it's
  1299. 48:5320N. Right? So, the arithmetic intensity
  1300. 48:55is, let's say, five. This is a kind of a
  1301. 48:58crude estimate. And in this case, are we
  1302. 49:01memory bound or compute bound?
  1303. 49:10Memory bound. Memory bound.
  1304. 49:12Right? Because five is still smaller
  1305. 49:13than 295.
  1306. 49:15Way smaller.
  1307. 49:17So, even though GELU does a lot of uh a
  1308. 49:20lot of work,
  1309. 49:21um more work than ReLU, in the way that
  1310. 49:24things are structured, it's still um
  1311. 49:27it's still memory bound.
  1312. 49:30Which means that if you were just
  1313. 49:31computing ReLU and GELU, you think
  1314. 49:34that, well, GELU is like so complicated,
  1315. 49:37it must be really expensive, but
  1316. 49:38actually it's exactly the same. Right?
  1317. 49:40Because that's not where the bottleneck
  1318. 49:42is.
  1319. 49:46Okay, so now let's uh look at some uh
  1320. 49:48linear operations. So, dot product.
  1321. 49:50Okay, so dot product, you have a vector
  1322. 49:52X, you have a vector W, size N, and you
  1323. 49:55take the dot product. So, how many bytes
  1324. 49:57are moved? So, you read X, which is 2N,
  1325. 50:00read W, which is 2N, and then you write
  1326. 50:02Y, which is a scalar, which is two.
  1327. 50:05Okay, and the number of flops is you do
  1328. 50:08the N multiplications and
  1329. 50:11uh and
  1330. 50:12N minus one additions, so that's 2N
  1331. 50:13minus one.
  1332. 50:16Okay, so um what's the arithmetic
  1333. 50:18intensity? Oh, for this one it's uh
  1334. 50:20about half.
  1335. 50:22Okay, so which is also pretty bad, which
  1336. 50:24means that hopefully by you get the
  1337. 50:27idea, it's memory bound.
  1338. 50:30Okay?
  1339. 50:33So, what about a matrix vector product?
  1340. 50:36Okay, so what I have this um X is a
  1341. 50:38vector, W is a M by M matrix, um and you
  1342. 50:41perform this product.
  1343. 50:43Um how many of you think this will be
  1344. 50:44compute bound?
  1345. 50:46How many of you think memory bound?
  1346. 50:48Okay.
  1347. 50:49So, let's see.
  1348. 50:51So, I'm going to read X, uh read, which
  1349. 50:53is 2N, read W, which is 2N squared, and
  1350. 50:56write Y, which is uh 2N. And there's uh
  1351. 51:00the number of flops is basically you're
  1352. 51:02you know uh doing N dot products, so
  1353. 51:05that's N times the cost of doing a dot
  1354. 51:07product.
  1355. 51:09And uh the arithmetic intensity is, you
  1356. 51:11know, barely barely higher.
  1357. 51:14Um
  1358. 51:16So,
  1359. 51:17uh so this is,
  1360. 51:18you know, um
  1361. 51:20also memory bound.
  1362. 51:21Okay.
  1363. 51:22So, now let's talk about matrix
  1364. 51:24multiplication. So, this is where things
  1365. 51:26get um interesting. So, I have a M by M
  1366. 51:28matrix, another M by M matrix, and I
  1367. 51:30multiply them.
  1368. 51:31And uh the number of bytes that were
  1369. 51:33moved was um
  1370. 51:36you know, 2N squared plus 2N squared,
  1371. 51:39and then I have to write a 2N squared um
  1372. 51:41bytes uh back to Y.
  1373. 51:43And the number of flops here is N
  1374. 51:46squared dot products.
  1375. 51:49And then what's arithmetic intensity?
  1376. 51:52It's 300, woohoo. Okay, so 340, and in
  1377. 51:56general it's roughly N over three.
  1378. 52:00Um and intuitively this makes sense
  1379. 52:03because you're sending N squared things,
  1380. 52:07but you're computing N cubed things.
  1381. 52:10Right? So, the number of things you're
  1382. 52:11computing over number of things you're
  1383. 52:13sending is order N.
  1384. 52:16And this gets better the larger you make
  1385. 52:18the matrices.
  1386. 52:19So, and in general,
  1387. 52:21this is why when you hear people talk
  1388. 52:24about, "Oh, we need to make large batch
  1389. 52:26sizes or have large matrices." It's
  1390. 52:28exactly this. If you if you're under uh
  1391. 52:32the accelerator intensity,
  1392. 52:34um
  1393. 52:35making things smaller doesn't actually
  1394. 52:37speed things up, right? It's all kind of
  1395. 52:39the same. Whereas, if you get to the
  1396. 52:41point where you're over uh this, then
  1397. 52:43you're actually saturating your um you
  1398. 52:45know,
  1399. 52:46uh GPUs.
  1400. 52:48Okay, so finally this is uh you know,
  1401. 52:50compute bound.
  1402. 52:52Okay.
  1403. 52:53So, as long we have large matrices,
  1404. 52:55we're actually pretty good. We're in uh
  1405. 52:57compute bound, saturating the
  1406. 52:58accelerator. And the question about
  1407. 53:00earlier about, you know, what about
  1408. 53:02transformers? Well, it turns out we'll
  1409. 53:04see um
  1410. 53:05in both your assignment, but also in the
  1411. 53:07next lecture, that transformers are
  1412. 53:09essentially big, you know, matrix
  1413. 53:11multiplications with some things
  1414. 53:13sprinkled in between. So, um
  1415. 53:16the so that's good news from arithmetic
  1416. 53:18intensity, and this is by design.
  1417. 53:20Transformers are designed in a certain
  1418. 53:21way to have high um arithmetic uh
  1419. 53:24intensity.
  1420. 53:26And one comment here, just to foreshadow
  1421. 53:29the inference lecture, is that matrix
  1422. 53:31vector products is essentially kind of
  1423. 53:33what goes on when you're doing
  1424. 53:34transformer inference. Right? Because
  1425. 53:36inference, you're generating one token
  1426. 53:38at a time,
  1427. 53:39and so you only get to it's like a
  1428. 53:41vector that you're trying to dot product
  1429. 53:43with a with a matrix.
  1430. 53:45Um and that's and that's where as we saw
  1431. 53:47was uh memory bound.
  1432. 53:50Whereas in training time, you get this
  1433. 53:51whole sequence, and you're processing
  1434. 53:53all at once.
  1435. 53:56Okay.
  1436. 53:58So, um note also that the intensity
  1437. 54:00depends on the the precision. So, by
  1438. 54:03default everything we're doing here is
  1439. 54:05FP uh you know, 16.
  1440. 54:08Um
  1441. 54:09Okay, so also the to tie it to the other
  1442. 54:11question about MFU, and the reason you
  1443. 54:13might be getting
  1444. 54:14um low MFU is that while you
  1445. 54:18um might be doing
  1446. 54:20>> [clears throat]
  1447. 54:21>> um so, MFU is a a promised flops uh
  1448. 54:25sorry, actual
  1449. 54:26over promised. Right? So, if you have
  1450. 54:29really large memory band uh bottlenecks,
  1451. 54:32then you're not actually going to get
  1452. 54:34very good you know, throughput even
  1453. 54:35though um
  1454. 54:37you know, you
  1455. 54:38you know, the the promise is like if you
  1456. 54:40didn't have memory bottlenecks, you were
  1457. 54:41just like going through and doing all
  1458. 54:43the computations of your model.
  1459. 54:47Okay, uh final thing on arithmetic
  1460. 54:49intensity, um
  1461. 54:51roofline plots.
  1462. 54:53So, there's a nice way to Yeah?
  1463. 54:56Yeah.
  1464. 54:58Generally speaking, these models tend to
  1465. 55:00be memory bound, as they say, for most
  1466. 55:02of the computation.
  1467. 55:04But yet, the accelerators are 50% is not
  1468. 55:08as good, and then maybe you get 70,
  1469. 55:10maybe you get 80, that's really good.
  1470. 55:12Mea- meaning that the accelerator is
  1471. 55:14outsized compared to memory. Uh
  1472. 55:17bandwidth.
  1473. 55:18Um why is it like that? Why do these
  1474. 55:21accelerators
  1475. 55:22big just big Well, they're just idling
  1476. 55:25or waiting for memory.
  1477. 55:27So, so um the question is uh could you
  1478. 55:30design maybe accelerators that had
  1479. 55:32better um you know, characteristics?
  1480. 55:35Um yeah, maybe when we talk about GPUs
  1481. 55:37and understand a bit more how they work,
  1482. 55:39we can talk about uh why this is.
  1483. 55:42Um but if you have uh an answer, you
  1484. 55:43should tell Jensen, and maybe he can um
  1485. 55:46design a better um better hardware.
  1486. 55:49>> [snorts]
  1487. 55:49>> Um
  1488. 55:49okay, so
  1489. 55:51let's visualize the relationship between
  1490. 55:53arithmetic intensity and performance.
  1491. 55:55So, this plot basically plots the
  1492. 55:57arithmetic intensity on the X axis. So,
  1493. 56:00every
  1494. 56:01um vertical uh slice here is corresponds
  1495. 56:04to a particular, let's say, algorithm.
  1496. 56:07And then we have these lines here, and
  1497. 56:09each of these lines corresponds to,
  1498. 56:11let's say, a particular accelerator.
  1499. 56:13Maybe
  1500. 56:14um you know, H100s or
  1501. 56:16uh B200s or so.
  1502. 56:19And so, uh what this shows is and then
  1503. 56:21on the Y axis is um the flops per second
  1504. 56:24that are realized.
  1505. 56:26So, if your algorithm has low arithmetic
  1506. 56:30intensity, like um value or dot
  1507. 56:32products, you're going to be over here,
  1508. 56:35which means that the flops per second
  1509. 56:36realized is going to be um
  1510. 56:39you know, not as high as your peak
  1511. 56:41accelerator.
  1512. 56:43And as you increase the memory
  1513. 56:45uh arithmetic intensity, that's going to
  1514. 56:48things are going to be better. You're
  1515. 56:50going to be able to saturate your um
  1516. 56:52your hardware up until a certain point.
  1517. 56:55And after a certain point, you're
  1518. 56:56compute bound, and uh you're
  1519. 56:59obviously you can't exceed the peak
  1520. 57:00flops um
  1521. 57:03Okay?
  1522. 57:04All right, so
  1523. 57:06um let's go on here. Okay.
  1524. 57:10So, um now I'm going to go back and talk
  1525. 57:15about memory and compute for
  1526. 57:18um the operations that we need in um in
  1527. 57:22training.
  1528. 57:23Right? So, so far we've done tensor
  1529. 57:24operations, basically matmuls, and um
  1530. 57:28you know, we saw basically how much
  1531. 57:29memory it took and how much compute it
  1532. 57:31took.
  1533. 57:32Um and the interaction between them.
  1534. 57:34So, now let's actually think about what
  1535. 57:36it takes to train. So, here is uh our
  1536. 57:39running example here. Um actually it's
  1537. 57:42not a linear network, uh just a deep
  1538. 57:44network. Um
  1539. 57:45so, I'm going to consider a case where I
  1540. 57:47have um a input, which is a B by D uh
  1541. 57:51input. Um and a number of layers, where
  1542. 57:54each layer is a D by D uh matmul.
  1543. 57:58Um pa- and then that produces some
  1544. 58:01um set of pre-activations, and then
  1545. 58:04element-wise value, that produces some
  1546. 58:06activations of the first layer, and then
  1547. 58:08this is repeated again and again.
  1548. 58:11Okay, and the output is just going to be
  1549. 58:13the same size.
  1550. 58:17Okay, so the deep network um I mean,
  1551. 58:20just to see what it looks like in
  1552. 58:21PyTorch, um just so it's basically a set
  1553. 58:24of blocks.
  1554. 58:25Each block uh
  1555. 58:27is
  1556. 58:29has a weight vector associ- uh sorry, a
  1557. 58:31weight matrix associated with it.
  1558. 58:33Okay?
  1559. 58:34And um
  1560. 58:38So, when you the number of parameters
  1561. 58:40here is D squared times the number of
  1562. 58:43layers, L.
  1563. 58:45And when you run the the model on a
  1564. 58:48batch of data,
  1565. 58:49um what we're doing is we're going
  1566. 58:51through the layers, and each layer um
  1567. 58:54we basically apply the linear
  1568. 58:57transformation, and then a point-wise uh
  1569. 58:59value activation.
  1570. 59:01Um and then we do that for all the
  1571. 59:03layers. Okay? So, this is hopefully
  1572. 59:05straightforward.
  1573. 59:06Um
  1574. 59:07very simple model.
  1575. 59:09>> [clears throat]
  1576. 59:09>> Um okay, so now let's talk about, you
  1577. 59:11know, uh
  1578. 59:13you know, gradients.
  1579. 59:15So, in uh let's use an actually a sim-
  1580. 59:17even simpler example. So, this is just
  1581. 59:20um a simple linear model regression.
  1582. 59:23Um
  1583. 59:24so, we have a a vector 1 2 3, a weight
  1584. 59:28uh a vec- weight vector um 1 1 1.
  1585. 59:32Um
  1586. 59:33and we take a dot product uh and we form
  1587. 59:36this uh MSE loss.
  1588. 59:38Okay? And what happens when we uh you
  1589. 59:40know, take the the do the backward pass
  1590. 59:43um
  1591. 59:43is that
  1592. 59:45um each of the the variables involved in
  1593. 59:47this computation graph has a gradient
  1594. 59:50that is um either set or not set. So,
  1595. 59:53um
  1596. 59:54the
  1597. 59:56w.grad gets set to, you know, 1 2 3.
  1598. 1:00:00Okay?
  1599. 1:00:01So, this is just basic uh mechanics of
  1600. 1:00:03uh you know, PyTorch. I think it should
  1601. 1:00:05be familiar to um all of you.
  1602. 1:00:08So now, the question is like how much uh
  1603. 1:00:11compute does
  1604. 1:00:13uh gradients you know, take?
  1605. 1:00:16Okay. So, let's count flops for
  1606. 1:00:18computing gradients.
  1607. 1:00:19>> [clears throat]
  1608. 1:00:21>> So, let's take a you know, simplified
  1609. 1:00:23model
  1610. 1:00:24um where you have um an X which is BID
  1611. 1:00:29and um times a W1 matrix.
  1612. 1:00:32Um well, we're going to it's sort of
  1613. 1:00:34ignore this uh value for now uh just for
  1614. 1:00:36simplicity times W2 which is the same D
  1615. 1:00:38by D may same sa- shape D by D matrix.
  1616. 1:00:42Okay, I'm going to use einsum here um
  1617. 1:00:45uh for reasons that will become clear.
  1618. 1:00:47Um so, the first thing you do in this
  1619. 1:00:49linear network as you take X and you
  1620. 1:00:50take W1. X is batch by input dimension.
  1621. 1:00:55Uh W1 uh is in by out and you get a
  1622. 1:00:58batch by out.
  1623. 1:01:00Okay? And H2 takes H1 and W2 and it's
  1624. 1:01:04this kind of the same story. These are
  1625. 1:01:05just matmuls.
  1626. 1:01:09So, and then you form a loss. I'm just
  1627. 1:01:10setting it to some arbitrary just to get
  1628. 1:01:12a number out.
  1629. 1:01:14Okay.
  1630. 1:01:15So, what happens in the in the backward
  1631. 1:01:17pass? Um
  1632. 1:01:18I'm going to call these retain gradients
  1633. 1:01:20for debugging purposes which you'll see
  1634. 1:01:22later. Uh you take the backward pass on
  1635. 1:01:24the loss and the question is how much
  1636. 1:01:27work did that uh how many floating point
  1637. 1:01:29operations was that?
  1638. 1:01:31Okay, so let's zoom in on one layer.
  1639. 1:01:33Let's focus on the second layer here.
  1640. 1:01:35So, this this uh layer.
  1641. 1:01:37Um
  1642. 1:01:39and so, the second layer takes H1 and
  1643. 1:01:42just multiplies by a matrix W2 to get
  1644. 1:01:45H2. And that's it.
  1645. 1:01:47So, if you look at the num- uh the flops
  1646. 1:01:50in the forward pass, this is just uh a
  1647. 1:01:52matmul.
  1648. 1:01:53And we just uh take the three dimensions
  1649. 1:01:56and we multiply them together and BF16
  1650. 1:01:58that's uh two bytes. Uh sorry, that's
  1651. 1:02:01that's irrelevant. It's um this is just
  1652. 1:02:03two because it's a addition and a
  1653. 1:02:05multiplication.
  1654. 1:02:06Okay.
  1655. 1:02:07Um
  1656. 1:02:08so, in the backward pass, you know, what
  1657. 1:02:10does this look like? Okay, so in the
  1658. 1:02:12backward pass, um if you remember your
  1659. 1:02:14chain rule and your backprop algorithm,
  1660. 1:02:17you have to compute two things. You have
  1661. 1:02:19to compute um
  1662. 1:02:21the the kind of backward message the the
  1663. 1:02:23gradient of the loss with respect to
  1664. 1:02:27your your kind of input
  1665. 1:02:29um
  1666. 1:02:30and you also have to compute the
  1667. 1:02:32gradient with respect to your
  1668. 1:02:33parameters.
  1669. 1:02:36Okay? And what do those gradients look
  1670. 1:02:38like? Um this is just by
  1671. 1:02:40you know, the definition of uh these um
  1672. 1:02:43I guess it's just the chain rule here
  1673. 1:02:46um written in einsum notation. You take
  1674. 1:02:48the
  1675. 1:02:49the D loss D H2 which is H2.grad.
  1676. 1:02:54That's a a batch by out uh you know,
  1677. 1:02:57matrix.
  1678. 1:02:58And you multiply it by um W2
  1679. 1:03:02which is in by out.
  1680. 1:03:04And
  1681. 1:03:06and then um
  1682. 1:03:07that gives you uh
  1683. 1:03:09H1.grad which is D loss uh D H1.
  1684. 1:03:12And and which is batch in.
  1685. 1:03:15Okay? So, I always uh get this uh you
  1686. 1:03:17know, confused whenever you see if you
  1687. 1:03:19see, you know, learn calculus there's
  1688. 1:03:21like, you know, one of them has a
  1689. 1:03:22transpose and like I always forget which
  1690. 1:03:24order to put them in. And einsum I think
  1691. 1:03:26makes it very clear because if you we
  1692. 1:03:29you can remember even just like looking
  1693. 1:03:31at the scalar case that it has to look
  1694. 1:03:33like H1.grad is H2.grad times W2 and the
  1695. 1:03:37only question is, you know, what uh how
  1696. 1:03:39do you index things? And you index
  1697. 1:03:41things by basically looking at the shape
  1698. 1:03:43of these uh in the named dimensions. And
  1699. 1:03:46in this case, it's just a matrix
  1700. 1:03:48multiplication where you are summing
  1701. 1:03:51over the output, you know, dimension.
  1702. 1:03:54Okay? And when you do that, we can check
  1703. 1:03:57that so H1.grad was the thing that was
  1704. 1:03:59computed by loss.backward and this is
  1705. 1:04:02actually indeed the same thing that I
  1706. 1:04:04kind of wrote out here just as a sanity
  1707. 1:04:05check.
  1708. 1:04:07Okay, so the other thing you have to
  1709. 1:04:08compute is
  1710. 1:04:09uh
  1711. 1:04:10W the gradient with respect to the
  1712. 1:04:12parameters. And that's uh
  1713. 1:04:15D loss D H2 times H1.
  1714. 1:04:18So, it's the same kind of backward
  1715. 1:04:20message that's coming back but
  1716. 1:04:21multiplying with other thing that you're
  1717. 1:04:23not taking the
  1718. 1:04:25you know, the gradient you know, with
  1719. 1:04:26respect to.
  1720. 1:04:27And you just write out the the
  1721. 1:04:29dimensions and in this case, you're
  1722. 1:04:32summing over um the the batch dimension.
  1723. 1:04:37Okay.
  1724. 1:04:38So,
  1725. 1:04:40but the you know, the nice thing is that
  1726. 1:04:42if you just look at this, we know how
  1727. 1:04:44expensive uh you know, this is. The
  1728. 1:04:47number of flops is essentially the
  1729. 1:04:50product of all the dimensions. It
  1730. 1:04:52doesn't matter which ones you're you're
  1731. 1:04:54batching or not. It's it's basically you
  1732. 1:04:55enumerate over the IJK and you you
  1733. 1:04:58aggregate the way you aggregate is
  1734. 1:05:00different but um
  1735. 1:05:02the the flops is the you know, the same.
  1736. 1:05:06Okay, so notice that the backward pass
  1737. 1:05:08is exactly twice as expensive as the
  1738. 1:05:10forward pass and this is because you
  1739. 1:05:14have to take you know, compute two
  1740. 1:05:16uh gradients. One with respect to the
  1741. 1:05:18parameters the for each parameter and
  1742. 1:05:20then one with respect to the input the
  1743. 1:05:23other thing that's not the parameter.
  1744. 1:05:27Okay?
  1745. 1:05:29Maybe I'll pause any questions about
  1746. 1:05:31that.
  1747. 1:05:38>> [clears throat]
  1748. 1:05:40>> Okay, so now that was just for W2.
  1749. 1:05:43And you just need to apply this to all
  1750. 1:05:45the parameters in network and if you put
  1751. 1:05:47it all together,
  1752. 1:05:48you will see that for this network,
  1753. 1:05:52the forward pass is two times
  1754. 1:05:55the number of data points which is B
  1755. 1:05:56times the number of parameters flops.
  1756. 1:05:59And the backward pass is twice of that
  1757. 1:06:01which is four times the number of data
  1758. 1:06:02points times number of parameters flops.
  1759. 1:06:05And so, the grand total is 2 + 4 is 6
  1760. 1:06:09and uh 6 times number of data points and
  1761. 1:06:11parameters. So, this is where the 6 uh
  1762. 1:06:14ND uh comes from that you might have
  1763. 1:06:16seen in various places. It's just like
  1764. 1:06:19uh counting forward and you know,
  1765. 1:06:21backward.
  1766. 1:06:24So, we did this for just these uh deep
  1767. 1:06:27networks. Um but it turns out that this
  1768. 1:06:29is actually a good approximation for uh
  1769. 1:06:32transformers as well as soon as long as
  1770. 1:06:34the uh context length isn't too large.
  1771. 1:06:36If the context length is too large then
  1772. 1:06:38you get the context length squared and
  1773. 1:06:39that's
  1774. 1:06:40uh more flops that isn't in this kind of
  1775. 1:06:42accounting.
  1776. 1:06:48Okay, so uh let's talk a bit about So,
  1777. 1:06:51now we have gradients.
  1778. 1:06:52Now we have to the other piece when you
  1779. 1:06:54do training is we have to do
  1780. 1:06:56optimization.
  1781. 1:06:57So, uh like here's our deep network. Um
  1782. 1:07:00I'm going to just so that we're I'm not
  1783. 1:07:02just giving you uh what's in the
  1784. 1:07:04assignment one, I'm going to use um the
  1785. 1:07:07Adagrad optimizer which is um uh from
  1786. 1:07:11this is from 2011. This is predates
  1787. 1:07:13Adam. It's where you can think about it
  1788. 1:07:15somewhere in between SGD and um Adam.
  1789. 1:07:18And uh basically it's SGD where you um
  1790. 1:07:23you know, look at the the you know, the
  1791. 1:07:25second moments of the gradient.
  1792. 1:07:28Um momentum is where you look at the
  1793. 1:07:29first moment of gradients and um Adam is
  1794. 1:07:32where you, you know, uh combine the the
  1795. 1:07:34two of them. Um I'm not going to have
  1796. 1:07:36time to go into the optimizer details
  1797. 1:07:38here since we're focusing mostly on the
  1798. 1:07:41uh the usage.
  1799. 1:07:42So, we can define an um
  1800. 1:07:45you know, optimizer.
  1801. 1:07:46And
  1802. 1:07:48and when you compute the the you know,
  1803. 1:07:50the gradients, um
  1804. 1:07:52uh
  1805. 1:07:54and you compute gradients and then when
  1806. 1:07:57you take an optimizer step, um
  1807. 1:08:00if you're defining your new optimizer,
  1808. 1:08:02so in the assignment one, you're going
  1809. 1:08:03to uh uh implement Adam.
  1810. 1:08:06Um for each of the parameter uh
  1811. 1:08:10you know, groups which are, you know, in
  1812. 1:08:11this case W1 or W2,
  1813. 1:08:14um
  1814. 1:08:14we look at there's something called the
  1815. 1:08:17optimizer state which is uh of you know,
  1816. 1:08:21storage that you can the optimizer uses
  1817. 1:08:23while it's it's it's running.
  1818. 1:08:26Um and what we're computing in Adagrad
  1819. 1:08:29is uh something is the squared
  1820. 1:08:32gradients. Um the sum of the squared of
  1821. 1:08:34gradients.
  1822. 1:08:35Um so,
  1823. 1:08:37here we're getting it from the optimizer
  1824. 1:08:39state. We're updating it with the
  1825. 1:08:41current gradient and we're storing it
  1826. 1:08:43back.
  1827. 1:08:44And then after you update the the G2,
  1828. 1:08:47then you update the parameters.
  1829. 1:08:49>> [clears throat]
  1830. 1:08:50>> Okay? So, in Adagrad it's you basically
  1831. 1:08:53divide by the square root of the
  1832. 1:08:54gradient you know, average gradient
  1833. 1:08:56squared or sum of gradient squared
  1834. 1:08:58rather.
  1835. 1:09:01Okay, so um
  1836. 1:09:05let's see. So, I'm actually going to uh
  1837. 1:09:08maybe skip the
  1838. 1:09:10you know, the training
  1839. 1:09:12loop. Actually,
  1840. 1:09:14wait.
  1841. 1:09:15Hold on.
  1842. 1:09:17I think I actually skipped something I
  1843. 1:09:18didn't mean to.
  1844. 1:09:19Okay, so that was uh the optimizer
  1845. 1:09:23state. And now let's look at how much
  1846. 1:09:25memory the optimizer is using or in
  1847. 1:09:27general what is the memory usage. So,
  1848. 1:09:30for parameters in this network, there's
  1849. 1:09:33D squared parameters
  1850. 1:09:35for each of the L layers, and each
  1851. 1:09:38parameter takes two bytes if we're
  1852. 1:09:39storing an FP16, so that's number of
  1853. 1:09:41parameters.
  1854. 1:09:42Activations,
  1855. 1:09:44um
  1856. 1:09:45this is
  1857. 1:09:46uh you know
  1858. 1:09:47two times
  1859. 1:09:49um the
  1860. 1:09:51which is for BF16
  1861. 1:09:53you know, batch times D times the number
  1862. 1:09:54of layers for every layer you have an
  1863. 1:09:56activation. You have gradients,
  1864. 1:09:59uh which is basically a copy of all the
  1865. 1:10:01you know, the parameters. And this is
  1866. 1:10:03all the BF16, and the optimizer state is
  1867. 1:10:06for every
  1868. 1:10:08actually this should be the
  1869. 1:10:10I think this should be sorry, not this
  1870. 1:10:12I'll fix this as a typo. This should be
  1871. 1:10:13the number of parameters, not parameter
  1872. 1:10:14memory.
  1873. 1:10:15Um
  1874. 1:10:16and this should be number of parameters.
  1875. 1:10:17So, number of parameters
  1876. 1:10:19uh times two, and then this is four
  1877. 1:10:21times the number of parameters.
  1878. 1:10:24And the reason for this is that it's
  1879. 1:10:25customary to use FP32 for the optimizer
  1880. 1:10:29states for stability reasons. Um
  1881. 1:10:32obviously people have tried using uh
  1882. 1:10:34BF16, and what tends up happening is
  1883. 1:10:37that now you're taking squares and
  1884. 1:10:38you're averaging all those steps, and it
  1885. 1:10:40doesn't uh you know, it's not very
  1886. 1:10:42stable.
  1887. 1:10:43Um
  1888. 1:10:45and
  1889. 1:10:46for AdaGrad it's here we have four bytes
  1890. 1:10:48per parameter for storing the optimizer
  1891. 1:10:50state. Adam, you store the first order
  1892. 1:10:52bit
  1893. 1:10:52uh and the second order moments, so
  1894. 1:10:54that's eight bytes.
  1895. 1:10:56So, if you think about the optimizer
  1896. 1:10:57state is actually a lot of the memory um
  1897. 1:11:01uh
  1898. 1:11:01you know, used.
  1899. 1:11:03Um one note is that
  1900. 1:11:06you know, remember memory serves kind of
  1901. 1:11:08two purposes. One is like how much you
  1902. 1:11:10have to store this thing in your HBM.
  1903. 1:11:13But the other thing is that it has to be
  1904. 1:11:14shipped to the um
  1905. 1:11:17uh to
  1906. 1:11:19the [clears throat] um
  1907. 1:11:20to to the accelerators, and in general
  1908. 1:11:24um the optimizer state is not really the
  1909. 1:11:26bottleneck for for compute. So, the
  1910. 1:11:28amount of memory here is not really uh
  1911. 1:11:31so important for um you know,
  1912. 1:11:33performance in terms of speed, but it is
  1913. 1:11:36uh means that you can't fit large models
  1914. 1:11:38in your your memory.
  1915. 1:11:41>> [clears throat]
  1916. 1:11:42>> Okay. So, to put it um you know,
  1917. 1:11:44together
  1918. 1:11:45uh as we mentioned, the number of
  1919. 1:11:47parameters here is D times D times L,
  1920. 1:11:49and the number of flops is D times
  1921. 1:11:51um number of tokens or number of data
  1922. 1:11:53points times number of parameters.
  1923. 1:11:55So, for transformers, this is going to
  1924. 1:11:57be uh a bit more complicated, but you're
  1925. 1:11:59going to do that in assignment one, and
  1926. 1:12:01you do it more um carefully.
  1927. 1:12:04Okay, I'm going to skip the training
  1928. 1:12:05loop. This is just a sort of a general
  1929. 1:12:07um
  1930. 1:12:08um
  1931. 1:12:08kind of review.
  1932. 1:12:09Two things I want to quickly touch on
  1933. 1:12:11before we conclude. One is that
  1934. 1:12:14as we see, memory does uh
  1935. 1:12:17have an important effect both on the
  1936. 1:12:19ability to store large models, but also
  1937. 1:12:21sometimes um your speed. And so, in
  1938. 1:12:24general you want to reduce your memory
  1939. 1:12:26usage. So, there's two
  1940. 1:12:27um things that people you know,
  1941. 1:12:29typically do. One is uh gradient
  1942. 1:12:31accumulation.
  1943. 1:12:33Um so, in general you want to use batch
  1944. 1:12:36sizes that are large enough uh to
  1945. 1:12:39improve stability up to a a critical
  1946. 1:12:41batch size which Tatsu will talk about
  1947. 1:12:43later.
  1948. 1:12:44Um
  1949. 1:12:45and as we saw that activation memory
  1950. 1:12:48scales with batch size. So, you might
  1951. 1:12:50going to run out of memory at some point
  1952. 1:12:52if you have too large batch sizes. So,
  1953. 1:12:54gradient accumulation says, well
  1954. 1:12:57you compute, you have these micro
  1955. 1:12:59batches, and you compute the gradient on
  1956. 1:13:01the micro batches, and then you just
  1957. 1:13:03accumulate the gradients. You don't zero
  1958. 1:13:04out the gradients, and every
  1959. 1:13:06um batch size over micro batch size
  1960. 1:13:08steps, you update the parameters and a
  1961. 1:13:10zero out the gradients.
  1962. 1:13:12Okay? So, this is actually a very simple
  1963. 1:13:14code change um which allows you to save
  1964. 1:13:17um you know, on compute.
  1965. 1:13:19So, the other thing I'll quickly mention
  1966. 1:13:21is activation checkpointing. So, in
  1967. 1:13:23training
  1968. 1:13:24we need to
  1969. 1:13:26uh uh
  1970. 1:13:27store
  1971. 1:13:28in general the activations of all the
  1972. 1:13:30layers. This is done by you know,
  1973. 1:13:32default.
  1974. 1:13:33Um actually interestingly for inference,
  1975. 1:13:35we don't need to compute uh the
  1976. 1:13:37gradients, so we only need to store the
  1977. 1:13:38current layers activations.
  1978. 1:13:40Um but for training
  1979. 1:13:43the memory usage is
  1980. 1:13:45um
  1981. 1:13:46um B times D times um
  1982. 1:13:49two [clears throat] times L.
  1983. 1:13:52And
  1984. 1:13:53um the question is can you reduce this?
  1985. 1:13:56So, the activation checkpointing also
  1986. 1:13:58known as gradient checkpointing or
  1987. 1:13:59rematerialization. The key idea here is
  1988. 1:14:01that um in the forward pass, you just
  1989. 1:14:04keep the activations for only a subset
  1990. 1:14:06of the layers.
  1991. 1:14:07And in the backward pass, you recompute
  1992. 1:14:09the missing activations from the last
  1993. 1:14:11checkpoint. That's why it's called
  1994. 1:14:13checkpointing. And this is a general
  1995. 1:14:15trick that you see in you know, systems
  1996. 1:14:17which is trading off if you want to
  1997. 1:14:19reduce memory, you can just recompute
  1998. 1:14:21things.
  1999. 1:14:22Um so, here we're going to define this
  2000. 1:14:25uh to operationalize this. This is
  2001. 1:14:27actually fairly
  2002. 1:14:28um
  2003. 1:14:29you know,
  2004. 1:14:30uh
  2005. 1:14:31uh
  2006. 1:14:32easy actually. I didn't
  2007. 1:14:35Okay, maybe I didn't see the Okay, so
  2008. 1:14:37basically what you do here is you have
  2009. 1:14:41the same model except for
  2010. 1:14:43um
  2011. 1:14:44you just add torch.utils.checkpoint
  2012. 1:14:46on a layer.
  2013. 1:14:48So, that means do this computation, but
  2014. 1:14:50don't store any of the intermediate um
  2015. 1:14:52um
  2016. 1:14:53you know, activations. Only store what
  2017. 1:14:56is uh needed. So, this means that um
  2018. 1:15:01if you store all the activations,
  2019. 1:15:03remember in your deep network, you have
  2020. 1:15:06the thing that's pre the ReLU, and the
  2021. 1:15:07thing that's after the ReLU.
  2022. 1:15:09Right? So, that's a lot of of storage,
  2023. 1:15:12and instead if you do activation
  2024. 1:15:13checkpointing on those blocks where each
  2025. 1:15:16block has a linear and a ReLU, you don't
  2026. 1:15:18store the pre ReLU, and you save
  2027. 1:15:20basically half. Uh you can get away with
  2028. 1:15:23half the memory.
  2029. 1:15:24And then at the when you're doing the
  2030. 1:15:26backward pass, um
  2031. 1:15:28you need G3, but you can compute G3
  2032. 1:15:30easily from H2.
  2033. 1:15:33Okay, you can go farther than this.
  2034. 1:15:35Um you can say
  2035. 1:15:37well, I can store all the layers, but
  2036. 1:15:40you know, in extreme case, I can just
  2037. 1:15:43store no layers.
  2038. 1:15:44Um and so, that will be memory
  2039. 1:15:47maximally memory efficient. The only
  2040. 1:15:49thing is that the compute is going to be
  2041. 1:15:51L squared because every for every one of
  2042. 1:15:53these layers you have to start from the
  2043. 1:15:55beginning.
  2044. 1:15:56Um and maybe a sweet spot is if you
  2045. 1:15:59store the checkpoints at square root L
  2046. 1:16:01layers, that means your activation
  2047. 1:16:03memory is square root of L, and your
  2048. 1:16:06your com recomputation overhead is also
  2049. 1:16:09um you know, uh square root of L. So,
  2050. 1:16:12that's uh that's balanced.
  2051. 1:16:15Okay, so um to summarize this lecture
  2052. 1:16:18um everything is operating on on
  2053. 1:16:20tensors, parameters, gradients,
  2054. 1:16:21activations, optimizer states, data.
  2055. 1:16:25Um we introduce this Einops, which
  2056. 1:16:26hopefully you guys can embrace as a way
  2057. 1:16:28to think about tensor operations.
  2058. 1:16:31Um six times number of data points times
  2059. 1:16:32number of parameters is a formula which
  2060. 1:16:35now we have demystified as number of
  2061. 1:16:37flops per
  2062. 1:16:39um
  2063. 1:16:39for training step.
  2064. 1:16:41And uh
  2065. 1:16:43actually if this is training step, this
  2066. 1:16:44should be batch size. Um
  2067. 1:16:46and then we talked about arithmetic
  2068. 1:16:48intensity and roofline analysis, which
  2069. 1:16:50allows us to diagnose whether a
  2070. 1:16:52computation is memory bound or uh
  2071. 1:16:54compute bound. Um matrix multiplications
  2072. 1:16:57are compute bound. Basically everything
  2073. 1:16:59else is memory bound.
  2074. 1:17:01Um and then finally, gradient
  2075. 1:17:03accumulation activation checkpointing
  2076. 1:17:05are ways to reduce the memory
  2077. 1:17:07and by reducing the memory, uh that
  2078. 1:17:09allows you to use um bigger uh batch
  2079. 1:17:12sizes.
  2080. 1:17:14Okay, so that's it for today's lecture.
  2081. 1:17:16So, next week uh Tatsu will talk about
  2082. 1:17:18architectures.

About this transcript

This page contains the full transcript of YouTube transcript (kuYAsz7zspQ) , generated from the public captions YouTube serves with the video. The transcript has 12,028 words across 2,082 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.