Is This the End of LLMs? This Stanford Paper Could Make Nvidia's Memory Chips Unnecessary — Transcript

Full transcript

1. 0:00What if I
2. 0:00told you that
3. 0:01a single Stanford
4. 0:02research paper might
5. 0:03just render Nvidia's
6. 0:04most expensive memory
7. 0:06chips completely unnecessary?
8. 0:08Here is the
9. 0:08thing about the
10. 0:09AI industry right
11. 0:10now.
12. 0:11We have hit
13. 0:12a massive wall.
14. 0:13Everyone talks about
15. 0:14compute limits.
16. 0:15But the real
17. 0:16silent killer of
18. 0:17traditional language models
19. 0:19is memory bandwidth.
20. 0:20We physically cannot
21. 0:21build memory chips
22. 0:22fast enough.
23. 0:23We need them
24. 0:24to feed these
25. 0:24massive AI models.
26. 0:26Let us back
27. 0:27up and look
28. 0:28at how things
29. 0:28work today.
30. 0:29You can think
31. 0:30of a standard
32. 0:30language model as
33. 0:31a gigantic matrix
34. 0:33of numbers.
35. 0:34You give this
36. 0:35model a prompt.
37. 0:36It predicts the
38. 0:37next most likely
39. 0:38token.
40. 0:39A token is
41. 0:40just a short
42. 0:40piece of word.
43. 0:42It predicts this
44. 0:43token based on
45. 0:44your prompt and
46. 0:44its training data.
47. 0:46Then it repeats
48. 0:47this process.
49. 0:48It moves forward
50. 0:49one token at
51. 0:50a time.
52. 0:51It does this
53. 0:53until the entire
54. 0:53answer is complete.
55. 0:55It is basically
56. 0:55a really big
57. 0:56autocomplete system.
58. 0:58This process takes
59. 0:59a huge amount
60. 1:00of resources.
61. 1:01To get a
62. 1:02one-thousand-word answer, the
63. 1:03model runs one
64. 1:04thousand times.
65. 1:06It runs in
66. 1:06a strict sequence.
67. 1:07During every single
68. 1:08pass, the graphics
69. 1:10card needs to
70. 1:11read the model
71. 1:11parameters.
72. 1:12It reads them
73. 1:13from the video
74. 1:13memory.
75. 1:14A massive model
76. 1:15has trillions of
77. 1:16numbers.
78. 1:16The processor has
79. 1:18to load these
80. 1:18numbers one thousand
81. 1:20separate times.
82. 1:21This method is
83. 1:22incredibly slow.
84. 1:24It also consumes
85. 1:25a massive amount
86. 1:25of electricity.
87. 1:27But most importantly,
88. 1:28it requires extreme
89. 1:30memory bandwidth.
90. 1:31The speed of
91. 1:32the memory limits
92. 1:33the speed of
93. 1:33the AI.
94. 1:35The processor often
95. 1:36sits there.
96. 1:37It waits for
97. 1:38the memory to
98. 1:38send the data.
99. 1:39And the problem
100. 1:40gets even worse
101. 1:41with long conversations.
102. 1:43The model has
103. 1:44to remember all
104. 1:45the previous tokens.
105. 1:46stores them in
106. 1:47a temporary cache.
107. 1:49We will call
108. 1:49this the KV
109. 1:50cache.
110. 1:51As the conversation
111. 1:52gets longer, this
112. 1:53cache grows.
113. 1:54In long-context models,
114. 1:56this memory cache
115. 1:58grows way too
116. 1:58fast.
117. 1:59It eats up
118. 2:00all the available
119. 2:00video memory.
120. 2:02Hardware companies try
121. 2:03to fix this
122. 2:03by packing more
123. 2:04memory onto the
124. 2:05cards.
125. 2:06They use High
126. 2:07Bandwidth Memory.
127. 2:08But these chips
128. 2:09are incredibly expensive.
129. 2:11They are also
130. 2:12very hard to
131. 2:13manufacture.
132. 2:14This creates a
133. 2:15massive hardware bottleneck.
134. 2:17So how do
135. 2:18we fix this
136. 2:18without just buying
137. 2:20more expensive memory
138. 2:21chips?
139. 2:22A research team
140. 2:22from Stanford University
141. 2:24recently dropped a
142. 2:25new architecture.
143. 2:27It is called
144. 2:27Next Concept Modeling.
145. 2:29Or NCM for
146. 2:30short.
147. 2:30NCM takes a
148. 2:31radically different approach.
149. 2:33It generates the
150. 2:34entire answer in
151. 2:34parallel.
152. 2:35To do this,
153. 2:36the researchers borrowed
154. 2:37an idea from
155. 2:38image generation models.
156. 2:39Think about how
157. 2:40an AI generates
158. 2:41a picture.
159. 2:42It start with
160. 2:43a blurry layer.
161. 2:44This layer represents
162. 2:45a rough image.
163. 2:46Then the next
164. 2:47layer adds more
165. 2:48detail.
166. 2:50You repeat this
167. 2:50until you get
168. 2:51a sharp final
169. 2:52image.
170. 2:52NCM does this
171. 2:54exact same thing
172. 2:54with text.
173. 2:56Words live inside
174. 2:57a collection of
175. 2:58layers.
176. 2:58At the top
177. 2:59layer, you have
178. 3:00a very rough
179. 3:00outline of a
180. 3:01concept.
181. 3:02The researchers call
182. 3:03these concepts discrete
183. 3:04codes.
184. 3:05A code is
185. 3:05just a tiny
186. 3:06digital representation of
187. 3:07an idea.
188. 3:08In each new
189. 3:09layer, the concept
190. 3:10gets refined.
191. 3:11The model stores
192. 3:12only the new
193. 3:13refinements as additional
194. 3:15codes.
195. 3:16So every subsequent
196. 3:17layer has more
197. 3:18codes than the
198. 3:19layer before it.
199. 3:21This continues down
200. 3:22to the bottom
201. 3:22layer.
202. 3:23At this final
203. 3:24layer, every code
204. 3:25becomes a real
205. 3:26output token.
206. 3:27If you combine
207. 3:28all these layers,
208. 3:29you get the
209. 3:30final text.
210. 3:31So you ask
211. 3:31the NCM model
212. 3:32a question.
213. 3:33The model runs
214. 3:34exactly one time
215. 3:35per layer.
216. 3:36Each layer predicts
217. 3:37the entire output
218. 3:38in parallel.
219. 3:40The top layers
220. 3:41are very coarse.
221. 3:42If you looked
222. 3:43at the words
223. 3:43in the top
224. 3:45layer, they would
225. 3:45look like complete
226. 3:46garbage.
227. 3:46But the fidelity
228. 3:48increases layer by
229. 3:49layer.
230. 3:49By the final
231. 3:50pass, you get
232. 3:51a high-quality text
233. 3:52output.
234. 3:53This text is
235. 3:54the answer to
236. 3:55your prompt.
237. 3:56Let us go
238. 3:56back to our
239. 3:57one-thousand-word example.
240. 3:59traditional model takes
241. 4:01one thousand passes
242. 4:02to write that
243. 4:02answer.
244. 4:03The Stanford NCM
245. 4:05model takes just
246. 4:0632 passes.
247. 4:07The graphics card
248. 4:08only reads 32
249. 4:10gigabytes of memory
250. 4:11instead of one
251. 4:11thousand.
252. 4:12This is a
253. 4:13massive drop in
254. 4:14memory bandwidth.
255. 4:15It is more
256. 4:15than 30 times
257. 4:16as efficient.
258. 4:17But generating everything
259. 4:19in parallel creates
260. 4:20a huge problem.
261. 4:22How do you
262. 4:22force an AI
263. 4:23to actually learn
264. 4:24a high-level concept
265. 4:25instead of just
266. 4:26memorizing the data?
267. 4:28Wait, how does
268. 4:30a computer even
269. 4:30know what a
270. 4:31concept is?
271. 4:32This brings us
272. 4:33to the first
273. 4:34technical secret.
274. 4:35The Stanford researchers
275. 4:36used a brutal
276. 4:37training trick.
277. 4:39They call it
278. 4:39nested dropout.
279. 4:41They use a
280. 4:41special program to
281. 4:42turn text into
282. 4:43those discrete codes.
283. 4:45During training, the
284. 4:46researchers randomly drop
285. 4:48the detailed bottom
286. 4:48layers.
287. 4:49Imagine you need
288. 4:50to study for
289. 4:51a big math
290. 4:51test.
291. 4:52But someone rips
292. 4:53out the last
293. 4:54three chapters of
294. 4:55your textbook.
295. 4:56You have to
296. 4:56deeply understand the
297. 4:57core concepts.
298. 4:59You cannot just
299. 4:59memorize the formulas
300. 5:01at the end
301. 5:01of the book.
302. 5:02If you are
303. 5:02enjoying our chat,
304. 5:03I have something
305. 5:04important to share
306. 5:05with you.
307. 5:07Looks like YouTube
308. 5:07algorithm is not
309. 5:08a big fan
310. 5:09of my videos.
311. 5:11This is why,
312. 5:11guys, I need
313. 5:12your support.
314. 5:13Hit the like
315. 5:13button to tell
316. 5:14the Google that
317. 5:15the content is
318. 5:16dope.
319. 5:16If you use
320. 5:17social media and
321. 5:18have friends in
322. 5:18tech, please share
323. 5:20this video.
324. 5:21If you are
325. 5:21not subscribed yet,
326. 5:23please do get
327. 5:23subscribed right now.
328. 5:25YouTube algorithm should
329. 5:26know that the
330. 5:27tech community is
331. 5:28not dead.
332. 5:29But getting back
333. 5:30to our topic.
334. 5:31Nested dropout does
335. 5:33exactly this to
336. 5:34the AI model.
337. 5:35It puts a
338. 5:35lot of pressure
339. 5:36on the system.
340. 5:37It forces the
341. 5:38top layers to
342. 5:39pack maximum meaning
343. 5:40into a tiny
344. 5:41sequence of codes.
345. 5:42The model cannot
346. 5:43rely on the
347. 5:44bottom layers to
348. 5:45fix the output
349. 5:46later.
350. 5:47The bottom layers
351. 5:48might be missing
352. 5:49entirely.
353. 5:50So the top
354. 5:51layers actually learn
355. 5:52true concepts.
356. 5:53They do not
357. 5:54just output compressed
358. 5:55static.
359. 5:56This is why
360. 5:57NCM actually understands
361. 5:59what it writes.
362. 6:00It learns the
363. 6:01core idea before
364. 6:02it writes the
365. 6:03specific words.
366. 6:04That is actually
367. 6:06wild.
368. 6:06But this coarse-to-fine
369. 6:07design has a
370. 6:08major vulnerability.
371. 6:10What happens if
372. 6:11the model makes
373. 6:12a mistake at
374. 6:12the very top
375. 6:13layer?
376. 6:14This brings us
377. 6:14to compounding errors.
378. 6:16If the top
379. 6:17layer hallucinates a
380. 6:18bad concept, that
381. 6:20mistake ripples down.
382. 6:22It corrupts the
383. 6:23final text.
384. 6:24It creates total
385. 6:25error death spiral.
386. 6:26To fix this,
387. 6:27the team engineered
388. 6:28something called conditioning
389. 6:30corruption.
390. 6:31This is our
391. 6:31second technical secret.
392. 6:33During training, they
393. 6:34deliberately inject wrong
394. 6:35codes into the
395. 6:36middle layers.
396. 6:37They use corrupted
397. 6:39data.
398. 6:39they keep the
399. 6:40correct final answers
400. 6:41intact.
401. 6:42Imagine you teach
402. 6:43a kid to
403. 6:44ride a bike.
404. 6:45You randomly push
405. 6:46the bike slightly
406. 6:48off balance.
407. 6:49The kid has
408. 6:50to learn how
409. 6:50to correct their
410. 6:52posture.
411. 6:52They learn to
412. 6:53recover their balance
413. 6:54before they crash
414. 6:55into a tree.
415. 6:56Conditioning corruption does
416. 6:57this for the
417. 6:58AI.
418. 6:59It trains the
419. 7:00lower layers to
420. 7:00be resilient.
421. 7:01It forces the
422. 7:03model to self-correct
423. 7:04its own mistakes.
424. 7:06It fixes bad
425. 7:07concepts before they
426. 7:08turn into garbage
427. 7:09text.
428. 7:09Because nobody wants
429. 7:10their AI to
430. 7:11confidently output complete
431. 7:13garbage faster than
432. 7:14before.
433. 7:15So now we
434. 7:16have a model
435. 7:16that is fast.
436. 7:18It is memory-efficient.
437. 7:19And it fixes
438. 7:20its own errors.
439. 7:22But it is
440. 7:22still a very
441. 7:23small model.
442. 7:25How does a
443. 7:25tiny model compete
444. 7:26with the reasoning
445. 7:27power of a
446. 7:28massive frontier model?
447. 7:30This is where
448. 7:31we see a
449. 7:31massive developer trend
450. 7:32for 2026.
451. 7:34It is called
452. 7:34test-time scaling.
453. 7:36And this is
454. 7:37our third technical
455. 7:38secret.
456. 7:39NCM is extremely
457. 7:40fast.
458. 7:41It generates entire
459. 7:42blocks of text
460. 7:44in a fraction
461. 7:44of the normal
462. 7:44time.
463. 7:45You can use
464. 7:46this saved time
465. 7:47to buy more
466. 7:48intelligence.
467. 7:49Instead of just
468. 7:50taking the first
469. 7:50answer, you generate
470. 7:5220 different candidate
471. 7:53answers.
472. 7:54You generate all
473. 7:55of them in
474. 7:55parallel.
475. 7:57Then you run
476. 7:57a mathematical filter.
477. 7:59This filter is
478. 8:00called Minimum Bayes
479. 8:01Risk.
480. 8:02Or MBR.
481. 8:03MBR looks at
482. 8:04all 20 answers.
483. 8:05It mathematically selects
484. 8:07the candidate that
485. 8:08has the highest
486. 8:08consensus among the
487. 8:09group.
488. 8:10Imagine you have
489. 8:11a really tough
490. 8:12coding problem.
491. 8:13You could ask
492. 8:13one senior developer
493. 8:15for an answer.
494. 8:16Or you could
495. 8:17ask 20 junior
496. 8:18developers.
497. 8:18Then you find
498. 8:19the common threads
499. 8:20in all their
500. 8:21answers.
501. 8:22You mathematically pick
502. 8:23the best overall
503. 8:24solution.
504. 8:25You literally trade
505. 8:26the model's extreme
506. 8:27speed to boost
507. 8:28its reasoning at
508. 8:29runtime.
509. 8:30You do not
510. 8:31need to train
511. 8:31a bigger model.
512. 8:32You just run
513. 8:33the small model
514. 8:34multiple times.
515. 8:35The researchers tested
516. 8:37this exact theory
517. 8:38on games of
518. 8:38Sudoku.
519. 8:39They used a
520. 8:4025 million parameter
521. 8:42NCM model.
522. 8:43It solved 86
523. 8:45out of 100
524. 8:46hard boards.
525. 8:47The researchers compared
526. 8:49these results against
527. 8:50a standard language
528. 8:51model of the
529. 8:52same size.
530. 8:53The standard model
531. 8:54solved exactly one
532. 8:55board.
533. 8:56That is an
534. 8:568600 percent improvement.
535. 8:59I love this
536. 9:00part.
537. 9:01It shows how
538. 9:01speed can directly
539. 9:02translate into better
540. 9:04logic.
541. 9:04But why should
542. 9:05the massive hardware
543. 9:06companies care about
544. 9:07an AI playing
545. 9:08Sudoku?
546. 9:09Because this paper
547. 9:10exposes a fatal
548. 9:11flaw in the
549. 9:12current AI economy.
550. 9:13NCM shifts the
551. 9:15main bottleneck entirely.
552. 9:17It moves the
553. 9:18bottleneck from memory
554. 9:19bandwidth to pure
555. 9:20compute.
556. 9:21This is the
557. 9:22billion-dollar hardware flip.
558. 9:24As always, there
559. 9:25is no free
560. 9:25lunch.
561. 9:26Look at the
562. 9:27bottom layer of
563. 9:27the NCM model.
564. 9:29It has four
565. 9:30times as many
566. 9:31codes as output
567. 9:32tokens.
568. 9:33Evaluating all those
569. 9:34extra codes takes
570. 9:35more processing power.
571. 9:37The architecture drastically
572. 9:38reduces the memory
573. 9:40bandwidth you need.
574. 9:41But it requires
575. 9:42more raw compute.
576. 9:43Right now, the
577. 9:44industry buys expensive
578. 9:45Nvidia chips mostly
579. 9:47for their memory.
580. 9:49We need that
581. 9:49High Bandwidth Memory
582. 9:50to hold the
583. 9:51massive cache.
584. 9:52But NCM does
585. 9:54not need that
586. 9:54massive memory bandwidth.
587. 9:56It just needs
588. 9:57raw compute.
589. 9:58This means we
590. 9:59could theoretically run
591. 10:00frontier-level AI on
592. 10:02consumer-grade silicon.
593. 10:04You could use
594. 10:04standard chips that
595. 10:05have massive compute
596. 10:07but limited video
597. 10:08memory.
598. 10:09This makes the
599. 10:10current token-by-token paradigm
600. 10:12look bloated.
601. 10:12It makes it
602. 10:13look like a
603. 10:14relic.
604. 10:14An algorithmic left-field
605. 10:16breakthrough might actually
606. 10:17solve our massive
607. 10:18hardware supply chain
608. 10:19problem.
609. 10:20It threatens to
610. 10:21upend the entire
611. 10:22hardware monopoly.
612. 10:24Will NCM hold
613. 10:25up when we
614. 10:26scale it to
615. 10:26trillions of parameters?
616. 10:28We do not
617. 10:29really know yet.
618. 10:30But it proves
619. 10:31something important.
620. 10:32We should not
621. 10:33accept today's hardware
622. 10:34limits as permanent
623. 10:35laws of physics.
624. 10:36New model architectures
625. 10:38show us that
626. 10:39massive breakthroughs are
627. 10:40still possible.
628. 10:41To sum it
629. 10:42all up.
630. 10:43NCM ditches token-by-token
631. 10:45generation entirely.
632. 10:46It generates text
633. 10:48in parallel layers.
634. 10:49It uses nested
635. 10:50dropout to force
636. 10:52the model to
637. 10:53learn real concepts.
638. 10:54It uses conditioning
639. 10:55corruption to fix
640. 10:56its own errors.
641. 10:58And it uses
642. 10:58test-time scaling to
643. 11:00artificially boost its
644. 11:01reasoning.
645. 11:03It trades memory
646. 11:03bandwidth for pure
647. 11:05compute.
648. 11:05If you want
649. 11:06to stay ahead
650. 11:06of these massive
651. 11:07algorithmic shifts, hit
652. 11:09the like button.
653. 11:10Subscribe to the
654. 11:11channel.
655. 11:11And let me
656. 11:12know in the
657. 11:12comments below if
658. 11:13you think consumer
659. 11:14graphics cards will
660. 11:15finally run frontier
661. 11:17models.
662. 11:17Thanks for watching.
663. 11:18See you in
664. 11:19the next one.
665. 11:20Bye