YouTube2Text

Ist das das Ende von LLMs? Ein Stanford-Paper könnte alles ändern — Transcript

by Tjark Harjes · 2,050 words · 346 segments · language en · Watch on YouTube

Full transcript

  1. 0:00What if I told you that a single paper
  2. 0:02from Stanford could make Nvidia's most
  3. 0:04expensive chips practically obsolete?
  4. 0:06Because here is the problem with the AI
  5. 0:07industry right now. We have hit a
  6. 0:09massive wall. Everyone is talking about
  7. 0:12compute limits, but the real silent
  8. 0:14killer of classic language models is
  9. 0:16memory bandwidth.
  10. 0:17Memory chips currently cannot be built
  11. 0:19fast enough to feed the massive AI
  12. 0:21models and keep them running.
  13. 0:23You can think of a standard language
  14. 0:25model as a huge matrix of numbers. You
  15. 0:28give the model a prompt and it predicts
  16. 0:30the next likely token. A token is
  17. 0:31essentially nothing more than a part of
  18. 0:33a word. Then this process repeats until
  19. 0:35the complete answer is finished. So you
  20. 0:37can basically think of it as a very
  21. 0:40sophisticated autocomplete system. This
  22. 0:42procedure costs an extreme amount of
  23. 0:43computing power.
  24. 0:44If you want to add 1,000 words to a
  25. 0:47prompt, meaning you want an answer that
  26. 0:49is 1,000 words or 1,000 tokens long,
  27. 0:52this process must run 1,000 times.
  28. 0:55Each time, everything has to be loaded
  29. 0:57into memory or read out of memory and
  30. 0:59sent through the entire model.
  31. 1:01With large models featuring trillions of
  32. 1:03parameters or more, this is extremely
  33. 1:05slow and also consumes an extreme amount
  34. 1:07of power. But in particular, it also
  35. 1:09requires an extreme amount of memory
  36. 1:11bandwidth. This means the speed of the
  37. 1:13memory also limits the speed of the AI.
  38. 1:17The processor often sits there waiting
  39. 1:19for data from memory so it can continue
  40. 1:21working.
  41. 1:22With larger models, this problem becomes
  42. 1:24even worse, or rather more intense. In
  43. 1:26the process, the model must remember all
  44. 1:29previous tokens and saves them in a
  45. 1:31temporary cache, the so-called KV-cache
  46. 1:34.
  47. 1:34The longer the conversation, the larger
  48. 1:37this cache becomes, and with models that
  49. 1:39have long chats and large contexts, this
  50. 1:41cache might take up the entire video
  51. 1:43memory.
  52. 1:45Hardware manufacturers are trying to
  53. 1:47solve this problem by simply putting
  54. 1:49more memory on the chips to provide more
  55. 1:51storage capacity.
  56. 1:53However, these chips are incredibly
  57. 1:54expensive and difficult to manufacture.
  58. 1:56Since memory is limited, there must be
  59. 1:58some solution to fix this.
  60. 2:00That is where a research team from
  61. 2:02Stanford comes into play, having
  62. 2:03developed a completely new approach.
  63. 2:05This new architecture is called Next
  64. 2:07Concept Modeling, or NCM for short, and
  65. 2:09the approach is truly radically
  66. 2:11different because instead of token by
  67. 2:13token, the entire answer is generated in
  68. 2:16parallel.
  69. 2:17To do this, the researchers looked at
  70. 2:19the concept from image generation, where
  71. 2:21the AI first creates a very, very rough
  72. 2:23, blurry image and then makes the image
  73. 2:26finer and more detailed with every
  74. 2:28iteration.
  75. 2:29NCM basically does the exact same thing
  76. 2:31with text.
  77. 2:32At the top level, a rough outline, a
  78. 2:35concept for the text, is stored, and
  79. 2:37this outline is then described as
  80. 2:39so-called discrete codes. But what
  81. 2:42exactly is such a code? You can imagine
  82. 2:44it as if the system has created a
  83. 2:46dictionary, and in this dictionary, all
  84. 2:47the references and codes are stored.
  85. 2:50It contains a fixed list of building
  86. 2:52blocks, the individual codes, and a code
  87. 2:54is simply a number from this dictionary
  88. 2:56. The exciting part is that, at the top
  89. 2:59level, very few codes describe the
  90. 3:01entire text. This means there is no room
  91. 3:03for details; rather, the entire text is
  92. 3:06represented by these few codes, focusing
  93. 3:08only on the concepts and the rough
  94. 3:10outline, much like the blurriness we see
  95. 3:13at the start of AI image generation.
  96. 3:15Accordingly, only what is general and
  97. 3:17important is saved.
  98. 3:19What is the text about, and how is its
  99. 3:20content structured?
  100. 3:22Every level below has more codes and
  101. 3:24represents additional knowledge about
  102. 3:27the text or extra information, until at
  103. 3:30the lowest level, there is a code for
  104. 3:32every word or token that describes it.
  105. 3:36I know the concept isn't easy to grasp,
  106. 3:38but it is truly the core, because you
  107. 3:40can only understand NCM if you
  108. 3:41understand this concept. Let me try to
  109. 3:43explain it using the example of an essay
  110. 3:44.
  111. 3:45When you want to write an essay, you
  112. 3:47don't just start writing blindly; you
  113. 3:50first think about the outline and the
  114. 3:52general content.
  115. 3:54Then you might continue by drafting the
  116. 3:56individual sentences or chapters, and
  117. 3:58only at the end do you start refining
  118. 4:01every single word. And that is exactly
  119. 4:03how this NCM concept, this NCM
  120. 4:05architecture, approaches text generation
  121. 4:07.
  122. 4:08From the rough outline to individual
  123. 4:10chapters and sentences, down to the
  124. 4:12actual individual words that are then
  125. 4:15perfected.
  126. 4:16When you ask an NCM model a question, it
  127. 4:19doesn't work by adding a single word
  128. 4:21each time; instead, the entire text is
  129. 4:24generated in every layer, but it becomes
  130. 4:26increasingly refined and detailed, so
  131. 4:29NCM runs through each layer once.
  132. 4:32Let's just assume you want to generate
  133. 4:34an answer of 1000 words. In a classic
  134. 4:36model, as I explained earlier, it would
  135. 4:39have to run 1000 times.
  136. 4:41In an NCM model, however, according to
  137. 4:43the paper, it only takes about eight to
  138. 4:46twelve passes, one for each layer.
  139. 4:48So, eight to twelve layers, and each one
  140. 4:50is processed once.
  141. 4:52If you compare this with other parallel
  142. 4:54methods, this NCM process is still about
  143. 4:5731 times faster than others, which is an
  144. 4:59extreme gain.
  145. 5:03While the total required computing power
  146. 5:05does not decrease, the memory bandwidth
  147. 5:07that must be used is reduced, and the
  148. 5:08focus is shifted more toward the
  149. 5:10individual calculation.
  150. 5:12But for this architecture to work at all
  151. 5:15, the researchers applied a few tricks.
  152. 5:18First, they trained their own translator
  153. 5:21and applied the first trick, so-called
  154. 5:23nested dropout.
  155. 5:25In about every second training step, the
  156. 5:28researchers cut off the lower, finer
  157. 5:30layers, so the text must be generated
  158. 5:32completely by the upper, general layers
  159. 5:34.
  160. 5:35This way, the layers learn to formulate
  161. 5:38the core idea of the text first, before
  162. 5:41really refining it and generating every
  163. 5:44word.
  164. 5:45Imagine you are on the phone and the
  165. 5:47connection is really bad. Then, the
  166. 5:49person tries to convey as much important
  167. 5:52content as possible in as few words as
  168. 5:54possible without getting lost in details
  169. 5:57. And that is exactly the same principle
  170. 5:59by which the upper layers are trained
  171. 6:01based on the concept.
  172. 6:03However, there is also a potential for
  173. 6:05error here, because if you rely on the
  174. 6:07upper levels for the rough concept, then
  175. 6:10an error is also passed on.
  176. 6:13If the upper level hallucinates a wrong
  177. 6:15concept and has misunderstood the text
  178. 6:17that should be written, then this error
  179. 6:19carries over to all layers below it,
  180. 6:21which leads to a potentially completely
  181. 6:23wrong text being written.
  182. 6:26To counteract this, the researchers
  183. 6:28developed so-called conditioning
  184. 6:30corruption.
  185. 6:31This is simply a process where the
  186. 6:33researchers, during training,
  187. 6:35intentionally input incorrect and flawed
  188. 6:37data sets into the middle layers so that
  189. 6:40it has to be corrected later, and this
  190. 6:42is how the model was effectively taught
  191. 6:44to correct errors that originated from
  192. 6:46above by itself later on.
  193. 6:48A kind of error correction directly
  194. 6:50within the model itself. A good
  195. 6:52comparison is perhaps a child learning
  196. 6:54to ride a bike.
  197. 6:55The child first learns to ride on a flat
  198. 6:58surface until they can do it roughly,
  199. 7:00and later, if the child can ride really
  200. 7:02well and you were to lightly bump the
  201. 7:04bike and push the child slightly off
  202. 7:06balance—I know, it might not be the best
  203. 7:08example—or they ride on an uneven
  204. 7:10surface, then the child has to correct
  205. 7:13and constantly adjust their posture,
  206. 7:15learning to ride more safely as a result
  207. 7:17. Through these two tricks, we now have
  208. 7:19a model that is really fast and corrects
  209. 7:22its own errors.
  210. 7:23However, that does not yet mean it can
  211. 7:25keep up with the processing power of
  212. 7:26large frontier models.
  213. 7:28This is where a third topic comes into
  214. 7:30play so that one can achieve similar
  215. 7:32computing power as with larger models
  216. 7:34using a significantly smaller model.
  217. 7:36This third trick is called Test Time
  218. 7:38Scaling and is based simply on taking
  219. 7:41advantage of the fact that NCM is
  220. 7:43incredibly fast. This allows us to
  221. 7:45generate multiple versions of an answer
  222. 7:48at once, using only the best one in the
  223. 7:51end.
  224. 7:52Instead of working with just one answer
  225. 7:55, we simply generate 20 answers in
  226. 7:57parallel, for example, which are then
  227. 7:59filtered out using a mathematical
  228. 8:01process.
  229. 8:03This way, only the best answer remains
  230. 8:04at the end.
  231. 8:05This concept can be compared to giving a
  232. 8:08programming task to 20 junior developers
  233. 8:11instead of one senior developer to work
  234. 8:13on it intensely, and then just picking
  235. 8:16the best result from them.
  236. 8:19In the paper from Stanford, this
  237. 8:21actually led to better results in
  238. 8:23translations and summaries compared to
  239. 8:25larger models.
  240. 8:26But there is another test the
  241. 8:27researchers conducted that I find even
  242. 8:29more exciting. They had the models play
  243. 8:31Sudoku.
  244. 8:33Normally, this is a real endurance test
  245. 8:35for models because one wrong entry means
  246. 8:37that a lot of work has to be redone or
  247. 8:40recalculated.
  248. 8:41Here, parallel processing is an extreme
  249. 8:44advantage. In this case, the researchers
  250. 8:47compared an MCM model with 25 million
  251. 8:50parameters to a standard language model
  252. 8:53also with 25 million parameters.
  253. 8:56The result was that of the difficult
  254. 8:59Sudoku puzzles, the normal model only
  255. 9:02solved about one in several attempts,
  256. 9:05while the MCM achieved an 86%success
  257. 9:08rate, which is a huge lead.
  258. 9:12To be fair, it must also be said that
  259. 9:15the researchers used other parallel
  260. 9:17models, diffusion models, which achieved
  261. 9:20about 80%.
  262. 9:22This means that the harder the puzzles
  263. 9:24became, the researchers found, the
  264. 9:26greater the gap grew; the MCM model
  265. 9:28performed better, but there are other
  266. 9:30parallel methods that could reach
  267. 9:32similar performance for these types of
  268. 9:34tasks.
  269. 9:35But why should hardware manufacturers
  270. 9:37even be interested in a model being good
  271. 9:40at playing Sudoku? Is this transferable
  272. 9:42to other areas? You could say in part,
  273. 9:45because MCM does not eliminate the
  274. 9:47bottleneck.
  275. 9:48However, it shifts the main bottleneck
  276. 9:50away from memory bandwidth, which is
  277. 9:52currently in very high demand and
  278. 9:54heavily taxed, toward pure computing
  279. 9:56power. It is just important to
  280. 9:58understand that nothing comes for free.
  281. 10:00In the end, the same amount of
  282. 10:02calculation must be done, only now the
  283. 10:04processor, which was previously waiting
  284. 10:06, can be used more and does not have to
  285. 10:08wait for the memory.
  286. 10:10That is an extreme advantage, but it
  287. 10:12naturally means there may be other
  288. 10:13bottlenecks that aren't apparent right
  289. 10:16now and will need to be solved later.
  290. 10:19Theoretically, this could make local AI
  291. 10:21on personal computers much faster, as
  292. 10:24memory bandwidth is often the biggest
  293. 10:26bottleneck there.
  294. 10:29For frontier models running in huge data
  295. 10:32centers, where all parameters must be
  296. 10:34loaded anyway, the advantage might be
  297. 10:37smaller than you'd expect, since NCM has
  298. 10:39so far only been tested on models with
  299. 10:42under a billion parameters.
  300. 10:44Nevertheless, I find the approach
  301. 10:46extremely exciting, and it makes
  302. 10:47token-by-token generation seem almost
  303. 10:49obsolete, like a relic, though that's
  304. 10:51not necessarily the conclusion to draw.
  305. 10:54Instead, it's important to think about
  306. 10:56how to solve current bottlenecks through
  307. 10:58new architectures, where NCM is a very
  308. 11:00interesting approach, because an
  309. 11:02algorithmic breakthrough like NCM can
  310. 11:04completely redefine the hardware
  311. 11:06requirements.
  312. 11:08The big question that needs to be
  313. 11:10answered now is whether NCM can be
  314. 11:12scaled to several trillion parameters to
  315. 11:14truly keep up with frontier models.
  316. 11:17Honestly, that question remains open.
  317. 11:19But it shows something else important:
  318. 11:21we shouldn't just accept the current
  319. 11:23hardware limits as given.
  320. 11:26Instead, we need to look at how to solve
  321. 11:28these hardware limits or even circumvent
  322. 11:30them through new algorithms and
  323. 11:31architectures.
  324. 11:33From my perspective, it would be a huge
  325. 11:35advantage not to have to wait forever
  326. 11:38for the AI and to get answers very
  327. 11:40quickly during everyday work.
  328. 11:43To summarize briefly, the biggest
  329. 11:45difference is that NCM no longer
  330. 11:46generates token by token, but actually
  331. 11:49everything simultaneously.
  332. 11:51In this process, nested dropout is
  333. 11:53applied to ensure that both broad
  334. 11:55concepts and detailed generation are
  335. 11:57possible.
  336. 11:58At the same time, errors are
  337. 12:00self-corrected, and it even ensures
  338. 12:02similar computing performance through
  339. 12:05the parallel generation of different
  340. 12:07answers.
  341. 12:08I have to say, I find this topic super
  342. 12:10exciting, especially what is happening
  343. 12:11with architectures. If you enjoyed this
  344. 12:13video, please leave a like and subscribe
  345. 12:14. I look forward to your take in the
  346. 12:16comments.

About this transcript

This page contains the full transcript of Ist das das Ende von LLMs? Ein Stanford-Paper könnte alles ändern by Tjark Harjes, generated from the public captions YouTube serves with the video. The transcript has 2,050 words across 346 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.