Is This the END of LLMs? NCM Could Change Everything — Transcript

Full transcript

1. 0:00Hello world. I'm an unemployed ex-big
2. 0:02tech software engineer with 25 years of
3. 0:04experience in tech. Now that I'm not
4. 0:07working a day job, I got plenty of time
5. 0:10on my hands to keep up with advances in
6. 0:12AI. And here's the thing, most of the
7. 0:15conversations around AI today, it
8. 0:18revolves around the large language model
9. 0:21or LLM. You can think of a large
10. 0:23language model as a gigantic matrix of
11. 0:26numbers. When you give this model a
12. 0:28prompt, it predicts the next most likely
13. 0:32token. You can think of a token as a
14. 0:34short word. It predicts this next token
15. 0:37based on your input prompt and its vast
16. 0:40amount of training data. And the large
17. 0:42language model, it repeats this process
18. 0:46token by token until the entire output
19. 0:49answer has been generated. Basically,
20. 0:51it's a really big, really complicated
21. 0:54auto complete. And running these LLMs,
22. 0:57they consume a huge amount of resources.
23. 1:00Resources like compute, memory,
24. 1:03electricity, and water. It's a
25. 1:05staggering amount of resources,
26. 1:07especially when compared to the human
27. 1:09brain. We're talking about millions,
28. 1:12tens of millions of times more
29. 1:14resources. And look, a human brain, even
30. 1:17the brain of a child, right? It can
31. 1:20perform reasoning tasks and solve
32. 1:22problems that the most advanced frontier
33. 1:24large language model cannot if it
34. 1:27doesn't have the training data. And the
35. 1:29human brain, it has an impossibly tiny
36. 1:33amount of short-term memory.
37. 1:35Based on some studies out there, that
38. 1:38short-term memory, it could be anywhere
39. 1:40from tens of bytes to tens of kilobytes.
40. 1:44And the brain, it's able to perform all
41. 1:47of these reasoning and computational
42. 1:49tasks while running on the energy
43. 1:51equivalent of a light bulb. But here's
44. 1:53the thing, despite all the marketing
45. 1:56hype and all the investments being
46. 1:58poured into large language models,
47. 2:01AI and machine learning in general, it's
48. 2:04a much bigger field than just LLMs. And
49. 2:08that's why I've mentioned in more than a
50. 2:10few rants now, right? That if there are
51. 2:12any algorithmic advances out there that
52. 2:15could bring us closer to true
53. 2:17intelligence and true reasoning, then
54. 2:19it's likely that such breakthroughs
55. 2:22would come outside of today's large
56. 2:25language model architecture. Well, guys,
57. 2:28it looks like a research team from
58. 2:30Stanford University, they may have found
59. 2:32a novel model architecture. They
60. 2:35recently published a paper introducing
61. 2:37this new AI model architecture. It's
62. 2:40called next concept modeling or NCM. And
63. 2:44this new model architecture, it
64. 2:47potentially offers orders of magnitude
65. 2:50improvements in speed and resource
66. 2:53efficiency over traditional large
67. 2:56language models. Now, maybe this is a
68. 2:58genuine breakthrough. Maybe it could
69. 3:01eventually dethrone large language
70. 3:03models, or maybe not. But I think this
71. 3:06model architecture, it's interesting
72. 3:09enough for us to talk about it. So,
73. 3:11let's jump in. So, how does an NCM model
74. 3:14work? Well, remember what I just said
75. 3:17about large language models, right? How
76. 3:19it predicts a token token by token until
77. 3:22it gets the complete answer. Imagine the
78. 3:25answer is a 10-word sentence. So, that's
79. 3:27about 10 output tokens, right? Well, to
80. 3:31get that answer from a large language
81. 3:33model, the model itself has to be
82. 3:35evaluated 10 times or 10 passes. And
83. 3:39that's 10 passes sequentially. This is
84. 3:42because each subsequent pass, it has to
85. 3:46be fed as input all of the output tokens
86. 3:50that were generated from the previous
87. 3:52passes. And let's say you turn on
88. 3:55reasoning mode for this large language
89. 3:57model. Well, then all you have really
90. 3:59done is you have increased the total
91. 4:02amount of tokens that has to be
92. 4:03evaluated. Now, suppose you ask ChatGPT
93. 4:07a really hard question and it returns a
94. 4:09long answer, say a thousand token long
95. 4:12answer. To get this answer, this
96. 4:15multi-trillion parameter model has to be
97. 4:18evaluated a thousand times. And that
98. 4:21means keeping those trillions of
99. 4:23parameters, those trillions of numbers
100. 4:26in memory for a long time, right?
101. 4:29Because in each pass, the GPU needs to
102. 4:32read some or all of those parameters
103. 4:35from memory. So, this method of
104. 4:37sequential inferencing,
105. 4:40it's slow and it consumes a huge amount
106. 4:43of memory and energy resources. But with
107. 4:46NCM, they took a radically different
108. 4:48approach. They took the approach of
109. 4:51generating the entire answer in parallel
110. 4:54rather than sequentially. So, to do
111. 4:57this, the researchers took inspiration
112. 4:59from various image generation models.
113. 5:03These are diffusion models where you
114. 5:05start with a layer representing an image
115. 5:08where each successive layer has a
116. 5:11progressively higher resolution of that
117. 5:13image until the final image is
118. 5:15generated. So, with NCM, during
119. 5:18training, words are stored in these
120. 5:20collections of layers. At the topmost
121. 5:23layer, you have this sort of super rough
122. 5:26outline of a concept. They call these
123. 5:29concepts codes. And in each subsequent
124. 5:32layer, the concept, it gets
125. 5:35progressively refined. And only the
126. 5:38refinements get stored in that layer as
127. 5:41additional codes. As such, every
128. 5:44subsequent layer has more codes than the
129. 5:46preceding layer. This goes on until the
130. 5:49very last layer where every code
131. 5:52represents an output token. So, if you
132. 5:55combined all of these layers, you get
133. 5:58the original text back. So, now to get
134. 6:00an answer for a given prompt, the NCM
135. 6:04model is inferenced once per layer. In
136. 6:07this way, each layer represents a
137. 6:10parallel prediction of the entire output
138. 6:14at a given level of fidelity. The top
139. 6:16layers are coarse fidelity. If we were
140. 6:19to look at the words in those layers, it
141. 6:22would look like an AI slop word salad.
142. 6:25But, that fidelity, it progressively
143. 6:27increases from layer to layer, pass by
144. 6:31pass. So, by that final pass in the
145. 6:34bottom layer, you end up with output
146. 6:37text of the highest fidelity, which
147. 6:40would then be the answer to that prompt.
148. 6:42In this way, the model is inferenced the
149. 6:45same number of times as the number of
150. 6:48layers that the model contains. Now, for
151. 6:50this experiment, the research team
152. 6:53trained up several NCM models. I believe
153. 6:56the biggest model that they trained only
154. 6:58had 730 million parameters, meaning it's
155. 7:02a tiny model when compared to the
156. 7:04multi-trillion parameter large language
157. 7:07models that the big AI companies have
158. 7:09today. But, with that said, they did do
159. 7:12some benchmarking. They benchmark their
160. 7:15NCM models against other models of the
161. 7:18same size. And in doing so, they found
162. 7:21some real advantages to this model
163. 7:23architecture. When generating a 1024
164. 7:26token output, so that's like that 1,000
165. 7:30word example that I gave earlier, to get
166. 7:32an answer, the researchers had to do
167. 7:34just 32 passes of their NCM model. So,
168. 7:38that's 32 inferencing passes as compared
169. 7:42to the thousand plus passes that has to
170. 7:44be done to a traditional large language
171. 7:46model. So, at least in theory, NCM
172. 7:50models should be significantly faster to
173. 7:53run. So, they ran some benchmarks and
174. 7:56for summarizing text at least, they
175. 7:59found that their NCM model, it's 31
176. 8:02times faster than the leading diffusion
177. 8:06language model. And I want to call this
178. 8:08out specifically, right? Because this 31
179. 8:12times improvement, it's against other
180. 8:15diffusion models, other models that
181. 8:18generates text in parallel and not
182. 8:21against the large language models that
183. 8:23generate text sequentially. So, this is
184. 8:26not really a straightforward comparison
185. 8:28here. And if we continue with the same
186. 8:31thought experiment, suppose we had a
187. 8:34large language model that's 1 GB in size
188. 8:37and we had an NCM model of exactly the
189. 8:40same size with 32 layers. And we
190. 8:42prompted both models to get that
191. 8:45thousand word long answer. And for the
192. 8:47sake of simplicity, let's exclude
193. 8:50various optimizations like mixture of
194. 8:52experts and so on. If we do that, then
195. 8:55the large language model, it would
196. 8:57require more than a thousand passes to
197. 8:59get an answer. And while doing this
198. 9:02inferencing, the GPU would need to read
199. 9:05more than a thousand GB of data from
200. 9:08memory. But with the NCM model, we only
201. 9:11need to do 32 passes. The GPU needs to
202. 9:15just read 32 GB of memory. So, in terms
203. 9:19of memory bandwidth requirements, the
204. 9:22NCM architecture is more than 30 times
205. 9:26as efficient as a large language model
206. 9:28architecture for the same length of
207. 9:30output. Now, these researchers, they
208. 9:33also benchmarked the models reasoning
209. 9:35capabilities via games of Sudoku. They
210. 9:38had a 25 million parameter version of
211. 9:41their NCM model solve 100 Sudoku hard
212. 9:45boards. The results are then compared
213. 9:48against a 25 million parameter large
214. 9:50language model doing the same thing.
215. 9:52Now, here the NCM model, it solved 86
216. 9:57boards out of the 100, while the large
217. 9:59language model, it solved just one.
218. 10:02That's like an 8600%
219. 10:05improvement. So, by now, you're probably
220. 10:07asking, "Asian Dad Energy, all of these
221. 10:11model performance improvements are
222. 10:13incredible, but what's the catch?" As
223. 10:15with anything else in life, there's no
224. 10:17free lunch. When we look at that layered
225. 10:20NCM model architecture, you can see that
226. 10:23there are actually way more codes than
227. 10:26there are output tokens at the bottom
228. 10:28layer. In the experimental model that
229. 10:31they released on GitHub, there are like
230. 10:33four times as many codes. So, if
231. 10:36everything else stays the same,
232. 10:38evaluating those codes should take about
233. 10:42four times as much processing power. So,
234. 10:45it's like the NCM model architecture, it
235. 10:48drastically reduced the amount of memory
236. 10:51bandwidth needed, but it comes at the
237. 10:53cost of more processing power needed.
238. 10:56Essentially, it just shifted the
239. 10:58resource bottleneck, right? From memory
240. 11:01to compute. And here's the other thing.
241. 11:04The biggest model that they trained up
242. 11:06is only 730 million parameters, right?
243. 11:09So, the big question is whether all of
244. 11:11these gains that we have seen, gains in
245. 11:13speed, in memory efficiency, in
246. 11:16reasoning, and so on, will these gains
247. 11:19hold up when we scale NCM models up to
248. 11:22trillions of parameters, putting them in
249. 11:25the same league as frontier large
250. 11:27language models like Astra or Claude
251. 11:30Opus? That's an open question. So, net
252. 11:33net, NCM it's a super interesting
253. 11:36development in the AI space and it's
254. 11:39happening well outside of large language
255. 11:41models. Will it eventually displace the
256. 11:44large language model paradigm? I don't
257. 11:47really know. Maybe today's LLM driven AI
258. 11:50industry will maintain dominance. Maybe
259. 11:52it will disrupt our entire economic
260. 11:55system before it hits its theoretical
261. 11:57scaling ceiling. Or maybe the whole AI
262. 12:00financial bubble could burst first
263. 12:02because it's underlying business model
264. 12:04is a giant dumpster fire. Who knows? But
265. 12:07if there's one takeaway from this little
266. 12:10experiment, I think it's this. We
267. 12:12shouldn't assume that today's AI
268. 12:15limitations will last forever. To me,
269. 12:18new model architectures like this NCM
270. 12:21over here, it shows that left field
271. 12:23algorithmic breakthroughs could happen.
272. 12:26Breakthroughs that can unlock far more
273. 12:28powerful forms of AI. Maybe even getting
274. 12:32us to AGI someday. Breakthroughs that
275. 12:34can change the game entirely. And that's
276. 12:37all I got to say about that. Hope it
277. 12:39helps.
278. 12:40Anyways, if you have a morbid curiosity
279. 12:42to join me in this life journey, please
280. 12:44subscribe to my YouTube channel and
281. 12:46subscribe to my Substack newsletter. If
282. 12:48you want to support my vlog creation
283. 12:50efforts, please become a paid member of
284. 12:53the channel and paid member of my
285. 12:54Substack. If you want a one-on-one
286. 12:56coaching session with me, just schedule
287. 12:58it. Anyways, thanks for listening. Talk
288. 13:01soon. Bye.