Ist das das Ende von LLMs? Ein Stanford-Paper könnte alles ändern — Transcript
Full transcript
- 0:00What if I told you that a single paper
- 0:02from Stanford could make Nvidia's most
- 0:04expensive chips practically obsolete?
- 0:06Because here is the problem with the AI
- 0:07industry right now. We have hit a
- 0:09massive wall. Everyone is talking about
- 0:12compute limits, but the real silent
- 0:14killer of classic language models is
- 0:16memory bandwidth.
- 0:17Memory chips currently cannot be built
- 0:19fast enough to feed the massive AI
- 0:21models and keep them running.
- 0:23You can think of a standard language
- 0:25model as a huge matrix of numbers. You
- 0:28give the model a prompt and it predicts
- 0:30the next likely token. A token is
- 0:31essentially nothing more than a part of
- 0:33a word. Then this process repeats until
- 0:35the complete answer is finished. So you
- 0:37can basically think of it as a very
- 0:40sophisticated autocomplete system. This
- 0:42procedure costs an extreme amount of
- 0:43computing power.
- 0:44If you want to add 1,000 words to a
- 0:47prompt, meaning you want an answer that
- 0:49is 1,000 words or 1,000 tokens long,
- 0:52this process must run 1,000 times.
- 0:55Each time, everything has to be loaded
- 0:57into memory or read out of memory and
- 0:59sent through the entire model.
- 1:01With large models featuring trillions of
- 1:03parameters or more, this is extremely
- 1:05slow and also consumes an extreme amount
- 1:07of power. But in particular, it also
- 1:09requires an extreme amount of memory
- 1:11bandwidth. This means the speed of the
- 1:13memory also limits the speed of the AI.
- 1:17The processor often sits there waiting
- 1:19for data from memory so it can continue
- 1:21working.
- 1:22With larger models, this problem becomes
- 1:24even worse, or rather more intense. In
- 1:26the process, the model must remember all
- 1:29previous tokens and saves them in a
- 1:31temporary cache, the so-called KV-cache
- 1:34.
- 1:34The longer the conversation, the larger
- 1:37this cache becomes, and with models that
- 1:39have long chats and large contexts, this
- 1:41cache might take up the entire video
- 1:43memory.
- 1:45Hardware manufacturers are trying to
- 1:47solve this problem by simply putting
- 1:49more memory on the chips to provide more
- 1:51storage capacity.
- 1:53However, these chips are incredibly
- 1:54expensive and difficult to manufacture.
- 1:56Since memory is limited, there must be
- 1:58some solution to fix this.
- 2:00That is where a research team from
- 2:02Stanford comes into play, having
- 2:03developed a completely new approach.
- 2:05This new architecture is called Next
- 2:07Concept Modeling, or NCM for short, and
- 2:09the approach is truly radically
- 2:11different because instead of token by
- 2:13token, the entire answer is generated in
- 2:16parallel.
- 2:17To do this, the researchers looked at
- 2:19the concept from image generation, where
- 2:21the AI first creates a very, very rough
- 2:23, blurry image and then makes the image
- 2:26finer and more detailed with every
- 2:28iteration.
- 2:29NCM basically does the exact same thing
- 2:31with text.
- 2:32At the top level, a rough outline, a
- 2:35concept for the text, is stored, and
- 2:37this outline is then described as
- 2:39so-called discrete codes. But what
- 2:42exactly is such a code? You can imagine
- 2:44it as if the system has created a
- 2:46dictionary, and in this dictionary, all
- 2:47the references and codes are stored.
- 2:50It contains a fixed list of building
- 2:52blocks, the individual codes, and a code
- 2:54is simply a number from this dictionary
- 2:56. The exciting part is that, at the top
- 2:59level, very few codes describe the
- 3:01entire text. This means there is no room
- 3:03for details; rather, the entire text is
- 3:06represented by these few codes, focusing
- 3:08only on the concepts and the rough
- 3:10outline, much like the blurriness we see
- 3:13at the start of AI image generation.
- 3:15Accordingly, only what is general and
- 3:17important is saved.
- 3:19What is the text about, and how is its
- 3:20content structured?
- 3:22Every level below has more codes and
- 3:24represents additional knowledge about
- 3:27the text or extra information, until at
- 3:30the lowest level, there is a code for
- 3:32every word or token that describes it.
- 3:36I know the concept isn't easy to grasp,
- 3:38but it is truly the core, because you
- 3:40can only understand NCM if you
- 3:41understand this concept. Let me try to
- 3:43explain it using the example of an essay
- 3:44.
- 3:45When you want to write an essay, you
- 3:47don't just start writing blindly; you
- 3:50first think about the outline and the
- 3:52general content.
- 3:54Then you might continue by drafting the
- 3:56individual sentences or chapters, and
- 3:58only at the end do you start refining
- 4:01every single word. And that is exactly
- 4:03how this NCM concept, this NCM
- 4:05architecture, approaches text generation
- 4:07.
- 4:08From the rough outline to individual
- 4:10chapters and sentences, down to the
- 4:12actual individual words that are then
- 4:15perfected.
- 4:16When you ask an NCM model a question, it
- 4:19doesn't work by adding a single word
- 4:21each time; instead, the entire text is
- 4:24generated in every layer, but it becomes
- 4:26increasingly refined and detailed, so
- 4:29NCM runs through each layer once.
- 4:32Let's just assume you want to generate
- 4:34an answer of 1000 words. In a classic
- 4:36model, as I explained earlier, it would
- 4:39have to run 1000 times.
- 4:41In an NCM model, however, according to
- 4:43the paper, it only takes about eight to
- 4:46twelve passes, one for each layer.
- 4:48So, eight to twelve layers, and each one
- 4:50is processed once.
- 4:52If you compare this with other parallel
- 4:54methods, this NCM process is still about
- 4:5731 times faster than others, which is an
- 4:59extreme gain.
- 5:03While the total required computing power
- 5:05does not decrease, the memory bandwidth
- 5:07that must be used is reduced, and the
- 5:08focus is shifted more toward the
- 5:10individual calculation.
- 5:12But for this architecture to work at all
- 5:15, the researchers applied a few tricks.
- 5:18First, they trained their own translator
- 5:21and applied the first trick, so-called
- 5:23nested dropout.
- 5:25In about every second training step, the
- 5:28researchers cut off the lower, finer
- 5:30layers, so the text must be generated
- 5:32completely by the upper, general layers
- 5:34.
- 5:35This way, the layers learn to formulate
- 5:38the core idea of the text first, before
- 5:41really refining it and generating every
- 5:44word.
- 5:45Imagine you are on the phone and the
- 5:47connection is really bad. Then, the
- 5:49person tries to convey as much important
- 5:52content as possible in as few words as
- 5:54possible without getting lost in details
- 5:57. And that is exactly the same principle
- 5:59by which the upper layers are trained
- 6:01based on the concept.
- 6:03However, there is also a potential for
- 6:05error here, because if you rely on the
- 6:07upper levels for the rough concept, then
- 6:10an error is also passed on.
- 6:13If the upper level hallucinates a wrong
- 6:15concept and has misunderstood the text
- 6:17that should be written, then this error
- 6:19carries over to all layers below it,
- 6:21which leads to a potentially completely
- 6:23wrong text being written.
- 6:26To counteract this, the researchers
- 6:28developed so-called conditioning
- 6:30corruption.
- 6:31This is simply a process where the
- 6:33researchers, during training,
- 6:35intentionally input incorrect and flawed
- 6:37data sets into the middle layers so that
- 6:40it has to be corrected later, and this
- 6:42is how the model was effectively taught
- 6:44to correct errors that originated from
- 6:46above by itself later on.
- 6:48A kind of error correction directly
- 6:50within the model itself. A good
- 6:52comparison is perhaps a child learning
- 6:54to ride a bike.
- 6:55The child first learns to ride on a flat
- 6:58surface until they can do it roughly,
- 7:00and later, if the child can ride really
- 7:02well and you were to lightly bump the
- 7:04bike and push the child slightly off
- 7:06balance—I know, it might not be the best
- 7:08example—or they ride on an uneven
- 7:10surface, then the child has to correct
- 7:13and constantly adjust their posture,
- 7:15learning to ride more safely as a result
- 7:17. Through these two tricks, we now have
- 7:19a model that is really fast and corrects
- 7:22its own errors.
- 7:23However, that does not yet mean it can
- 7:25keep up with the processing power of
- 7:26large frontier models.
- 7:28This is where a third topic comes into
- 7:30play so that one can achieve similar
- 7:32computing power as with larger models
- 7:34using a significantly smaller model.
- 7:36This third trick is called Test Time
- 7:38Scaling and is based simply on taking
- 7:41advantage of the fact that NCM is
- 7:43incredibly fast. This allows us to
- 7:45generate multiple versions of an answer
- 7:48at once, using only the best one in the
- 7:51end.
- 7:52Instead of working with just one answer
- 7:55, we simply generate 20 answers in
- 7:57parallel, for example, which are then
- 7:59filtered out using a mathematical
- 8:01process.
- 8:03This way, only the best answer remains
- 8:04at the end.
- 8:05This concept can be compared to giving a
- 8:08programming task to 20 junior developers
- 8:11instead of one senior developer to work
- 8:13on it intensely, and then just picking
- 8:16the best result from them.
- 8:19In the paper from Stanford, this
- 8:21actually led to better results in
- 8:23translations and summaries compared to
- 8:25larger models.
- 8:26But there is another test the
- 8:27researchers conducted that I find even
- 8:29more exciting. They had the models play
- 8:31Sudoku.
- 8:33Normally, this is a real endurance test
- 8:35for models because one wrong entry means
- 8:37that a lot of work has to be redone or
- 8:40recalculated.
- 8:41Here, parallel processing is an extreme
- 8:44advantage. In this case, the researchers
- 8:47compared an MCM model with 25 million
- 8:50parameters to a standard language model
- 8:53also with 25 million parameters.
- 8:56The result was that of the difficult
- 8:59Sudoku puzzles, the normal model only
- 9:02solved about one in several attempts,
- 9:05while the MCM achieved an 86%success
- 9:08rate, which is a huge lead.
- 9:12To be fair, it must also be said that
- 9:15the researchers used other parallel
- 9:17models, diffusion models, which achieved
- 9:20about 80%.
- 9:22This means that the harder the puzzles
- 9:24became, the researchers found, the
- 9:26greater the gap grew; the MCM model
- 9:28performed better, but there are other
- 9:30parallel methods that could reach
- 9:32similar performance for these types of
- 9:34tasks.
- 9:35But why should hardware manufacturers
- 9:37even be interested in a model being good
- 9:40at playing Sudoku? Is this transferable
- 9:42to other areas? You could say in part,
- 9:45because MCM does not eliminate the
- 9:47bottleneck.
- 9:48However, it shifts the main bottleneck
- 9:50away from memory bandwidth, which is
- 9:52currently in very high demand and
- 9:54heavily taxed, toward pure computing
- 9:56power. It is just important to
- 9:58understand that nothing comes for free.
- 10:00In the end, the same amount of
- 10:02calculation must be done, only now the
- 10:04processor, which was previously waiting
- 10:06, can be used more and does not have to
- 10:08wait for the memory.
- 10:10That is an extreme advantage, but it
- 10:12naturally means there may be other
- 10:13bottlenecks that aren't apparent right
- 10:16now and will need to be solved later.
- 10:19Theoretically, this could make local AI
- 10:21on personal computers much faster, as
- 10:24memory bandwidth is often the biggest
- 10:26bottleneck there.
- 10:29For frontier models running in huge data
- 10:32centers, where all parameters must be
- 10:34loaded anyway, the advantage might be
- 10:37smaller than you'd expect, since NCM has
- 10:39so far only been tested on models with
- 10:42under a billion parameters.
- 10:44Nevertheless, I find the approach
- 10:46extremely exciting, and it makes
- 10:47token-by-token generation seem almost
- 10:49obsolete, like a relic, though that's
- 10:51not necessarily the conclusion to draw.
- 10:54Instead, it's important to think about
- 10:56how to solve current bottlenecks through
- 10:58new architectures, where NCM is a very
- 11:00interesting approach, because an
- 11:02algorithmic breakthrough like NCM can
- 11:04completely redefine the hardware
- 11:06requirements.
- 11:08The big question that needs to be
- 11:10answered now is whether NCM can be
- 11:12scaled to several trillion parameters to
- 11:14truly keep up with frontier models.
- 11:17Honestly, that question remains open.
- 11:19But it shows something else important:
- 11:21we shouldn't just accept the current
- 11:23hardware limits as given.
- 11:26Instead, we need to look at how to solve
- 11:28these hardware limits or even circumvent
- 11:30them through new algorithms and
- 11:31architectures.
- 11:33From my perspective, it would be a huge
- 11:35advantage not to have to wait forever
- 11:38for the AI and to get answers very
- 11:40quickly during everyday work.
- 11:43To summarize briefly, the biggest
- 11:45difference is that NCM no longer
- 11:46generates token by token, but actually
- 11:49everything simultaneously.
- 11:51In this process, nested dropout is
- 11:53applied to ensure that both broad
- 11:55concepts and detailed generation are
- 11:57possible.
- 11:58At the same time, errors are
- 12:00self-corrected, and it even ensures
- 12:02similar computing performance through
- 12:05the parallel generation of different
- 12:07answers.
- 12:08I have to say, I find this topic super
- 12:10exciting, especially what is happening
- 12:11with architectures. If you enjoyed this
- 12:13video, please leave a like and subscribe
- 12:14. I look forward to your take in the
- 12:16comments.
About this transcript
This page contains the full transcript of Ist das das Ende von LLMs? Ein Stanford-Paper könnte alles ändern by Tjark Harjes, generated from the public captions YouTube serves with the video. The transcript has 2,050 words across 346 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.