YT YouTube2Text

YouTube2Text → Transcripts → Dory - NVIDIA's Looped Transformer

Dory - NVIDIA's Looped Transformer — Transcript

by Sam Witteveen · 2,828 words · 387 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00Okay, so this is three models running
0:02the same prompt, but actually it's not
0:05three models. It's the same model but
0:08with different amounts of looping going
0:11on in the actual layers of the model. So
0:14this is a 4 billion parameter model from
0:16Nvidia. And you can see that going
0:18through this, we've basically got
0:20different token speeds for each of the
0:22different versions of the model that's
0:25actually running here. And each of these
0:27versions is looping a number of times.
0:29So this one is the basic one loop
0:31version, then we've got a two loop
0:33version, and then we've got a four loop
0:35version going on here. And and Nvidia's
0:37own composite score puts this model
0:40ahead of the Gemma 4 26B, but that's not
0:43actually the interesting part here. For
0:44me, the interesting part here is really
0:46about how it does it. The model is the
0:48same. The only thing that changes here
0:50is how many times the model runs one
0:53block of its layers for each token. Once
0:56for one loop, twice for two loops, four
0:59times for four loops. And if you look at
1:02each of these that's running the same
1:03prompt, we get very different final
1:06answers out, and we get different
1:08amounts of tokens that actually got us
1:10here. So sure enough, the fastest one
1:12here was the one loop version, and it
1:14looks like it did get to the right
1:16answer there. The two loop version just
1:18gave us a five and didn't even give us
1:21the breakdown in the answer. But the
1:23four loop version went into a detailed
1:25analysis of breaking down what it was.
1:29You can clearly see it's got nice LaTeX
1:31there, and it's coming to the right
1:33answer. And it's giving us a full
1:35explanation of how it got to the right
1:37answer. Okay, so this is Dory, a looped
1:40transformer, also called a latent
1:42recurrent transformer. So this is a 4B
1:45model that comes from Nvidia's NeMo-Tron
1:47labs. It came out a couple of weeks ago,
1:49and while this is mostly a research
1:51release, it really does show where loop
1:53transformers could actually be going. So
1:56I've been wanting to do a video on loop
1:57transformers for a while, and this is a
1:59good excuse because it's a loop model
2:01that you can actually download and run
2:03on your own hardware. Unfortunately,
2:05most of what people are putting out
2:06there about this is just blog posts or
2:09walkthroughs and not actually showing
2:11you the results of the model. So,
2:12unfortunately, there's no blog post for
2:14this. The model card is basically all
2:16you get. So, I decided to actually
2:18install this, have a play with it, and
2:20build a UI for actually showing you sort
2:22of what this actually does. So, the
2:24basics of this, it's a 3.97 billion
2:26parameter model. It's got up to 256K of
2:29context. And NVIDIA has actually
2:31released this on their Open MDW license.
2:35So, you can use it commercially if you
2:36want to. They've also released a BF16
2:39version, an MXFP8 version, and an NVFP4
2:43version along with other GGUFs, etc. So,
2:47the model itself was pre-trained on 6
2:49trillion tokens. And interestingly, the
2:51first 4.5 trillion of those tokens were
2:53done natively in NVFP4. They then
2:56switched to 16-bit, gave it 300 billion
2:59tokens of context extension, and finally
3:02did SFT and reinforcement learning with
3:04GRPO. All right, so how does the loop
3:06actually work? So, a normal transformer
3:08runs every layer once from the input
3:11right through to the output. What Dory
3:13does is it splits its layers into three
3:16separate blocks. There's an input block
3:18at the start and an output block at the
3:20end, and each of those only runs once.
3:23The middle bit is where we've got the
3:25recurrent block. And that's the part
3:27that loops. If we sort of look at how
3:29those layers are actually spread out,
3:31you've got 16 for the input block, got
3:3316 for the output, and then you've got
3:35the recurrent block, which is actually
3:3740 layers. So, if you look at the
3:38animation here, what you're seeing here
3:40is a token going round that recurrent
3:43block. And we can set it to be once
3:46round, twice round, four times round,
3:49eight times round. And you can think of
3:51it as those representations as it goes
3:53through the recurrent part are just
3:56refining those representations to get
3:59better and better. Now, so it doesn't go
4:02totally off on its loops, you can see
4:05that it's actually sort of re-adding
4:07what came out of the input block to it
4:10each time. So, in many ways, it's very
4:12much like the ResNet models where you've
4:14got the original input being added to a
4:18modification of it. The difference here
4:20is just that that modification, rather
4:23than being different sets of weights
4:24each time for each block, it's basically
4:27the same set of weights that are just
4:29getting re-enforced here. So, you can
4:31kind of think of this as every lap that
4:33it goes through, the model gets reminded
4:35of what the token is actually working
4:37on. So, in some ways, it's kind of like
4:38re-reading the question or re-reading
4:40the input, basically, before you have
4:43another shot at getting the best output.
4:46Now, if we were to unroll it like this,
4:48you can see that we're getting a much
4:50deeper model here. Even though the
4:52weights for that repeated block are the
4:55same each time, compute-wise, we've got
4:57a lot more depth going on there. The
4:59other thing that's interesting here, and
5:00that makes it perhaps more practical, is
5:02that the output block is trained to read
5:05the state after any of the loops. So,
5:08one loop, two loops, four loops are all
5:11valid models for this. You don't have to
5:13retrain it for any particular number of
5:17loops. You can basically just set that
5:18at inference time, and you're making use
5:20of eight times through these shared
5:22weights. So, this whole idea is what
5:24Nvidia calls this elastic inference
5:27compute. And as I turn up these loops
5:30here, the compute per token is going up,
5:33but the weights in memory don't change
5:35at all. In this case, it's still only
5:373.9 billion parameters sitting on the
5:40GPU. So, already you should start to see
5:43that one of the huge advantages of this
5:46is that if we can just loop through the
5:48same weights multiple times, we don't
5:51need to have GPUs with huge amounts of
5:53memory to be able to run a model like
5:55this. I can see here at one loop, we've
5:58basically got 72 layers of effective
6:01depth. At two loops, we're up to 112. If
6:04we jump to four loops, we're already at
6:06192 layers there. But still, it's the
6:10same amount of GPU RAM being used for
6:13this. And you can see looking at the
6:15four loop version here, it is actually
6:17doing roughly the compute of a 9-bit
6:19dense model, even though the memory
6:22footprint of the weights in here is the
6:24size of a 4-bit model. Now, it's not a
6:26total win for your memory, etc.
6:29Obviously, one place where the memory
6:30will grow is the KV cache. In per loop
6:33mode, every loop actually keeps its own
6:35cache. So, more loops here means that
6:38you will need a bigger KV cache and
6:41memory to actually support that in here.
6:43Now, you can actually run it in a sort
6:45of last loop mode, where it each
6:47recurrent attention layer keeps just one
6:49cache and then overwrites the old ones.
6:52And you can see if we go from one loop
6:54to four loops there, that last loop
6:57doesn't change at all. So, that is
6:59something that you can actually try out
7:01to see how it goes. Now, NVIDIA says
7:03that the accuracy and output length stay
7:06close to per loop. So, to give you a
7:08sense of it, at a 32 context window and
7:11two loops, you're looking at roughly 2
7:14gig per sequence versus just one loop or
7:17the last loop out, and that it's only
7:19going to require a 1.32. So, you could
7:22imagine that this is where you set it up
7:24so that if you're serving long context
7:26with lots of concurrent users, then this
7:29is a setting you certainly would look at
7:31trying out to just use the last loop as
7:34opposed to the per loop KV cache. Okay,
7:36so you can see when I actually run this,
7:38I built a little demo in here that
7:40allows us to run these once. I'm
7:43actually running multiple versions of
7:46the model. So, just to be a sort of
7:48fully open with you here is one of the
7:50challenges with this at the moment is
7:52that when you start v l m, you have to
7:55tell it how many loops you want it to
7:57do. So, I'm actually running three
7:59versions of the actual same model so
8:03that I can do this. Anyway, you can see
8:05that the prompt gets sent to all of
8:07these. Now, it's not a hard and fast
8:10rule that one of them will have more
8:12output tokens than the other. Certainly,
8:14the speed will be faster on the one loop
8:17versus the two loops versus four loops,
8:20etc. I think depending on the actual
8:23problem, you'll see the quality of the
8:26output change as well. Often, I see that
8:29the four loop one does seem to get
8:31better quality outputs than the one loop
8:34one, for example. But, certainly not all
8:36the time, and it's not something that I
8:37would just sort of say it's the same as
8:39sort of turning up reasoning or
8:40something like that. You'll also find
8:42for a lot of prompts that they actually
8:45get the same quality of outputs going
8:48through this. In this case here, you can
8:50see that basically the four one actually
8:53generated the most amount of tokens just
8:56by a fraction over the one loop one. And
8:59obviously, the wall time for that was
9:01actually a lot longer. Obviously, one
9:03loop is going to decode faster, etc. We
9:05can also come over to a chat and see
9:08what the actual chat is like when we're
9:10looking at different outputs. So, there
9:14does seem to be some difference in the
9:16reasoning, but I wouldn't say there's a
9:18huge difference in the sense that you're
9:20going to get much better all the time
9:22from four as per three. If I run the
9:25same thing with four loops, you will
9:27notice you get different kinds of
9:30responses out. Looking at this one,
9:32actually it got to an answer much
9:34quicker. And we didn't get the numbered
9:36thinking in the actual reasoning here.
9:38We just got sort of more general
9:40thinking as opposed to the two loops,
9:42which gave us numbered thinking, etc.
9:44Okay, so this is the part that for me I
9:47was most curious about was I wanted to
9:50see what the model's actually doing on
9:52each of the laps of this. So, to run
9:55this, I basically ported this to sort of
9:56plain PyTorch and just used the BFloat16
9:59weights in here. And then I can read out
10:01every internal state of the loop. So,
10:04this is what I'm calling sort of like a
10:05loop lens in here. And then it allows it
10:08to match feel em's log probabilities and
10:10to see how those outputs actually
10:13change. So, you can see that if this is
10:15the sort of rough outputs for one loop,
10:17we can actually see that there are
10:19certain things that paid attention to
10:21more at six loops versus one loop. And
10:24you can see for example that the model
10:26gets more confident about predicting
10:28certain tokens after more loops. So, at
10:32one loop here it's predicting this
10:34bracket only at 15%after two it's at 68,
10:38three 74, four 75, etc. right up to sort
10:41of five and six loops here seems to be
10:43the same. So, it's going to be something
10:45to see as these models get built for
10:47production and not just for research.
10:50How do we actually know for what kind of
10:52questions do we want to actually do
10:55this? I talked about that at the moment
10:57with VLM I need to run a different
10:59instance of VLM at each point. And it is
11:02kind of cool that you can imagine
11:03putting a router in front of that so
11:05that if you do have harder questions it
11:07routes to the six loop version as
11:10opposed to the one loop version. We can
11:12also imagine in the future that VLM just
11:14does that itself. All right, so going
11:16back into the benchmarks against Gemma
11:18426B, Dory at one loop wins one out of
11:2210 of these benchmarks. You can see at
11:25two loops that's already now jumped up
11:28to four out of 10 here. And sure enough
11:31if we go to four loops, it's now winning
11:34five out of 10 here. Now, interestingly
11:37where this seems to win is at agentic
11:39work. So, instruction following and long
11:42context, things like SweeBench,
11:44verified. The ones where it tends to
11:46lose are things that you would kind of
11:49expect that it's not just about
11:51reasoning, it's about knowledge. So,
11:54knowledge obviously requires a lot more
11:56weights to store the knowledge. And sure
11:58enough on things like GBQA, humanities
12:01last exam, etc. This doesn't win. It
12:04just doesn't win at all. It actually
12:05does quite a lot worse. And it kind of
12:07makes sense that that's probably because
12:10the model just doesn't have enough
12:12weights uh compared to the 26B version.
12:16So, remember the 26B version still only
12:18got a small amount active, but it's got
12:20all those extra experts that are storing
12:23those weights that can be turned on when
12:26it needs to basically get the results
12:27out there. Just to remind you that Gemma
12:2926B is actually a mixture of experts
12:32model with 4 billion active parameters
12:34per token. So, the compute is roughly
12:37sort of the same for these. It's just
12:40how they're doing that compute and what
12:41weights are actually being used is
12:43actually quite different. And you can
12:45see if we come down here and look at the
12:47amount of loops versus the quality that
12:50we're getting here, certainly more loops
12:53seems to score higher with the same
12:54amount of weights. And then don't forget
12:56we're just looking at a 4B model here.
13:00You could imagine this just doubling or
13:02tripling in size and then being able to
13:04compete with mixture of experts models
13:07that are sort of 50 billion or greater.
13:09So, anyway, just to finish it up, I
13:10think this is a really interesting
13:12direction that people are going in. Loop
13:15transformers is definitely a thing.
13:17We've had many rumors that that's what
13:18GPT-6 is using. Supposedly sort of one
13:21versus two versus three loops depending
13:23on the different sizes, etc. I would say
13:26if you're going to deploy this today,
13:27probably you want to run two loops as
13:30the default and I think that's what
13:31Nvidia is recommending anyway. And then
13:34maybe you have another version of it
13:35that's running actually at sort of four
13:37loops where you send harder agentic
13:40tasks or things that do need that extra
13:42thinking in there. I think the really
13:44big idea here is that this is a
13:46different way to scale. Instead of the
13:48model just writing longer and longer
13:50chains of thought, it thinks harder per
13:52token rather than more tokens. And the
13:55cool thing with that is that you get to
13:57choose how hard when you actually deploy
14:01it. So, the model's certainly not
14:02perfect. I don't think this sort of
14:04research version is going to replace
14:06anything that people are doing in
14:08production that much. But I do think as
14:10we sort of look at how to sort of
14:12fine-tune these kind of models for the
14:15different numbers of loops, etc. You
14:18could imagine suddenly being able to do
14:19a lot of different tasks with a similar
14:22kind of size model that doesn't need a
14:24huge memory footprint like a large MOE,
14:27etc. Anyway, I put the links to the
14:30model card and all the details about
14:32that in the description. If you are
14:34interested in me doing a broader video
14:35about Loop Transformers, let me know.
14:37I'll sort of start looking at some of
14:39the other ones that are being proposed
14:41and maybe even look at some ideas of how
14:43to train a small one, etc. And let me
14:45know in the comments where you think
14:46this is going. If it is true that GPT-6
14:50is one of these Loop Transformers, we're
14:52going to expect that the open models are
14:54going to catch up pretty quickly to
14:56something like that. Anyway, as always,
14:58if you found the video useful, please
15:00click like and subscribe and I will talk
15:02to you in the next video. Bye for now.

About this transcript

This page contains the full transcript of Dory - NVIDIA's Looped Transformer by Sam Witteveen, generated from the public captions YouTube serves with the video. The transcript has 2,828 words across 387 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.