YT YouTube2Text

YouTube2Text → Transcripts → Flint - The First RSI Coding Model?

Flint - The First RSI Coding Model? — Transcript

by Sam Witteveen · 3,628 words · 513 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00Okay, so on the channel, I've covered
0:02the Qwen 3.8 27B model quite a lot.
0:06We've looked at variations of it, and I
0:08got to say it's still one of my favorite
0:10models. The problem though for a lot of
0:12people with that model is just speed.
0:15Because it's a dense model, when you're
0:16decoding, you've got to go through all
0:18of the layers. Now, on a MacBook Pro
0:20with an M4, it does about 13 tokens a
0:23second. And I guess that's fine for
0:25chat, but it's kind of painful for
0:27anything that you want to do with agents
0:30where you've got dozens of calls, you've
0:31got a loop going, etc. So, in this
0:33video, I want to talk about a new model
0:35that's trying to take on these mid-level
0:37coding models, but do it from the point
0:40of an MLE rather than a dense model. So,
0:43this is Flint. Running on that same
0:45MacBook that I was talking about before,
0:47it can hit 130 tokens a second. It's
0:50roughly 10 times faster than the Qwen
0:52dense model. And on some of the coding
0:54benchmarks, it's getting really close to
0:56the same kind of performance. On others
0:59though, it's kind of well behind. So,
1:01we'll get into that. But I got to say
1:02before we even do that, that's not even
1:04the most interesting part here. The
1:06company that made this, Recursive, say
1:09that they built the whole model from
1:11scratch in under 4 months with basically
1:14a very small machine and a set of
1:16agents. And they did it with their
1:18automated research system, which they
1:20call the Eureka machine, which helped
1:22design the architecture, the
1:24quantization, and even the GPU kernels.
1:27So, let's dig into whether Flint is
1:29actually any good. And also, what do
1:30they mean when they actually say an AI
1:32helped build this? Okay, so what is
1:35Flint? It's a mixture of experts model
1:37with 40 billion parameters in total, but
1:40only about 2 and 1/2 billion active for
1:43each token. There are 256 experts, it
1:47picks six of them per token, and then
1:49there's one shared expert that's always
1:52on. And this is the thing that really
1:53matters for running something like this
1:55locally on a laptop or on a desktop box.
1:58Your speed is mostly limited by your
2:01memory bandwidth. So, how many weights
2:03you have to read for every token? The
2:06active count here is really what
2:08predicts the speed. The total count or
2:10the number of parameters in total is
2:13what decides whether it's going to fit
2:14at all. So, I guess the two questions
2:17really could think about here is one, is
2:192.5 billion parameters too small to
2:23basically have for your active
2:24parameters? And then also, is 40 billion
2:27parameters too big for many people to be
2:29able to fit this on their GPU? There are
2:31some other interesting things about the
2:33architecture of this model as well. So,
2:35most of the layers, 35 of them, are
2:38using what's called a sliding window
2:40attention of about 500 tokens. And then
2:43only 14 of the total layers actually use
2:46a full attention system. So, that keeps
2:49long context a lot cheaper, and they're
2:51documenting that out of the box this can
2:53do 262K
2:55of context with up to 65K output tokens.
2:59Another interesting thing here is
3:00reasoning is always on. Now, one of the
3:03cool things here is that this is a fully
3:05Apache 2 open license. It's not gated.
3:07They've released four official repos on
3:10Hugging Face. You've got a BFloat16
3:12version. You've got an Inf4 version.
3:15They've got a standard GGUF quantized
3:17version. And they've got an NVFP4 1,
3:20which is what I'm going to mostly show
3:21in here. Now, if we just go for a quick
3:23look at the benchmarks against other
3:25mixture of experts models in its class,
3:26things like the Qwen 3.6 35B, which is a
3:31model that a lot of people wanted to see
3:33updated for the 3.8 and it just never
3:35came. That's the mixture of experts
3:38model that's sort of closest to the 27B.
3:41And then other open MOEs like Laguna XS.
3:45Flint's leading pretty much on
3:46everything. And it's basically tied with
3:49Qwen 3.6 on live code bench. On
3:52instruction following, it seems to be
3:54beating out all of its rivals there.
3:56When we compare it straight up against
3:58the Qwen 3.8-27B,
4:00it does lose, right? On terminal bench,
4:03it's 55 versus 73, and it's behind both
4:07on SweBench numbers and the live
4:09codebench. So, it's not that this is the
4:11best coding model out there. It's that
4:14this is a model that gets really close
4:16to the dense model performance, but is
4:19an MWE and can run much, much faster.
4:22And obviously, this chart makes a better
4:24case than any of the benchmarks here.
4:26You can see sort of higher scores up
4:28here, more speed out here. And looking
4:30at this, Flint is sitting out all on its
4:32own over there. All right, so if you
4:34want to run this, you've got a whole
4:35bunch of different ways you can do it.
4:37It's already out in a llama. It's also
4:39in LM Studio and llama.cpp. And there's
4:43vLLM and SGang support if you want to
4:45serve it properly. Recursive themselves
4:48have made their own engine called Rec.
4:50And that's where the 130 tokens a second
4:52is actually coming from. But it seems at
4:54the moment that will only install on a
4:56Mac, right? And a pretty powerful Mac at
4:59that. Unfortunately, I don't have one of
5:00those. So, the Mac footage that you've
5:02been seeing is actually from Recursive's
5:03own launch video. I'm going to mostly
5:05show the NVFP4 version running on an RTX
5:09Pro 6000. Now, if you want to try and
5:11run this yourself, Recursive's own blog
5:13says that Flint may actually need more
5:15guidance than bigger closed coding
5:17models. So, I guess the goal there is
5:20when you've only got two and a half
5:21billion active parameters, break bigger
5:24jobs into smaller pieces with explicit
5:27instructions. And another key thing is
5:29the harness. So, their docs recommend
5:31pie over open code. There are further
5:34reports of pie being about one and a
5:36half times faster with workflows. And
5:38I've got to say we're really getting
5:39into the point now where testing a model
5:43comes down to as much about the harness
5:45that you're using as just the model
5:47itself. All right, so let's talk a
5:49little bit about who's behind it. And
5:51why would a brand new lab ship a small
5:53local coding model as its very first
5:55product? All right, so the main public
5:57face of Recursive is Richard Socher. And
6:00if you've worked in NLP at all, you
6:03probably know who he is. He used to
6:05basically teach a lot of the core
6:07courses at Stanford sort of 10 years
6:09ago. He used to be the head of AI at
6:11Salesforce after they bought his
6:13startup. And for the past few years,
6:15he's been working on a startup called
6:16you.com to try and take on Google at
6:19search. Richard has always been one of
6:21the people in deep learning who's had
6:23really interesting ideas. He was one of
6:25the people on the original ImageNet
6:26paper. He also did things like glove
6:29embeddings. And the interesting thing is
6:31that even if you go back to his 2014
6:34Stanford PhD thesis, it was titled
6:38Recursive Deep Learning for NLP and
6:40computer vision. So he's had a lot of
6:42the ideas around recursive self-
6:45improvement and this whole idea of sort
6:47of continual learning, getting models to
6:49be better, etc. Just recently, he
6:51basically published a book with a lot of
6:53these ideas called The Eureka Machine.
6:55In fact, I think it just came out in
6:57September. It's something that I've been
6:59reading over the past few weeks. And
7:00like I said before, many things with
7:02Richard Socher, he has lots of good
7:04interesting ideas. Now, he put together
7:06a team of eight co-founders from a whole
7:09bunch of different places like Google,
7:11OpenAI, DeepMind, Meta, etc. And a lot
7:15of seriously impressive people in there
7:17who've been working on a lot of original
7:19ideas for quite a while, not just on
7:22LLMs. So you've got Tim Rockstål in
7:25there who came from Google DeepMind,
7:27where he was leading the Open Ended
7:29team, which was the team behind Genie,
7:31which you've probably seen with Google's
7:32big world model. You give it a single
7:34image and even a rough sketch, and they
7:37can then turn it into a game. It can do
7:39a whole bunch of different things with
7:41with it. And in fact, the entire team
7:43here is really well pedigreed, right?
7:45They've all been working on really
7:47interesting ideas. And I think that kind
7:49of shows when in May they came out of
7:52stealth having raised $650 million
7:55at a $4.65 billion valuation. The team
7:59is still only about 30 people or less.
8:01And you got to think here that investors
8:03really weren't sort of buying into any
8:05particular product, they were buying
8:07into a team and a system for how to
8:11basically make these models just improve
8:13themselves and be able to build that
8:16recursive self-improvement loop. And
8:18that's what brings us to the Eureka
8:19machine. So, the name gets used for two
8:21things. One being the big sort of vision
8:24from Sacha's book, and then two, the
8:26actual system running inside Recursive.
8:29So, let's focus on the real system. So,
8:31it's a loop. An agent proposes an idea
8:34to try. It writes the code, it
8:36challenges itself, it runs a bunch of
8:38experiments, it trains something or
8:40benchmarks something. And then winners
8:42become sort of stepping stones for the
8:44next round. And the interesting thing is
8:46that it runs lots of these ideas in
8:49parallel, right? So, there's lots of
8:51these research paths going on in
8:52parallel. And then it combines just the
8:55most promising ones. And to be honest,
8:57this is kind of familiar territory to
8:59anyone that's sort of looked at the
9:00open-endedness research. It's a very
9:03similar sort of line of work, but scaled
9:05up with a lot more money, a lot more
9:07compute. And the thing that they talk
9:09about, which I think really does clarify
9:10a lot of things around this, is the hard
9:13part wasn't coming up with the ideas, it
9:16was coming up with the judge. And the
9:18challenge is as Search got stronger, the
9:21candidates for judging sort of end up
9:23starting to be gaming the evaluations.
9:26So, things like cashing outputs,
9:28exploiting timing. And the only way you
9:30can sort of do well out of that is to
9:32then keep getting your evaluator to be
9:35getting stronger as well. Very similar
9:37to things like GANs, if you remember the
9:39old days of GANs, where you had a
9:41discriminator and a generator. So, you
9:43himself has said that the team have
9:45found lots of bugs in their own test
9:48harness when they were building this.
9:49And I think there are lessons here that
9:51apply to anyone running agent loops or
9:53doing RL. Your evaluator or your
9:56verifier really is kind of like an
9:58attack surface. And the better that your
10:00optimization gets, the more holes it
10:03will actually find in those kind of
10:05rewards. So, you can kind of think of
10:07this is like a really clever intern who
10:09find a way to hit whatever metric you
10:11set, whether or not that's actually
10:13going to lead to sort of the final goal.
10:15Now, back in June, they were playing
10:16with this system on things like nanoGPT
10:19or nanoChat, basically trying to get the
10:21loss down there. But, Flint is really
10:24their first foray into actually making
10:27their own model, which is a decent size
10:29and people can actually use, etc. One of
10:31the criticisms leveled here is that we
10:33still don't know what's actually running
10:35it. Which models, how many agents, how
10:38much compute, etc. Are they, for
10:40example, just using a frontier model
10:42from one of the big companies to
10:43optimize their own models? All right.
10:46So, when we actually look specifically
10:47at Flint, Recursive says the system here
10:50sort of consisted of three main parts.
10:52So, the first is architecture search.
10:55And it started with a model with 35
10:57billion total, 3 billion active.
10:59Basically, something very similar to the
11:01Quen 3.6B model. They then explored
11:04different expert setups and widths. And
11:06landed on 40 billion total with 2 and
11:081/2 billion active. So, the fewer active
11:11parameters basically gives us the speed
11:13up here. And I will mention something
11:15that I've mentioned in other videos. It
11:17does seem to be kind of coincidence that
11:20the right size for your active
11:23parameters seems to be roughly around
11:255-6%
11:27of what your total parameters are. And
11:29while that may change at really big
11:31sizes, Emily models that reviewed that
11:33have tended to do really well tend to
11:35stick to that formula and ones where
11:38they go up to 10% active really don't
11:39seem to do that well. The second thing
11:41that they did, which was really kind of
11:43interesting, was the output layer. So,
11:45the output layer obviously is going to
11:47use a lot of parameters because you're
11:49going to be making a prediction where
11:51you softmax over all the possible
11:53tokens. And interestingly here, Flint
11:55has gone for a tokenizer of about
11:57150,000 tokens. And just that output
12:00layer alone is around sort of 300
12:02million of those parameters. We're going
12:05to have that to be roughly an eighth of
12:06everything that you're going to put each
12:08token through. Now, most models won't
12:10quantize that, but here they did
12:12quantize that to four bits. And they
12:15actually recovered the accuracy by
12:16distilling from a 16-bit version of the
12:19model. This is where they're
12:20implementing that quantization-aware
12:22training. This is actually could be one
12:25of the key things that gets their NVFP4
12:28version to score so close to the full
12:31precision model, which is kind of
12:33amazing there. The third thing that they
12:35did was they actually sort of
12:36incorporated the whole idea of kernels
12:38in here. So, on Apple silicon, their
12:40system actually wrote a low-level metal
12:43mega kernel, which sort of fuses the
12:45whole decode path into one kernel. And
12:48that just speeds up everything because
12:50there are fewer kernel launches, fewer
12:51trips to memory, etc. Now, it's really
12:54interesting that they're claiming that
12:55their Eureka machine system actually
12:57sort of found these things and did
13:00Unfortunately, there is no published
13:02paper for this at the moment, so really
13:05we don't have a lot to go on, but it
13:06would be really interesting to see that.
13:08Let's jump into the demo and have a look
13:09at how the model itself actually
13:11performs. Okay, so you can see when I
13:13come in here, I'm basically running the
13:15NVFP4 one. I've run a bunch of the
13:18benchmarks on this myself. I definitely
13:20see some differences compared to their
13:24official ones, both positive and
13:26negative, but that also does sort of
13:28show how much these things can actually
13:31sort of change as you run them. If we
13:33come in here and just give it some sort
13:35of standard test, you'll see that we're
13:38getting super fast tokens out. That's
13:40because I'm running the NVFE4 version
13:42here, right? So, you can see also just
13:44looking at the thinking in here.
13:47I'll let that just run. If we look at
13:49the thinking in here, it's quite verbose
13:51in the thinking. It has gone through in
13:54this case, it's basically just sort of
13:56done an evaluation out. It hasn't
13:58actually done any sort of what I would
14:00call numbered thinking in here. We go
14:02through, it ran the task, and it passed.
14:05If we give it something a little bit
14:06harder, we'll see for a number of these
14:08that we're not getting what I would call
14:09numbered thinking in here. If we look at
14:11the output, this is basically generating
14:14pretty quickly. I'm generally seeing
14:16sort of like around the 200 tokens per
14:17second. If I give it something that's
14:19more sort of free-form like this, let's
14:22see what the output that we're getting
14:25is. And you can see here, now this time
14:27I've basically given it a system prompt
14:29as well. We're getting quite a bit of
14:31thinking going on here, and the thinking
14:33can vary quite a bit on this model. I've
14:35seen with these kind of things,
14:36sometimes the thinking is just sort of a
14:38few hundred tokens, sometimes it can be
14:40in the thousands. Now, you'll notice the
14:42other thing here is we're getting the
14:43numbered thinking.
14:47So, you can see here we've got the
14:48numbered thinking going on. We've got
14:50like deconstruct the request, initial
14:52brainstorm, structuring the essay,
14:55drafting the content. Really sort of a
14:58stock standard this. Certainly not like
15:01the neuralese that we've seen from some
15:03of the more modern models where they
15:05tend to sort of abbreviate the thinking
15:07down. But here you can see that we've
15:09generated almost 6,000 tokens. We're
15:11getting a bit over 210 tokens per
15:14second, which is kind of around the
15:16average. I would say I'm sort of seeing
15:17around anything sort of 200, 212, that's
15:21quite common here. Now, you can see that
15:23there wasn't a lot of pre-fill in this
15:25one. So, what I will do is I'll
15:26basically give it the same essay back
15:29and then ask it to basically grade it.
15:31So, we've actually got quite a lot of
15:32pre-fill going on. Okay, so I'm asking
15:34it, "Please look at this essay and
15:36measure and check it for accuracy." This
15:37is exactly the text that it wrote
15:39before. We can see coming out of it,
15:41we've got a fair bit of thinking in
15:43there, just going through that. And then
15:45we've got the overall assessment. It's
15:47interesting that it only scores itself
15:49around 85% accurate. In this case, if we
15:52go through, it talks about different
15:54things that were that it sort of wrote
15:57in the first one that are incorrect. It
15:59then goes through and gives some
16:00revisions for it. You see this one we
16:02have pre-fill was over 2,000 tokens. And
16:05the token speed there for the pre-fill
16:06was super fast. And you can see the
16:09generation speed is almost exactly what
16:11we had before. And when we ask it to
16:13rewrite it, it's basically then going
16:16through and doing that. If we look at
16:17sort of agent tasks, you'll see that it
16:21seems to be very good at using tools.
16:23From my tests, there's no issues around
16:25things like that. It's able to go
16:28through different steps, run all those
16:30things quite well. And the main
16:32advantage of this is just that it's a
16:33lot faster, especially if you don't have
16:36a very fast GPU. Now, the version I'm
16:38using here is definitely not as
16:40optimized as the one that they made the
16:42mega kernel for on the Mac. That seems
16:45to be a very unique thing that they've
16:47done. It would have been a nice thing to
16:49see them take that Eureka machine and
16:51then point it at the CUDA kernels and
16:54actually improve those as well. But I
16:56could imagine that that's actually
16:57something that's reasonably quite
16:59valuable. They perhaps don't want to
17:00give away the CUDA optimizations, just
17:03like we've seen with Thinking Cap, where
17:04a lot of these companies are giving away
17:06one version, but then charging
17:08enterprises for other versions, etc. If
17:10you're looking to see where this will
17:11fit on your GPU and perhaps what you
17:14could actually do with this, I put this
17:16together basically just showing
17:17different GPUs and will you be able to
17:19run this or not? Even on the RTX Pro
17:22with 96 GB, I still can't run the full B
17:25float 16 version in there. You're going
17:27to probably need like an H200 or you're
17:29going to need multiple GPUs to do this.
17:32You can run the NVFP version though
17:34quite easily. Not to mention a bunch of
17:36the unsloath quants and other quants
17:38that are out there. They can run on much
17:40smaller GPUs. All right, so where does
17:42that leave us? Well, if you've got a
17:44pretty powerful Mac Pro with 48 gigs of
17:46memory and you want a fast local coding
17:49model, I think Flint is really well
17:51worth trying. If you're on something
17:53like a DGX Spark or a bigger GPU, you'll
17:55still get a fast mixture of experts
17:58coder. Probably not going to get that
17:59130 tokens per second headline there.
18:02Now, for the Eureka machine itself,
18:04we're going to have to see do they scale
18:06up to try and do a frontier level model.
18:08I'd love to see them take on a dense
18:10model like the 27B and actually beat
18:13that. And then don't forget we've got
18:15Quinn 4 around the corner with a whole
18:17bunch of N grams ideas and stuff like
18:19that which they took from Deep Seek. So,
18:21I do think we're going to see a lot more
18:22ideas of this recursive self-improvement
18:25stuff starting to take off. Anyway, let
18:27me know in the comments what you think.
18:29If you've tried out Flint, how was it
18:31how has it held up for you? I know many
18:33of you are really interested in the MOEs
18:36rather than a dense model just because
18:38it's easier to run. But, it still does
18:40seem that you take quite a hit when you
18:42do that as opposed to going for a fully
18:45dense model. Anyway, as always, if you
18:47found the video useful, please click
18:48like and subscribe and I will talk to
18:49you in the next video. Bye for now.

About this transcript

This page contains the full transcript of Flint - The First RSI Coding Model? by Sam Witteveen, generated from the public captions YouTube serves with the video. The transcript has 3,628 words across 513 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.