Which is The Best Qwen3.8-27B? — Transcript
Full transcript
- 0:00Okay, so this is the same question going
- 0:03into two versions of the same model.
- 0:06Same weights underneath, similar answer
- 0:08at the end, but one of them has used a
- 0:10lot more thinking tokens to get there,
- 0:12and the other has used hardly any in
- 0:14comparison to get there. So on my
- 0:16machine, that's a difference between 18
- 0:18seconds and 90 seconds. And if you think
- 0:20about it, that's sort of the problem
- 0:22with reasoning models now, right? They
- 0:24get the answer, but they take a long
- 0:26time to get there. It's almost like one
- 0:27model flies there to get the answer as
- 0:30fast as possible and the other one
- 0:32drives around the coast to take a scenic
- 0:33route. And you'll see this in ways that
- 0:35are often not totally bad. They'll do
- 0:38things like they restate the question,
- 0:40they check. They check again. They say
- 0:42to themselves, "Wait, let's reconsider."
- 0:45And every one of those tokens is time
- 0:47that you're spent waiting for this
- 0:49answer. And of course, money that you're
- 0:52paying for the inference for this. Now,
- 0:54in the past, I've talked about Open AI
- 0:56focusing on reducing the number of
- 0:59tokens between the 5.0 model and the 5.5
- 1:03model. Well, they're not the only ones
- 1:05that have been focused on this, right?
- 1:07Most of the major labs have been looking
- 1:09at how do they take these long chains of
- 1:11thought and reduce them but still keep
- 1:13the accuracy. And the cool thing is that
- 1:15there are also teams trying to do this
- 1:17with open-source models. So, in the
- 1:19past, I covered the thinking cap model
- 1:20for the Quen 3.627B 627B. And no
- 1:24surprise, I'm going to cover their new
- 1:25model in this video, but also I want to
- 1:27look at some of the other teams that
- 1:29have taken on the Quen 3.827B
- 1:33and fine-tuned it to think less. So on
- 1:36top of thinking cap, we've also got
- 1:38Quenpi, which is sort of like a
- 1:40specialized version with less thinking
- 1:42for coding agents. Got Swift 1.5. They
- 1:46all claim fewer tokens for about the
- 1:48same accuracy, but it seems they all go
- 1:51about it in really different ways. And
- 1:53it also seems that they're probably
- 1:55really good at different things. So, in
- 1:56this video, I want to look at how each
- 1:58one was made, then I'll run all of them
- 2:00against the base model and see if the
- 2:02claims actually hold up. All right, so
- 2:04before we do the fine tunes and look at
- 2:06those, you need to know a bit about the
- 2:09sort of thinking and reasoning on Quen
- 2:113.8 that it already gives you out of the
- 2:14box. and they call it reasoning effort.
- 2:16It has three settings, low, medium, and
- 2:19X high. And amazingly, out of the box, X
- 2:21high is actually the default. So, if
- 2:23you're serving with VLM, you actually
- 2:25set this in the chat template. And you
- 2:28can see that when we set that reasoning
- 2:30effort, that's going to affect how long
- 2:32the chains of thought are that's coming
- 2:33back. Now, I talked about this when I
- 2:35made the video about serving Quen
- 2:373.827B,
- 2:39and I got to say that model is still one
- 2:41of the best open models that you can
- 2:43get. and serve on reasonably small
- 2:45hardware. For some uses, the 3.8 next is
- 2:48probably better, but that's going to
- 2:50move you into the realm of needing a
- 2:52bigger GPU or needing multiple units in
- 2:54there. Now, back to the 3.827B.
- 2:57The whole reasoning effort there is you
- 3:00are basically telling the model to try
- 3:02less hard. So, yes, you can get less
- 3:05tokens out of it, but when you actually
- 3:07turn that reasoning down, it's often
- 3:09going to cost you inaccuracy. And if
- 3:11you're running this locally on 1GP,
- 3:13you're really going to be decode bound.
- 3:15So the thinking or reasoning tokens
- 3:17there match pretty much directly to
- 3:20seconds for you getting your final
- 3:21result out. If you can have those
- 3:23tokens, you're roughly having the weight
- 3:26that you're going to have to wait for
- 3:27your final answer out. Secondly, these
- 3:29models have multi-token prediction heads
- 3:31for speculative decoding. And that
- 3:34actually works better on shorter traces
- 3:36than longer traces. And we actually see
- 3:39this in some of the reported results.
- 3:40So, thinking cap's model card actually
- 3:43reports that speculative decoding goes
- 3:45from about 2.6 tokens per step at X high
- 3:49to over three tokens per step at medium
- 3:52and low. So, you can actually think
- 3:54about this as thinking less doesn't just
- 3:56mean fewer tokens. Each token comes out
- 3:59faster as well. All right. So, the big
- 4:00question here is what are these fine
- 4:02tunes actually trying to do? And are
- 4:04they going to get you the savings
- 4:06without any sort of accuracy hit here?
- 4:08All right. So here's the family tree.
- 4:10Everything starts from Quen 3.827B.
- 4:14On one branch of the tree, you've got
- 4:16Thinking Cap from Bottlecap AI. Their
- 4:19first thinking cap model was the one
- 4:20that I covered a couple of months back
- 4:21and that was built on the Quen 3.6 model
- 4:24then. And you could say related to this
- 4:26was the UKIS AI swift model. So when
- 4:30they built their first version of Swift,
- 4:32part of that recipe was a transfer
- 4:34component from the thinking cap model.
- 4:36So you can sort of think of Swift of
- 4:38having some of the thinking cap DNA in
- 4:40it and being related. Now on a
- 4:43completely separate branch here, you've
- 4:44got Quen Pi which is playing a different
- 4:47game totally. I'll get to that as we go
- 4:49through each of the models, but let's
- 4:50start with Thinking Cap. So Bodcap AI is
- 4:53a lab in Prague and one of the
- 4:55co-founders there is Thomas Michelov. So
- 4:57at the time that I recorded the original
- 4:59video about the first Thinking Cap, I
- 5:01didn't actually know that until one of
- 5:02you pointed it out in the comments. And
- 5:05so for those of you who don't know who
- 5:06Thomas Mikolof is, he was actually the
- 5:09first author on a very famous paper word
- 5:11tovec. So this is one of the key papers
- 5:13that really changed natural language
- 5:16processing. The whole idea of having
- 5:18vectors that could represent words and
- 5:21then could be used for things like
- 5:22classification like a whole bunch of
- 5:24these things. At the time Michelof was
- 5:26working at Google from memory he was
- 5:28also working on a lot of the ideas for
- 5:30seek to seek which all of these were
- 5:32very early precursors to the transformer
- 5:34idea of sequence in sequence out. Now
- 5:37after one of you had pointed that out. I
- 5:39actually reached out to him to send the
- 5:40video and message him where they're
- 5:42going to do one for 3.8. And I found out
- 5:45they're working on an interesting
- 5:46project that hopefully I can cover in
- 5:47the near future. But clearly I wasn't
- 5:49the only one giving feedback that the
- 5:51community wanted a 3.8 eight version of
- 5:54the thinking cap models. So I think they
- 5:56reallocated some of the compute that
- 5:58they were using for their other project
- 6:00to this. And just a very quick side note
- 6:02here, I would say that we really should
- 6:04be grateful for startups like this. His
- 6:06startup is not some billion-dollar
- 6:07startup in San Francisco that's training
- 6:09their own frontier models to compete
- 6:11with OpenAI or Anthropic, etc. Bottlecap
- 6:15is a startup that's raised a tiny
- 6:16fraction of that money compared to some
- 6:18of these bigger ones and most likely has
- 6:21a fraction of the compute that those
- 6:22more famous startups actually have. So
- 6:25the fact that they're allocating compute
- 6:26to do something like this for the
- 6:28community, I think is very cool. All
- 6:30right, so how was this actually made?
- 6:32Well, Bodcap didn't publish the actual
- 6:34training recipe, but they're clear about
- 6:36the objective here. The target was one
- 6:38thing, how many thinking tokens it takes
- 6:40to reach the same answer. And they
- 6:43mentioned that the aim was deliberately
- 6:44conservative here. They didn't try to
- 6:46make the model smarter, to teach it
- 6:48anything new, to change how it talks.
- 6:51They haven't tried to add any knowledge
- 6:52in here or even improve the instruction
- 6:55following or safety or any of those
- 6:57things. Those are meant to come through
- 6:59totally untouched. So stream to X high
- 7:01and this time they put more focus on
- 7:03much harder sort of benchmarks and
- 7:05tasks. So things like math, reasoning,
- 7:07long context, agentic tasks, etc. You
- 7:10can kind of think of this is teaching
- 7:12the model when it's done rather than
- 7:14teaching it to be brief. So the headline
- 7:17here is 37% fewer tokens on average
- 7:20across 12 benchmarks with the average
- 7:23accuracy going from 86.6 to sort of
- 7:2685.8. So it's just a tiny percentage
- 7:29less accuracy there. So, for example, on
- 7:32the GPQA diamond, this goes from an
- 7:34average of 12,800 thinking tokens on the
- 7:37original version down to 7,300 on the
- 7:40thinking cap one. And interesting on
- 7:42some of them, like the long context
- 7:44result is a really interesting one. On
- 7:46their long context retrieval benchmark,
- 7:48it uses 39% less thinking, but the
- 7:51accuracy actually goes up a couple of
- 7:53points there. The chart I think that is
- 7:55most useful in their blog post about
- 7:58this is this one because it ties
- 8:00straight back to where we set the
- 8:01reasoning effort. If you take the base
- 8:04model and just drop it to medium, you
- 8:06remove about half the thinking, but you
- 8:08give up about nine points of accuracy.
- 8:11Thinking cap at X high removes 37% for
- 8:15under a point of accuracy. So suddenly
- 8:18that X high level gets really useful
- 8:19there. All right. So what's this model
- 8:22actually good for? It's basically just a
- 8:24drop-in replacement for the Quen version
- 8:27at X High. So anything that needs
- 8:29general reasoning, knowledge questions,
- 8:31long documents, etc. All of those
- 8:33things, you can basically just drop this
- 8:35model in. Perhaps the only downside I
- 8:37would say here would be the agentic
- 8:39traces. And not because they're not
- 8:41shorter. They're shorter, but it seems
- 8:43that the reasoning only drops by about
- 8:4511% there. Now, interestingly, the other
- 8:47thing here is that this version is made
- 8:49for X high. If you want a version that's
- 8:52made for the medium or lowinking and
- 8:55optimized for that, that's something
- 8:56that they're actually selling to
- 8:58enterprises. And along with that, the
- 9:00other change here is the actual license.
- 9:03So, it's gone from being Apache 2 for
- 9:05the 3.6 version to this one being a
- 9:08small business plus personal use kind of
- 9:10license, which totally makes sense if
- 9:12they're trying to make some money to
- 9:13actually cover their compute costs, etc.
- 9:16All right. So, next up is Swift 1.5 from
- 9:20Yucus AI. And this one's quite a lot
- 9:23more aggressive. So, this one is going
- 9:25for up to 58.5%
- 9:28fewer thinking tokens on the Quen
- 9:313.827B.
- 9:33They built it in two steps. So, for
- 9:35Swift 1.0, they were really looking for
- 9:37specific tokens in Quen's reasoning that
- 9:40seem to trigger overthinking. And you
- 9:42could think of some examples of those
- 9:44are where things like it says wait let
- 9:46me ask what the user wants hm and then
- 9:49you know asks itself something let me
- 9:51reconsider and then they fine-tuned the
- 9:53model to penalize those tokens rather
- 9:56than just penalizing the length and then
- 9:58they added that to the original version
- 10:00of the thinking cap model for 1.5 though
- 10:02they took the model and scaled up the
- 10:04reinforcement learning and on policy
- 10:06distillation with their main focus being
- 10:09on long horizon agentic and coding ing
- 10:12tasks. Now, the cool thing here is that
- 10:14not only have they published the models,
- 10:16the training data is also on
- 10:17HuggingFace, which is really good to see
- 10:20here. You can see it in the coding
- 10:21numbers here on their model card. Live
- 10:23Codebench goes from just under 77% for
- 10:26the base model with around 11,200 tokens
- 10:30out to actually an increase of accuracy
- 10:32of 81.7%
- 10:34with only 8,400 tokens out. So, the
- 10:37other interesting thing here is that
- 10:39they also claim that the savings that
- 10:40they're getting hold at every effort
- 10:43level. At low, it actually scores
- 10:45slightly above the base model with about
- 10:4729% fewer thinking tokens. Practically
- 10:50too, this one has probably the most ways
- 10:52for you to run it. There are standard
- 10:54Lama CPP GGUFS for this. There are small
- 10:57ones to fit on small GPUs. There's MLX.
- 11:00There's an NVF4 version and even an AMD
- 11:03build here. And they even have like a
- 11:05free research API for this with no API
- 11:08key if you want to just try this out.
- 11:10Now, that headline of the 58% fewer
- 11:12thinking tokens probably turns out to be
- 11:15a bit less when you start looking at the
- 11:17mean thinking reduction across this, but
- 11:19it's still very impressive here. Another
- 11:22thing that they've focused on is that
- 11:23they claim that the base model loops and
- 11:25fails early, which they're claiming that
- 11:27their version doesn't do. Now, the
- 11:29license for this one is their own custom
- 11:31license, which is free up to a million
- 11:33dollars in revenue. After that, you need
- 11:36to basically contact them. All right, so
- 11:38that brings me to the third one, which
- 11:39is kind of the odd one out here. And
- 11:41incidentally, this is the only one of
- 11:43the three that actually has an Apache 2
- 11:45license. So, this is Quenpi. To
- 11:48understand it, you first need to know
- 11:49about what PI is. Pime is an open-
- 11:51source coding agent that's very
- 11:53deliberately minimal. It has a very
- 11:56limited set of tools for doing read,
- 11:58write, edit, and bash. And last time I
- 12:01checked, it even has a system prompt
- 12:02that's under a thousand tokens. So Quen
- 12:05Pi is Quen 3.827B
- 12:08fine-tuned specifically to work inside
- 12:10of the Pi harness. And I want to cover
- 12:12this one because I think this is
- 12:13something we're going to see a lot more
- 12:15of where people fine-tune a particular
- 12:17model for a very specific harness. And
- 12:21it totally makes sense. If you can get
- 12:23the model and the harness to work
- 12:25together, and the model can be primed
- 12:27for what kind of inputs it's going to
- 12:28see, what kind of inputs it should
- 12:30generate out, you're probably going to
- 12:32end up with a model that's a lot more
- 12:33suited to a very particular vertical
- 12:35harness there. And this idea is
- 12:37something that I've been running tests
- 12:38on recently with the number of vertical
- 12:40harnesses in the co-scientist area. So,
- 12:42I feel like this model shows a little
- 12:44bit of where that direction is actually
- 12:46going. So, how was this made? First,
- 12:49it's basically supervised fine-tuning on
- 12:51real PI sessions, but only the
- 12:53successful ones. So, the model learns
- 12:55the whole workflows, how to look at
- 12:57code, how to make a change, how to run
- 13:00tests, how to fix what's broken, etc.
- 13:02Secondly, they make sure in those tasks
- 13:04that they actually check for working
- 13:06code versus broken code separately
- 13:08themselves. Thirdly, they're doing
- 13:10reinforcement learning with GPO and a
- 13:13custom reward for reasoning efficiencies
- 13:15there. So obviously the goal there is to
- 13:17basically reduce the long chains of
- 13:19thought. Interestingly that reward is
- 13:21aimed at the low and medium settings
- 13:23while X high is left to focus on getting
- 13:25it right. The other interesting thing
- 13:26that's mentioned in here is that they
- 13:28actually pick the checkpoints based on
- 13:31actual agent results not just based on
- 13:34training loss. So you could imagine that
- 13:36as the model gets better and the RL
- 13:39kicks in to reduce that training loss
- 13:41that doesn't always lead to the best
- 13:43agent runs. So, they're much more
- 13:45focused on getting this to actually work
- 13:47for agent runs than purely just for some
- 13:50LLM training goal. Now, my guess here is
- 13:52a big part of why this works is that PI
- 13:55is a minimal harness. When it's only got
- 13:57a few tools and a really tiny prompt
- 13:59like that, every session looks
- 14:01reasonably similar. And you can imagine
- 14:03that that generates really nice data to
- 14:05actually fine-tune on. So, the idea with
- 14:08this one is different. Thinking Cap and
- 14:09Swift are trying to make your X high
- 14:11shorter. Pi is actually trying to make
- 14:14the low and medium effort settings
- 14:16actually good. On their numbers, Pi's
- 14:18medium setting matches the base models X
- 14:21high on Terminal Bench with about 41%
- 14:24fewer output tokens. They're trying to
- 14:26reduce the number of tokens you're using
- 14:28by actually getting you to use the lower
- 14:30and the medium levels by increasing
- 14:32their intelligence up to the high level.
- 14:35Now, obviously, the weakness for this
- 14:36one is that it's a very narrow use case.
- 14:40All the evidence comes from using it
- 14:42inside of Pi. From what I've seen,
- 14:43nobody has really shown that this
- 14:45carries over to other harnesses. If you
- 14:47wanted to use open code or you wanted to
- 14:49use other things, etc. And on the
- 14:51general sort of reasoning tasks like the
- 14:53GPQA, the base model still actually wins
- 14:56at the medium level. So, in many ways,
- 14:58I'd say this is sort of like a community
- 15:00testing effort that perhaps isn't
- 15:02totally proven yet, but is certainly
- 15:04very promising here. So, let's jump in
- 15:06and actually take a look at some of the
- 15:07outputs that we've got here. Okay, so
- 15:09each of these models is running on a
- 15:11Dell Pro Max tower here and they're
- 15:13running on an RTX Pro 6000 with standard
- 15:17VLM here. I haven't really optimized any
- 15:19of that in here. So, thank you to Dell
- 15:21for sponsoring the compute for this. It
- 15:24really makes it possible for me to be
- 15:25able to benchmark these models out.
- 15:27Okay, so let me set up the demo here.
- 15:29What I've basically done is I've gotten
- 15:32Claude Fable and Opus to generate a
- 15:35bunch of questions or prompts that that
- 15:39are going to create a large amount of
- 15:40thinking tokens or reasoning tokens etc.
- 15:43And then what I've done is I've gone
- 15:45through and actually rendered out the
- 15:47inference for each of these models and
- 15:50saved it with exact timestamps so that
- 15:53we can look at all four of them playing
- 15:55back at once here. So you can see here
- 15:57we've got the original Quen 3.827B.
- 16:01This is technically the fine-tuned model
- 16:03that they released of their base model,
- 16:06but this is sort of the standard version
- 16:08of the model. I've got the Quen Pi one,
- 16:12I've got the Thinking Cap one, and I've
- 16:14got the Swift one in here. So you'll see
- 16:16that if I come up and I pick something
- 16:18like this, I'm actually playing back at
- 16:2116 times the speed that it was recorded.
- 16:25And you can see sure enough here this is
- 16:27now thinking cap has done. It's
- 16:30completed. It's got the correct answer.
- 16:33We've now got swift 1.5 done completed
- 16:36with again with the correct answer.
- 16:38We've got the quen pi model done with
- 16:40the correct answer. And finally at some
- 16:43point we will get the original 3.827b
- 16:48looking at this one also with the
- 16:50correct answer there. So in this case,
- 16:52each of the models has given us the
- 16:54exact same answer back. But we can see
- 16:57very clearly in this case that thinking
- 16:59cap was way shorter, right? With only
- 17:034,000 tokens. You can see that even the
- 17:05inference speeds are roughly the same.
- 17:07Second is the swift model and third is
- 17:11the pi model and then finally is the
- 17:14long version. So all three of them are
- 17:16doing much better than the original
- 17:19version. Let's try it with a different
- 17:21prompt. Okay, so again you can see these
- 17:24are all off and we can see in this case
- 17:27the swift model actually was fastest
- 17:30followed by the pi thinking followed by
- 17:33the original model and in this case
- 17:35thinking cap actually came last. All
- 17:37right. Now, I will point out that this
- 17:39one is more of a sort of coding task
- 17:42with this. And remember that the pi
- 17:44model is made for coding tasks and tends
- 17:48to specialize in that kind of thing. All
- 17:50right. If we look at another one like
- 17:52this, we can see that sure enough, in
- 17:55this case, the Swift model did really
- 17:57well followed by thinking cap. The pi
- 17:59model took longer than both of those,
- 18:01but still was way shorter than the
- 18:03original one. All right. If we give it
- 18:05sort of a logic puzzle to go through,
- 18:08you see in this case the thinking cap
- 18:10one, swift came second followed by the
- 18:13base model and then pi. If we give it
- 18:15another mathematics one, we can see in
- 18:16this one that thinking cap 1, swift came
- 18:19second, pi came third and then the
- 18:22original model was quite a long way
- 18:23behind that. And you see if we translate
- 18:25these into time that we're getting huge
- 18:29savings with some of these models, the
- 18:31thinking cap one is just a lot faster, a
- 18:34lot more concise in what it's actually
- 18:36producing there. Another example, again,
- 18:39thinking cap first followed by Swift.
- 18:42And in this case, the base version
- 18:43actually beat the PI version. So just to
- 18:46finish up, I'll show you one that is
- 18:48kind of a bit surprising. In this one,
- 18:51you've got the Swift model actually
- 18:54doing slightly better. We've then got
- 18:57the base model coming second, followed
- 18:59by thinking cap, and finally the PI
- 19:01model in here. And for me, the
- 19:03interesting thing here is that even
- 19:05though we've got the same prompt for
- 19:07making the SVG, each of them is making a
- 19:09slightly different SVG, you know, coming
- 19:12out of it. And I would say this is sort
- 19:14of the perfect example of where you
- 19:16probably should try each of them out for
- 19:19your specific use case because just as
- 19:21the SVGs change in quality in here, you
- 19:25will find that other answers that don't
- 19:27necessarily always have an exact answer
- 19:30also can change quite a lot from model
- 19:32to model in here. While in my test,
- 19:34thinking cap seems to be winning most of
- 19:36the time, the variance can be really
- 19:39different on different kinds of
- 19:41questions. And I think that's got a lot
- 19:43to do with just how people are doing the
- 19:46fine-tunings here. Whatever they make
- 19:48the model focus on and use the RL on,
- 19:50it's going to get good at that. And I
- 19:52think for each of these three fine tunes
- 19:54here, they've gone for slightly
- 19:56different data curation and perhaps
- 19:58curriculum learning in the way that
- 20:00they've actually put these together. So
- 20:01if you remember back, I also talked
- 20:03about that the Quen Pi model was
- 20:05actually trained to do better on Medium.
- 20:08And the thing I find is that sure
- 20:10enough, this model actually is a lot
- 20:13faster out on medium than the original
- 20:16base model. It doesn't mean though that
- 20:18it's always going to be faster for all
- 20:20of them. Often the thinking cap one will
- 20:23still be faster for some of these. And I
- 20:26find that even the thinking cap one
- 20:28tends to be a little bit more accurate
- 20:30at a lot of the different tasks and
- 20:32stuff when you have it actually set to
- 20:34medium. In this particular use case
- 20:36here, you can see though when we compare
- 20:38this that the PI model is actually being
- 20:41faster than both of them when we've got
- 20:44the setting on medium there. So anyway,
- 20:46let me know in the comments if you've
- 20:47actually had a play with these, how you
- 20:49found them out, which one of the three
- 20:51is your favorite. I would say that all
- 20:53three of these are actually very good.
- 20:55In general, they all tend to do better
- 20:58than the base model for a lot of the
- 21:00particular tasks. Now, the Pi 1 seems to
- 21:03me to be definitely more suited to
- 21:04coding kind of tasks and it really
- 21:06shines at those. The thinking cap and
- 21:08the Swift one are much more sort of
- 21:10generalist models. So, anyway, as
- 21:12always, if you like the video, click
- 21:14like and subscribe and I will talk to
- 21:16you in the next video. Bye for now.
About this transcript
This page contains the full transcript of Which is The Best Qwen3.8-27B? by Sam Witteveen, generated from the public captions YouTube serves with the video. The transcript has 4,030 words across 568 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.