MiniCPM5-2B: The Best Sub-Agent Model Yet? — Transcript

Full transcript

1. 0:00Okay, so a few months ago MiniCPMQ
2. 0:03released their 1B model and I covered
3. 0:05this on the channel talking about the
4. 0:07whole concept of sort of having a small
5. 0:10cognitive model that could do a lot of
6. 0:12different things. Well, after a few
7. 0:14months they are back with a new model
8. 0:17and they've jumped from 1B to 2B here.
9. 0:21In fact, I think it's actually a 2.5B
10. 0:23model, but we'll give them that. So,
11. 0:26this is the second model in this mini
12. 0:28series from OpenBMB. And in many ways
13. 0:32they've taken the same sort of recipe,
14. 0:33but they've scaled it from 1B to 2B. So,
15. 0:37I guess the big question here is when
16. 0:38you scale the data and post training
17. 0:41recipe by 2x, does that suddenly make
18. 0:44this model a lot more viable for the
19. 0:46hard function calling and along with
20. 0:49that 2x scaling, does the agentic stuff
21. 0:52scale as well? So, the interesting thing
22. 0:54here is that the 1B model was pitched at
23. 0:58sort of coding agents and tools use. The
24. 1:002B model is claiming to be 4B models at
25. 1:05the same kind of thing. If we come in
26. 1:07here and look at their benchmarks, we
27. 1:08can see that they're edging out Qwen
28. 1:103.5B
29. 1:12on things like SweetBench verified, etc.
30. 1:15Now, when we look at the training
31. 1:16recipe, it seems to be mostly the same
32. 1:19and it seems that these scaling
33. 1:20improvements really did sort of have an
34. 1:22effect here. In fact, in some places
35. 1:25more than I expected. So, let's take a
36. 1:27look at this stuff. So, where the
37. 1:29supervised fine-tuning for the 1B was
38. 1:31200 billion deep thinking tokens and 200
39. 1:35billion hybrid tokens, the 2B has gone
40. 1:38for 400 billion tokens of deep thinking
41. 1:41only. But, we can see where some of
42. 1:42these changes have come in both the RL
43. 1:45and the on-policy distillation here. We
44. 1:48can see that they've changed up how
45. 1:49they're doing the RL stage. So, they've
46. 1:52got this new thing called RL2. It looks
47. 1:54like they're going to release a whole
48. 1:56paper about this. This is just sort of
49. 1:58like an early pre-print here. And the
50. 2:00key thing here is that they've changed
51. 2:01how they're doing this here to basically
52. 2:03include a critic in the RL as well. All
53. 2:06right, so if we take a look at their
54. 2:07training recipe, we can see that they
55. 2:10have this sort of pre-training and
56. 2:13mid-training here. And we can see that
57. 2:15that's basically setting up the model
58. 2:16for the post-training processes. So if
59. 2:19we look at the post-training, the
60. 2:20post-training is breaking down to three
61. 2:22different parts, right? You've got your
62. 2:24standard SFT. This is sort of providing
63. 2:27the format for outputs, providing sort
64. 2:30of some of the key thinking behavior,
65. 2:32etc. They then do RL per domain. So
66. 2:36they've got multiple experts that
67. 2:37they're doing the RL and the on-policy
68. 2:39distillation stuff from there. And we
69. 2:42can see that those are sort of breaking
70. 2:43down into sort of reasoning task RL,
71. 2:47general task RL, and agentic stuff
72. 2:49there. And the key thing with that is
73. 2:51that each of those experts only needs to
74. 2:54be good at one thing. This is the new
75. 2:57form of RL here, this critic-based just
76. 3:00RL2. And then they're claiming that this
77. 3:03actually has better stability than the
78. 3:05sort of traditional GRPO stuff. But it's
79. 3:08interesting that this is showing here
80. 3:09that this makes the model a lot better
81. 3:12on a lot of their different benchmarks.
82. 3:14Now finally, they have their on-policy
83. 3:15distillation layer. Here the model is
84. 3:17sampling out its own trajectories that
85. 3:19were used to train the teacher models.
86. 3:22And then from that, they can then
87. 3:23basically look at how different it is to
88. 3:25the teacher models and then update in a
89. 3:27loop, etc. So when we go back into the
90. 3:29benchmarks, they're doing a lot better
91. 3:32than obviously a lot of the 2B models,
92. 3:34but they're doing very well compared to
93. 3:37a lot of the 4B models in here. And so
94. 3:41the trade-off here is that it's not a
95. 3:42huge amount to go from 1 to 2B if we're
96. 3:45getting a model that's substantially
97. 3:46better and now also has things like
98. 3:49general knowledge in it. If we come down
99. 3:51and look at the things like the GBQA
100. 3:53diamond, etc., we've got a model now
101. 3:56that, okay, is a bit behind the 4B
102. 3:59model, but this is way ahead of the
103. 4:01other sort of 2B models that are out
104. 4:03there. And of course, it's miles behind
105. 4:06the frontier models that are well up
106. 4:08into the high 90s, etc., at this point,
107. 4:10but not that long ago, this kind of
108. 4:12score was considered a good score for
109. 4:15models that were 10x to 100x bigger than
110. 4:18this particular model. Now, the cool
111. 4:20thing here is that they've released a
112. 4:22bunch of different versions of this
113. 4:24model. You've got sort of the final
114. 4:26model that's got everything in it.
115. 4:28You've got just the SFT models before
116. 4:31RL. You've got mid-training models.
117. 4:33You've got the base model in there. And
118. 4:35then you've got a bunch of quantizations
119. 4:37and a deep spark model in here, which is
120. 4:40good for the speculative decoding, etc.
121. 4:42Of course, one of the good things about
122. 4:43this size of model is that it's very
123. 4:45easy to fine-tune for your specific use
124. 4:47case. Then you face the challenge of
125. 4:50what if you wanted to have that
126. 4:50fine-tune on whenever you wanted to
127. 4:52access that? And that actually brings me
128. 4:54to today's sponsor. So, one of the
129. 4:56things that you often want to do is get
130. 4:58a model like this set up as an OpenAI
131. 5:01compatible API that scales. And that's
132. 5:03where RunPod Serverless comes in. So,
133. 5:06thanks to them for sponsoring this
134. 5:07video. Let me just show you how quickly
135. 5:09you can go from having a model on
136. 5:11Hugging Face to being able to deploy it
137. 5:14to VL on a serverless endpoint with a
138. 5:17GPU that you choose. So, you can see
139. 5:19here that I've picked the model from
140. 5:20Hugging Face. I now have to pick a GPU
141. 5:22with enough memory to run it on. Then I
142. 5:25can just set my scaling limits and
143. 5:27create an endpoint. Once it's ready, I
144. 5:29can now send a prompt and the model
145. 5:30returns a response. It's literally that
146. 5:33easy to have an API that my applications
147. 5:35or my agents can call. Behind the
148. 5:37scenes, RunPod routes the request to a
149. 5:40worker running my model on a GPU. And if
150. 5:42demand grows, RunPod can automatically
151. 5:45scale up those workers within the limits
152. 5:47that I've set. And if I don't use it for
153. 5:49a while, it can scale it back down to
154. 5:51zero. The cool thing here is whenever I
155. 5:53ping a new request, the model will spin
156. 5:56up and then be able to serve it.
157. 5:58Computers billed by the second while
158. 6:00workers run, so you don't have to keep a
159. 6:02GPU rented around the clock. They also
160. 6:04have flash boot, which helps reduce cold
161. 6:06starts. And if responsiveness is the key
162. 6:09priority, I can actually pay to keep the
163. 6:11workers running and ready all the time.
164. 6:14So, all of this just gives you the
165. 6:15control to serve the model where the GPU
166. 6:17infrastructure is all managed for you,
167. 6:20and you only have to pay for the time
168. 6:22that you're actually using it. If you're
169. 6:23building any AI app or deploying your
170. 6:25own model, check out RunPod Serverless
171. 6:28at the link below and get started today.
172. 6:30All right, so did everything actually
173. 6:32scale here and this clearly is beating
174. 6:344B models? Actually, the answer is no.
175. 6:36If we look at things like SweetBench
176. 6:38Pro, we can see that the Qwen model is
177. 6:41substantially better than this,
178. 6:43basically double it. And if we look at
179. 6:45Terminal Bench, we can see that the
180. 6:47model is way behind the Qwen model. But,
181. 6:49it still is a lot further advanced than
182. 6:52a lot of the other 4B models out there.
183. 6:54And don't forget that the Qwen models
184. 6:55probably have a much bigger budget for
185. 6:57pre-training, for a lot more tokens, for
186. 7:00a lot more things like that.
187. 7:02Unfortunately, we haven't seen the
188. 7:03really small Qwen models being updated
189. 7:06to say the 3.8 models. Hopefully, that's
190. 7:09going to come with the not too far off
191. 7:11Qwen 4 models. So, let's jump in and
192. 7:13have a look at the speed of this of
193. 7:14running it locally. Have a look at just
194. 7:16how it actually does on some of the
195. 7:17function calling and some of the agentic
196. 7:20tasks in my sort of testing suite. All
197. 7:23right, so I've got three versions of the
198. 7:25model running here. I've got the
199. 7:27BFloat16 version, I've got a DeepSpark,
200. 7:30so basically a speculative decoding
201. 7:32version, and I've got the 4-bit GGUF
202. 7:35version with llama.cpp. I'm running the
203. 7:37two bigger versions on SG Lang here, and
204. 7:41you can see that if I kick it off, I'm
205. 7:43actually getting pretty good token
206. 7:45speech, which is to be expected. I'm
207. 7:47running all of these on an RTX Pro 6000.
208. 7:51Now, if we look at the performance of
209. 7:53this model, pretty much it's as I
210. 7:55expected in here, that the model is not
211. 7:58great with knowledge sorts of things.
212. 8:00You can see here I asked it for a 5000
213. 8:02word essay. It's giving me a reasonably
214. 8:04short amount of thinking on it, but the
215. 8:06essay is nowhere near 5000 words, not
216. 8:09even close in here. If we go in have a
217. 8:12look at things like the logic puzzles,
218. 8:15we'll see that this will fly along and
219. 8:18it gets to the right answer here quite
220. 8:20quickly. So, the logic stuff is actually
221. 8:22going quite well. The same here for when
222. 8:25we look at something like the hard math
223. 8:27stuff, it's doing well. And that kind of
224. 8:29reinforces what this model's actually
225. 8:31good at, right? But if we come over and
226. 8:33we look at something like the SVG
227. 8:35Pelican test, this thing just doesn't do
228. 8:38well at this kind of thing. You can see
229. 8:40that we're starting to get really high
230. 8:41token speeds on this, but when we
231. 8:44actually look at the outputs, maybe they
232. 8:46actually are sort of better as abstract
233. 8:48art for certain things, but that's okay
234. 8:51because that's not what the model's for.
235. 8:53Exactly the same kind of thing when we
236. 8:55look at the generating HTML page, right?
237. 8:58If we look at the code that comes out,
238. 9:01it's not that it's necessarily anything
239. 9:03hugely wrong. It's just that it's not
240. 9:05nowhere near as good as any of the other
241. 9:08models that we would use for coding
242. 9:10these days. That's not what this model
243. 9:12is actually about. Where this model
244. 9:14shines is going to be not at the coding
245. 9:18outputs, but it's going to be at the
246. 9:20tool use. So, you can see now I've moved
247. 9:22over to the GGUF version. It's flying at
248. 9:25well over 400 tokens per second, no
249. 9:27problems here. But let's come over and
250. 9:29look at how this actually does on the
251. 9:32Agentech tests. Remember, these are a a
252. 9:35of tests that test for function calling
253. 9:38to see does it provide the right
254. 9:41arguments to functions, is it able to
255. 9:43use tool use well and stuff like that.
256. 9:45And this is sort of like the basic set
257. 9:47of tests here. And not only is it flying
258. 9:51speed-wise here, it's getting all of
259. 9:53them right. So we're getting some really
260. 9:55nice results out of them. So then we
261. 9:57come over to the more advanced one. So
262. 10:00here the goal is to basically test
263. 10:02things that often go wrong when you're
264. 10:05running these kinds of things. And so
265. 10:08again, I'm using the GGUF version. I'll
266. 10:10just say going through this that the
267. 10:12current setup with the SG Lang is
268. 10:14actually seems to be the tool use seems
269. 10:16to be returning the wrong format. So
270. 10:18when I test out the Bfloat16 and the D
271. 10:21Spark versions, I'm getting some issues.
272. 10:24But it's pretty clear that they're not
273. 10:25because of the model, just seems to be
274. 10:26the configuration on SG Lang. When I try
275. 10:29it on llama.cpp with the GGUF version,
276. 10:33so this is a four-bit version, and of
277. 10:35course with this you would expect that's
278. 10:37actually going to be worse, it works
279. 10:39fine. And you can see here that it's
280. 10:41gone through and passed eight out of
281. 10:43eight for all of these tests. Now some
282. 10:45of the tests in here that I think are
283. 10:46interesting are the sort of planning
284. 10:49tests where it's using multiple
285. 10:51different tools as it goes through. It
286. 10:53has to deal with errors often before it
287. 10:56gets to a correct output. With something
288. 10:58like the prompt injection resistance, it
289. 11:00has to work out how to basically extract
290. 11:03out the right kind of thing going in
291. 11:05here. And I'll point out while I'm
292. 11:07showing you one run of these tests here,
293. 11:09I run them at least 10 times going
294. 11:11through. Now this one did fail a couple
295. 11:13of the times, but it's been passing
296. 11:15around about 80% of that. The other one
297. 11:18too, which is one that a lot of models
298. 11:20in the past used to have lots of
299. 11:21problems with, were things like failure
300. 11:24retry tests. So what's going on here is
301. 11:27this is deliberately pretending to fail
302. 11:30for two of the times, and to see does
303. 11:34the model keep going at trying to get
304. 11:37the result out. So, you can see that
305. 11:39these failures were actually created
306. 11:40deliberately. The model then keeps
307. 11:42going. It's able to basically get the
308. 11:44result out we want. The distractor tool
309. 11:46bait also very impressive in that it's
310. 11:49able to basically find the right
311. 11:50information in there. If I come and test
312. 11:53things out in the tool playground, here
313. 11:55I think it's really interesting to see
314. 11:58that this model has no problem both
315. 12:01dealing with multiple tool calls here.
316. 12:03But, you can see that this one it's
317. 12:04having to use DuckDuckGo search and a
318. 12:06fetch URL here. And it's lightning fast
319. 12:10going through this. I'm getting sort of
320. 12:12end-to-end over 500 tokens per second.
321. 12:15And I guess partly because it's so
322. 12:17small, it has no problems with large
323. 12:19page pre-fills, being able to extract
324. 12:21out what's the right information in
325. 12:23there to pay attention to. Again, on
326. 12:25multi-tool stuff, again we've got very
327. 12:27fast result. We've got accurate use of
328. 12:30the tools that are actually in there.
329. 12:32And one of the things that I really like
330. 12:33in here is that it's not trying to use
331. 12:37its knowledge, right? The fact that it
332. 12:39doesn't have the knowledge is fine as
333. 12:41long as it doesn't pretend to have the
334. 12:43knowledge. And we want it to actually
335. 12:45defer to its tools rather than try to
336. 12:48basically make everything up itself. So,
337. 12:50for me this is the biggest issue with a
338. 12:52model like this is that if we do
339. 12:54actually give it a lot of tools and we
340. 12:56ask it to defer to the tools rather than
341. 12:58its own knowledge, does it have a
342. 13:00problem with that or not? And it's great
343. 13:02to see that if you give it the tools, it
344. 13:04seems to know not to use its own
345. 13:06knowledge. And that's exactly the kind
346. 13:08of model that we want for sub agents,
347. 13:12right? For anything that you're doing
348. 13:13agentic where you've got sub agents with
349. 13:15lots of tool calls, you don't want them
350. 13:18trying to use their knowledge too much
351. 13:20in there. You want them just getting the
352. 13:22right stuff out for the right inputs for
353. 13:25the different tools, etc. And that's
354. 13:27exactly what's going on here with this
355. 13:29model. Okay, so to finish up, there are
356. 13:31a number of different versions that
357. 13:32they've got of this up there. The ones
358. 13:35that I showed in the video were the
359. 13:36standard one, the GG UF, and the G Spark
360. 13:38one. Definitely, this is worth checking
361. 13:41out if you're looking for a very, very
362. 13:44quick model that needs to deal with
363. 13:46perhaps a little bit more complicated
364. 13:48function calling and tool use than the
365. 13:51previous 1B model. You can see in here
366. 13:54that its strength has really been
367. 13:56trained and pushed like crazy for this
368. 13:59tool use. And while it also did well in
369. 14:01the math reasoning and some of the other
370. 14:03instruction following stuff that I gave
371. 14:05it, the way that I would suggest that
372. 14:06you use the model is for sub agents and
373. 14:09for where you need a very quick but
374. 14:11reasonably simple kind of model that can
375. 14:14handle a lot of different tools and can
376. 14:16deliver good arguments or the
377. 14:18appropriate information to pass into
378. 14:20those tools. So anyway, let me know in
379. 14:22the comments what you think of it. As
380. 14:24always, if you found the video useful,
381. 14:25please click like and subscribe, and I
382. 14:27will talk to you in the next video. Bye
383. 14:28for now.