Ripple 65B: A Chip Design Lab Just Beat Every US Open Model — Transcript

Full transcript

1. 0:00Okay, so if you look at the state of
2. 0:02open models at the top of the Aentic
3. 0:04coding benchmarks right now, they're
4. 0:06pretty big. In fact, most of them are
5. 0:09kind of huge. You're talking about 500
6. 0:11billion parameters, a trillion,
7. 0:13sometimes even multiple trillions. Now,
8. 0:15that's great if you've got a data
9. 0:16center, but it's not much use if you're
10. 0:18trying to run something locally on your
11. 0:19own box. Well, yesterday a lab called
12. 0:22Recursive released a 65 billion
13. 0:24parameter dense model trained from
14. 0:26scratch that they say beats every US
15. 0:29openweight model on terminal bench 2.1.
16. 0:32Now, I've actually got this running on
17. 0:33the ZGX Fury in my office at the moment,
18. 0:36hooked up to DSH, the Deep Seek harness.
19. 0:39So, I'm actually going to put it to the
20. 0:40test of how well this does for Aentic
21. 0:42coding. The other thing though that's
22. 0:44really interesting about this release is
23. 0:46that they're being very open about some
24. 0:48of the stats. So, the whole thing cost
25. 0:50$32 million and they've actually told us
26. 0:53how they actually did it. Okay, so who
27. 0:55are this group of people? Because I
28. 0:56hadn't heard of them before. A quick
29. 0:58note, this is Recursive with an I, not
30. 1:02to be confused with Recursive, Richard's
31. 1:04startup, which put out the Flint model
32. 1:06recently. That's a totally different
33. 1:08company. So, Recursive is a fairly new
34. 1:10lab, and their main business isn't
35. 1:12language models at all. It's chip
36. 1:14design. The founders are the team behind
37. 1:17Alpha Chip at Google. The reinforcement
38. 1:19learning research for laying out chips.
39. 1:22And the pitch for that company is sort
40. 1:23of closing the loop. Use AI to design
41. 1:26better chips. Use better chips to train
42. 1:28better AI. Certainly a story we're
43. 1:31hearing more and more often now. So why
44. 1:33would a chip design company pre-train an
45. 1:35LLM from scratch? Well, the answer is
46. 1:37because they needed one to power their
47. 1:39chip design platform and they wanted
48. 1:41control over the whole stack. The model,
49. 1:43the training setup, the servings, the
50. 1:45whole works there. Now, what they've
51. 1:47released is a generalpurpose version of
52. 1:49the model that runs inside that
53. 1:51platform. Obviously, they're fully
54. 1:53decked out AI chipm version is staying
55. 1:56proprietary. And for me, I don't really
56. 1:58mind about that. That kind of makes
57. 2:00sense with what their company is focused
58. 2:02on. And it's kind of cool that they've
59. 2:03released this model and all the
60. 2:05artifacts that they actually have. So,
61. 2:07clearly a chip design company putting
62. 2:09out a coding model isn't something that
63. 2:10we're seeing every week. But when you
64. 2:12read their blog post, it's one of the
65. 2:14more detailed writeups of how a model
66. 2:16was actually trained that we've seen
67. 2:18from any lab in quite a while. And
68. 2:20that's a big part of why I wanted to
69. 2:22make this video. All right, so let's
70. 2:23look at what actually came out. So the
71. 2:25model is Ripple 65b preview. It's a 65
72. 2:29billion parameter dense model, so not a
73. 2:31mixture of experts. It's pre-trained on
74. 2:3412 trillion tokens with support for up
75. 2:37to a million tokens of context. And
76. 2:39already there there are a number of
77. 2:40things that are really interesting.
78. 2:42While 12 trillion tokens is huge by any
79. 2:45sort of standard, it's certainly a lot
80. 2:47less than what we're seeing a lot of
81. 2:49other models actually being trained on.
82. 2:51They've also released the instruction
83. 2:53tune model, which is what I'm going to
84. 2:54be testing with DSH. They've released
85. 2:56that in a Vflat 16 version, an NVFP4
86. 3:00version, and a base model as well if you
87. 3:02wanted to do your own sort of fine
88. 3:04tuning from scratch. And I got to say
89. 3:06that that's pretty nice because a lot of
90. 3:08the labs are no longer giving you the
91. 3:10base model. And if you want to do your
92. 3:12own post training, that's really what
93. 3:13you kind of want there. All right.
94. 3:15Another cool thing is it's Apache 2.0.
95. 3:18You can use it commercially. You can
96. 3:19fine-tune it. You can pretty much do
97. 3:21whatever you want with this with no
98. 3:23custom license full of weird
99. 3:24restrictions in there. If you don't want
100. 3:26to run it yourself, it's going to be on
101. 3:27a bunch of different clouds including
102. 3:29together, Open Router. I think Fireworks
103. 3:32AI is also coming. Okay, so back to the
104. 3:34headline numbers. Well, on Terminal
105. 3:36Bench 2.1, they're reporting 80.52.
106. 3:41Reflections beam model, which is a 500
107. 3:44billion parameter model, scored 80.1.
108. 3:47The Metamuse Spark open model, which I
109. 3:49did a whole video about, that's 816
110. 3:52billion parameters, and that only scored
111. 3:5476.4. And there are a couple of others
112. 3:57further down on that link that come from
113. 3:59other labs and are huge models, but are
114. 4:01scoring less than this. So, a 65B model
115. 4:04edging out a 500B model, that's going to
116. 4:06get my attention pretty much every day
117. 4:08of the week. But here's the thing, and
118. 4:10you have to read the claim carefully.
119. 4:12It's a US openw weight state-of-the-art.
120. 4:14And that qualifier there is doing a lot
121. 4:16of work. But before I get into that, one
122. 4:18of the things I want to ask from you is
123. 4:20that my editors pointed out that the
124. 4:21majority of people watching our videos
125. 4:23are not subscribed. And nowadays with
126. 4:26YouTube's algorithm, it really is
127. 4:27basically a game of luck, if you're
128. 4:29going to see the next videos we make, if
129. 4:31you're not already subscribed to the
130. 4:32channel. So, as much as I hate to do the
131. 4:34YouTuber thing, if you find the videos
132. 4:35useful, please click like and subscribe
133. 4:38so the videos actually get in front of a
134. 4:40lot more people and that hopefully you
135. 4:41see them as they come out. All right, so
136. 4:43back to that qualifier. So, let's put it
137. 4:45into perspective. If you read the
138. 4:47announcement, you think that this is
139. 4:49almost the best open coding model out
140. 4:51there, and it's not, right? If you look
141. 4:53at the Chinese open models and the
142. 4:55numbers are quite a bit higher there.
143. 4:56You've got DC flash 4.1 which is
144. 4:59reporting 90.6 on the same benchmark.
145. 5:02You've got Xiaomi's MIMO 2.6 Pro report
146. 5:06at 89.9 and Kimmy K3 and GLM 5.3 are
147. 5:10both around 88. In fact, I think there
148. 5:13are roughly around 10 Chinese open
149. 5:15models sitting at least 5 to 10 points
150. 5:18above Ripple. Now, some people might say
151. 5:20that some of those models are actually
152. 5:22benchmarks and stuff like that. That may
153. 5:24be the case. And one of the things that
154. 5:26interests me the most about this model
155. 5:27is one, it is a very big dense model.
156. 5:30And two, that it kind of wasn't made
157. 5:32specifically just for agentic coding. It
158. 5:35was made specifically for another use
159. 5:37case and just being repurposed for this.
160. 5:40So while there are mostly big mixture of
161. 5:42experts models out there like DeepSeek
162. 5:45V4.1 Flash which is around 763 billion
163. 5:48total parameters the fairer comparison
164. 5:51for Ripple I think is the closest dense
165. 5:54model and that would be Quen 3.827B
166. 5:57also Apache 2.0 if you've watched
167. 5:59[clears throat] the channel you know
168. 6:00that I'm a big fan of that model and
169. 6:02that's scoring at 73. So, Ripple is a
170. 6:04bit over twice the size of that, but it
171. 6:06is 7 and a half points better on the
172. 6:09terminal bench. The other thing to keep
173. 6:11in mind here is that almost all of these
174. 6:13numbers that Ripples included are
175. 6:15self-reported using different harnesses.
176. 6:18When Valves ran Terminal Bench 2.1
177. 6:20independently, models like MIMO 2.6 Pro,
178. 6:23Quen 3.8 Max came back actually in the
179. 6:27high60s, well below what other vendors
180. 6:30had reported. So if you saw the video I
181. 6:32made about the deepseek harness there,
182. 6:34you would have heard my comments talking
183. 6:35about how everything is moving towards
184. 6:38harnesses for getting the talk
185. 6:39benchmarks, not just the model. So
186. 6:41generally the thing I would say here is
187. 6:43that perhaps this isn't the best model.
188. 6:45But what it is is the strongest small
189. 6:47dense model from a US lab with open
190. 6:50weights, a base model, a published
191. 6:52recipe, and something that's very
192. 6:54competitive with the Chinese models that
193. 6:56are already out there. So, if you're in
194. 6:58the market for any of those things, this
195. 7:00is definitely a model that you've got to
196. 7:02check out. All right. So, how did they
197. 7:04actually do this for $32 million? Well,
198. 7:06one of the things that I really like
199. 7:08here is that they're clear that the $32
200. 7:10million is the full R&D cost, not just
201. 7:14the final training run. So, while we've
202. 7:16seen some really good figures recently
203. 7:18from things like MIMO 2.6, talking about
204. 7:20$2.6 $6 million for the RL training run
205. 7:23and from perhaps some of the other labs.
206. 7:25This actually is covering all the
207. 7:27experiments, the data work, the evals,
208. 7:30the post training, all of it. And it's
209. 7:33interesting that they estimate it's
210. 7:34about a tenth of what the previous open
211. 7:37weight leader cost. Now, that's their
212. 7:40estimate based on reported compute
213. 7:41deals. So, you could take it with a
214. 7:43grain of salt, but I got to say it does
215. 7:45ring very true to other things that I've
216. 7:47heard as well. Anyway, there are three
217. 7:48things that stand out here. The first is
218. 7:51that they went for dense, not a mixture
219. 7:53of experts. Pretty much every other lab
220. 7:55lately has been going for thee models.
221. 7:58Ripple's argument is that a dense is
222. 8:00just simpler to train and simpler to
223. 8:02serve. They reported around 40% MFU
224. 8:05through pre-training and post- training.
225. 8:07And it's interesting that they didn't go
226. 8:09for any custom kernels or quantization
227. 8:11in here. For a small team, all of that
228. 8:14makes a lot of sense.
229. 8:16can give you cheaper inference per
230. 8:17token, but you often pay for it in the
231. 8:20infrastructure and complexity of
232. 8:22actually having to have all of those
233. 8:23experts and their weights in memory when
234. 8:26you're actually serving it. And it seems
235. 8:28that Ripple just decided that wasn't
236. 8:30worth it. So, the architecture itself is
237. 8:32thin and deep. It's got 80 layers, which
238. 8:35is kind of interesting there. They're
239. 8:36actually using more attention heads than
240. 8:38you would normally see. And to make up
241. 8:41for that, they use more sliding window
242. 8:43attention layers with a smaller model
243. 8:45and fewer global parameters. And I can't
244. 8:48help but wonder, is this a decision that
245. 8:50they made based on gossip that they've
246. 8:53learned from friends working on the
247. 8:55Gemini team, perhaps working on some of
248. 8:57the other frontier labs as well? On top
249. 8:59of those details, there's also a handful
250. 9:01of recent tricks that they've put in
251. 9:03there. So, they've got gated attention,
252. 9:05they've got no positional encoding on
253. 9:06global layers, and none of these are new
254. 9:09on their own. The interesting bit I
255. 9:11think is really how they chose them.
256. 9:13Which brings me to the second thing that
257. 9:14stands out here and that's lading. They
258. 9:17started from the OMO 3 model ladder from
259. 9:20AI2 which up until about a year ago were
260. 9:22the only ones that really sort of
261. 9:24published full open model weights,
262. 9:28training data sets, recipes of how they
263. 9:31were actually building their models. But
264. 9:33because of their funding, most of the
265. 9:34time their models were actually very
266. 9:36small. Now you can kind of think of this
267. 9:38ladder as a series of small models at
268. 9:41increasing sizes. So you can test a
269. 9:44change and see where it holds up to
270. 9:46scale. So they tested changes one at a
271. 9:49time. Some worked, some didn't. They
272. 9:51tried some variants of things like
273. 9:53hyperconnections that looked really good
274. 9:55at a small scale, but really did nothing
275. 9:57at 4 billion parameters or above. So
276. 9:59they dropped it. And I got to say this
277. 10:01is something that's really cool because
278. 10:03you don't actually see labs publishing
279. 10:05these kind of things that often. To give
280. 10:07you a little sidebar example, if you go
281. 10:09and look at the architecture on the
282. 10:11small Gemma models versus the bigger
283. 10:13Gemma models, you will see they are
284. 10:15different architectures. And the reason
285. 10:17is because what often works at a small
286. 10:20level doesn't work when you go up to a
287. 10:2212B, 26B, 31B size of a model. The third
288. 10:26thing that I found really interesting in
289. 10:27here is their post training. So they
290. 10:29didn't buy any post-training data. They
291. 10:31built their own synthetic data pipelines
292. 10:34and environments. And they kept adding
293. 10:36harder RL tasks as the model improved.
294. 10:39And they got that to the 80.5 with 1.4
295. 10:42million RL rollouts. And it's
296. 10:45interesting that they comment that that
297. 10:46was probably around 70 times fewer than
298. 10:49being used. So I think the story here
299. 10:52isn't some secret new architecture. It's
300. 10:54a lot of careful smallcale experiments
301. 10:58on only scaling up things that actually
302. 11:00held up. And the other cool thing is
303. 11:02that they've been upfront in that
304. 11:04they've been building on the work from
305. 11:06organizations like AI2, Prime Intellect,
306. 11:09Nvidia, and others to do it. And I think
307. 11:12this is one of the biggest challenges
308. 11:13that we're seeing at the moment as the
309. 11:16sort of big frontier proprietary model
310. 11:18labs stay fully closed, don't publish
311. 11:21any research, don't really give back
312. 11:24anything beyond just at best an openw
313. 11:27weights model which they don't tell you
314. 11:28how it was actually trained, etc. Makes
315. 11:30it really hard for academia and for
316. 11:33startups to find ways to actually make
317. 11:35better models and improve the whole
318. 11:38ecosystem as a whole. All right. So,
319. 11:40they claim that this can basically run
320. 11:41on a bunch of different hardware out
321. 11:43there. I'm sure we're going to see a
322. 11:44bunch of different quantizations and
323. 11:46stuff like that. Today, I'm going to be
324. 11:48running the NVFP4 version on the ZGX
325. 11:51Fury, which is a DGX station. I should
326. 11:53stress that this video is not sponsored
327. 11:55by them or by anyone. I just feel like
328. 11:57while I've got this machine on load, I
329. 11:59want to put it through its paces for as
330. 12:01many models as possible. Let's jump in
331. 12:03and have a look at both the model and
332. 12:04actually using it with the Deep Seek
333. 12:06harness. Okay, so jumping into the demo
334. 12:08here. I've already come through and run
335. 12:10the Aentic Pro tool calls and stuff like
336. 12:13that. All of that works as long as the
337. 12:15model is configured right and as long as
338. 12:18the serving setup is configured right.
339. 12:20So, initially I had some issues with SG
340. 12:22Lang. Once that was fixed, everything
341. 12:24was working really fine in here. So, the
342. 12:26complicated ones work and not
343. 12:28surprisingly all of the easier ones work
344. 12:31in here as well and they're coming
345. 12:33through very quickly which is really
346. 12:35cool here. So that's a full pass on all
347. 12:38the different tool calling, all the
348. 12:40different resistance to prompt injection
349. 12:42and tool baiting and things like that in
350. 12:44there as well. If we come up here and
351. 12:45just run the test for writing a long
352. 12:48essay in here, I think [laughter] this
353. 12:49is one of the things that I find
354. 12:51interesting about this model in general
355. 12:54is that because it seems to be
356. 12:56pre-trained on different data and
357. 12:59perhaps not just trained on the same
358. 13:01stack that everyone else has gone for,
359. 13:03it definitely has its own feel to it
360. 13:06more than you know some of the other
361. 13:08ones. So both for generating things like
362. 13:11new essays and stuff out or as in this
363. 13:14case where it's actually going through
364. 13:17and critiquing an essay out that it
365. 13:19generated itself. The Chinese models
366. 13:22will often generate out a really nice
367. 13:24essay and then you get it to fact check
368. 13:26that essay and it will claim that there
369. 13:27were lots and lots of errors in there.
370. 13:30This one finds a few errors in its own
371. 13:32output, but it definitely seems more to
372. 13:35the point. Now, to get there, it did use
373. 13:3810,000 tokens of thinking, you know,
374. 13:40going through there. So, that's
375. 13:42something you definitely want to take
376. 13:43into account as well. If we look at
377. 13:45things like the standard rag tests in
378. 13:48here, again, it likes to use a decent
379. 13:50amount of thinking. Not a huge amount
380. 13:52here, though, but the answers out are
381. 13:54very succinct and to the point. In this
382. 13:57case, we've got 1,700 tokens of
383. 13:59thinking, but our output is very sort of
384. 14:01clear. again for things like the logic
385. 14:03puzzles and stuff like that. The one
386. 14:05that I find really kind of interesting
387. 14:06is if we look at the SVG tests. So all
388. 14:11the kind of Chinese models and a lot of
389. 14:14the models nowadays seem to produce very
390. 14:16similar SVGs of the sort of pelican on a
391. 14:20bicycle or like when I do the red dragon
392. 14:22on a bicycle etc. And that really makes
393. 14:25me suspicious of like are they actually
394. 14:28just being distilled for doing these
395. 14:30kind of tasks or what sort of training
396. 14:32data are people using for these kind of
397. 14:34tasks. One of the things here that I
398. 14:36find and I've run this about four or
399. 14:38five times now is that it really does
400. 14:41produce out different looks. I won't say
401. 14:44that they're better. Some of them are
402. 14:46not necessarily better, but I do find
403. 14:49that the look to them are very sort of
404. 14:51original as opposed to what we've seen
405. 14:54for a lot of the other models. Okay, so
406. 14:56finally it's done. And if we come and
407. 14:58look at it, we can see that the look of
408. 15:00this one is very different. And when
409. 15:03I've rendered them out, they're
410. 15:05definitely not as good as some of the
411. 15:07Chinese models for this. My guess is
412. 15:09that this is not something that they've
413. 15:11prioritized for really at all in here.
414. 15:13The thing that I find really interesting
415. 15:15though is sometimes we get like almost a
416. 15:18monochrome version of the picture. They
417. 15:20just have a very different style than
418. 15:23what you see in the other models etc.
419. 15:26Okay. If we give it a design task like
420. 15:28the Dario's wellness retreat, we can see
421. 15:31that actually does an okay job. I
422. 15:33wouldn't say it's going to win awards
423. 15:35here for design, but compared to a lot
424. 15:38of the other things out there, this is
425. 15:40doing, you know, pretty good at being
426. 15:42able to have all the elements and stuff
427. 15:43in here. The thing I strongly suspect
428. 15:46with this is that it just was never
429. 15:48actually built for doing this kind of
430. 15:50task. We look at the thinking, we've got
431. 15:52good, clear thinking in there for
432. 15:54design, for doing things like the colors
433. 15:57and managing, you know, all of that. But
434. 15:59my guess is that if you wanted to use
435. 16:01this model for this kind of task, you
436. 16:03probably want to do either some more
437. 16:05continual pre-training or certainly a
438. 16:07bunch of SFT for the design kinds of
439. 16:11tasks in here. Let's jump into the deep
440. 16:13sea harness and try it out for different
441. 16:15agentic tasks. Okay, to set this up with
442. 16:18deep sea harness, we can basically just
443. 16:20come in and set up the model. I've got a
444. 16:23custom model in here. So this is where
445. 16:26I've basically just defined it. I've
446. 16:28passed in a base URL. I can basically
447. 16:31just fetch the available models there
448. 16:33and then it will show up. And then if I
449. 16:35want to use it in here, you can see that
450. 16:37I can basically just kick off a sort of
451. 16:40standard job. And you'll notice here
452. 16:42that okay, one, I'm getting pretty good
453. 16:44token speed going through this. And I've
454. 16:46got multiple ones going on at the same
455. 16:49time. As we go through, you can see I'm
456. 16:51using quite a bit of tokens. And also
457. 16:53I've got the cache set up here so that I
458. 16:57can see currently I've got the context
459. 16:59window set to I think 256k but I could
460. 17:03make that bigger if I wanted to. But
461. 17:05here this is basically showing me the
462. 17:07cache hit in here. And I can have it
463. 17:09doing multiple projects at the same time
464. 17:11just like I would with Ty Code or
465. 17:13something else like that. Now if we come
466. 17:15in and look at some of these. So this
467. 17:18was one of the standard prompts from
468. 17:19deepseek to basically just get it to do
469. 17:22you know things like the Eiffel Tower
470. 17:24and stuff like that. Not the best sort
471. 17:27of things out of here which is probably
472. 17:29more telling about that this model is
473. 17:31not necessarily being sort of RL for
474. 17:34this kind of task. You know the new
475. 17:36models that are being RL for sort of
476. 17:38Blender more for sort of 3JS kind of
477. 17:41tasks and stuff like that have a better
478. 17:43sense of this kind of thing. We can see
479. 17:45with some of these that, you know, okay,
480. 17:47we're getting very sort of basic renders
481. 17:50in here. But sure enough, it will keep
482. 17:52going and grind away on these tasks. And
483. 17:56you can see things like making different
484. 17:58sort of sidebar plugins and stuff like
485. 18:00that is going on. I'm burning sort of
486. 18:02millions of tokens going through this.
487. 18:05But there actually that's including the
488. 18:07cash hit there. So the cash hit, the
489. 18:09highest I think I've seen it is sort of
490. 18:11high70s. So it does seem to really help
491. 18:14when you can configure that. To actually
492. 18:16set that up, I'm using SG Lang for doing
493. 18:18the serving here. And that allows me to
494. 18:20configure all the different elements of
495. 18:22the model in here to basically try and
496. 18:25get the best results out for using it
497. 18:27with Deep Seek Harness at the moment.
498. 18:29You want to make sure that you do
499. 18:30configure it to actually support the
500. 18:32tool calls properly and stuff like that.
501. 18:34Otherwise, you'll find it doing a bunch
502. 18:36of different strange things in here. But
503. 18:39running it like this has no problems of
504. 18:41doing background jobs of doing, you
505. 18:43know, different things like this. And
506. 18:44don't forget this is 100% local in here.
507. 18:46We can see that it's basically doing the
508. 18:48thinking as it goes through and could
509. 18:50certainly be a decent model for working
510. 18:52with this kind of thing. Okay, so Ripple
511. 18:5565B, maybe it's not the best open coding
512. 18:58model out there. Perhaps there are some
513. 18:59huge models out there that are going to
514. 19:01beat it, but at 65 billion parameters
515. 19:04and being a dense model that's Apache
516. 19:062.0 O with the base model available, a
517. 19:10really useful write up that shows how it
518. 19:12was made. This really is a big deal from
519. 19:14a company that whole goal is not about
520. 19:17training open LLMs for people to use. So
521. 19:20I got to say a lot of respect to
522. 19:22recursive for actually making this model
523. 19:24open for being so open about what they
524. 19:27actually did here and for giving us a
525. 19:29really good license to be able to use
526. 19:30it. Also don't forget that this is the
527. 19:33preview release. So, they say that
528. 19:35they're scaling toward the frontier. So,
529. 19:39we may even see something that's a lot
530. 19:40better from them over the next couple of
531. 19:42months. As always, I've put the links to
532. 19:44the blog post and the Hugging Face
533. 19:46models in the description. Let me know
534. 19:48in the comments what you think, what
535. 19:49you're interested to try this model out
536. 19:51on. If you do try it, let me know how it
537. 19:53actually stacks up for you. Anyway, as
538. 19:55always, if you found the video useful,
539. 19:57please click like and subscribe, and I
540. 19:59will talk to you in the next video. Bye
541. 20:00for now.