Open Jev Models Are Here!! — Transcript

Full transcript

1. 0:00Okay, in my last video, I talked about
2. 0:02Jev. And that project really has taken
3. 0:05off massively over the past few days.
4. 0:07And like many things, I guess, in AI,
5. 0:09you've got the extremes of people who
6. 0:12absolutely love it and think it's the
7. 0:14best thing ever. And on the other
8. 0:16extent, people saying that this is
9. 0:17nothing special, nothing new. But
10. 0:19really, in this video, I want to talk
11. 0:21about what if you could do all of this
12. 0:22locally? If you could run the weights
13. 0:24locally, if the weights were open
14. 0:25models, etc. So, in this video, I'm
15. 0:28going to look at five possibly six
16. 0:30versions of this idea that are open
17. 0:33projects. And I've got to say, the
18. 0:35amount of open projects around this kind
19. 0:37of thing has just exploded over the past
20. 0:40day or so. Yesterday, when I was looking
21. 0:42at this, there were like a couple of
22. 0:43these things. This morning, going
23. 0:45through it, there's well over 20, maybe
24. 0:47even 30 different projects out there of
25. 0:50people trying to replicate this model.
26. 0:52And some of them are actually getting
27. 0:53pretty close results to Jev. And one of
28. 0:56the things that I noticed researching
29. 0:57this, and shout out to Susan Jang for
30. 0:59pointing this out, that part of the
31. 1:01terms of service of actually using the
32. 1:03Jev model are that you won't run any
33. 1:05benchmarks on it to compare it to other
34. 1:08models, which is kind of insane in
35. 1:10itself. But anyway, let's talk about
36. 1:12what actually makes Jev special. So,
37. 1:14really, Jev is a universal classifier
38. 1:16with perhaps a little bit of regression
39. 1:17thrown in there. And of course,
40. 1:19classification is not new. It's been
41. 1:21around a long time. We've had things
42. 1:23like the BERT models, more recently the
43. 1:26modern BERT models, and even things like
44. 1:28text CNNs can be really good and really
45. 1:30fast for text classification. But
46. 1:33generally, the challenge with most of
47. 1:35those is that each one needs to be
48. 1:37trained or fine-tuned on your specific
49. 1:40use case. So, you change your labels,
50. 1:42you're basically having to train again.
51. 1:44And that's something that I've done
52. 1:45plenty of times myself over the years.
53. 1:48When I was training simple NLP models to
54. 1:50be able to do different kinds of text
55. 1:52classification for my previous startup,
56. 1:54where we were constantly looking at
57. 1:55reviews and trying to predict sentiment,
58. 1:58trying to predict what people were
59. 1:59interested in, etc. Now, there actually
60. 2:01have been a couple of different sort of
61. 2:04attempts at creating a universal
62. 2:06classifier with no training. Even going
63. 2:08back to sort of 2019, there were papers
64. 2:10that re-framed classification as
65. 2:12entailment. And it turned out that
66. 2:14entailment tasks were something that the
67. 2:16BERT models could actually do quite
68. 2:17well. So, for example, there are
69. 2:19different kinds of entailment, but I'll
70. 2:20give you a really simple example is
71. 2:23where if you've got two sentences, is
72. 2:25the second sentence random and not
73. 2:28related to the first sentence, or is it
74. 2:30related to the first sentence? If I have
75. 2:32one sentence about pizza, and then the
76. 2:35next sentence is the boy loves pizza, we
77. 2:36would say that that entails. And you can
78. 2:38actually sort of think about that those
79. 2:40kinds of things could be trained for a
80. 2:42more generalized sort of yes-no kind of
81. 2:44thing, which is one of the tasks that
82. 2:46Jev is actually very good at. So, the
83. 2:47question was, "Well, why didn't that
84. 2:49eventually become Jev 7 years ago or
85. 2:52something like that?" Well, really the
86. 2:53models back then just didn't really have
87. 2:55enough world knowledge to be able to
88. 2:57generalize to all the different things.
89. 3:00Clearly, whatever the model that Jev is
90. 3:01using, it has enough sort of world
91. 3:04knowledge in its weights that it is able
92. 3:06to generalize to lots of different
93. 3:07ideas. And I think where Jev is really
94. 3:10kind of interesting is that it's caused
95. 3:12a lot of people to start thinking about
96. 3:14what can we do just with really fast
97. 3:17classification. And what kind of tasks
98. 3:20can be offloaded away from a reasoning
99. 3:22LLM to things that can be just done with
100. 3:25simple choices or simple sort of yes-no
101. 3:27answers. Now, in my last video, I said
102. 3:30that we'd probably see open versions of
103. 3:32this. And sure enough, like within 24
104. 3:34hours, we saw a whole bunch of these
105. 3:37come out. And as I'm recording this,
106. 3:38like I said earlier, there's probably at
107. 3:40least 20-plus, maybe even 30-plus
108. 3:43different open implementations of this
109. 3:45idea. So, how close do these open
110. 3:47implementations actually get? Well,
111. 3:49someone's already built a benchmark for
112. 3:51this called Jev bench, and it scores
113. 3:54these systems on intelligence,
114. 3:56calibration, speed, and cost. And you
115. 3:59can see looking at this on the overall
116. 4:01score, Jev is at 75.3, and right behind
117. 4:05it is a project called Sem If. So, I
118. 4:07think this was originally called Open
119. 4:09Jev. Now, this project is basically a
120. 4:11frozen Qwen 3.5 4B model with no
121. 4:15training at all. Then we've got a
122. 4:17diffusion model called D Jev at 74.3,
123. 4:20followed by Leila, and then followed by
124. 4:22a whole bunch of different models
125. 4:23scoring in the 60s. And this leaderboard
126. 4:26that we're looking here is actually a
127. 4:27waiting that they're using for Jev
128. 4:29bench, but you could actually emphasize
129. 4:30things like accuracy, where D Jev
130. 4:32actually goes up. And the big thing here
131. 4:34is that all of these you've got some
132. 4:36form of trade-off between speed, cost,
133. 4:39and accuracy. If we look at things like
134. 4:41GPT 5.6 Luna here, we can see that its
135. 4:44intelligence level is much higher than
136. 4:47the other models. So, in that sense, the
137. 4:49system 2 model still wins for different
138. 4:52hard tasks, but it's costing roughly six
139. 4:54times as much as Jev, and is much, much
140. 4:57slower here. Let's look at a few of
141. 4:58these projects, see what differentiates
142. 5:00them, and see how you could actually run
143. 5:02these yourself. Okay, so what I've done
144. 5:04here is just build a demo with six of
145. 5:08these. So, all six of these are running
146. 5:10on an Nvidia RTX Pro 6000 Blackwell, and
147. 5:14a big thanks to Dell for sponsoring the
148. 5:16compute here. This is running on a Dell
149. 5:18T2 Pro Max workstation here. So,
150. 5:22basically what I've done is for all of
151. 5:23these is just pull the repos, set them
152. 5:25up, put them on the GPU, and we can
153. 5:27actually test them out. Okay, so the
154. 5:29first one here is Sem If, which was
155. 5:31originally called Open Jev. So, the idea
156. 5:34here is pretty simple. You take a normal
157. 5:36open model, you put the question and the
158. 5:38options in the prompt, and then rather
159. 5:40than letting it generate an answer, you
160. 5:42look at the logits for the option tokens
161. 5:44that come out, and you basically softmax
162. 5:46over those. And if you remember in the
163. 5:48last video, my guess was that Jeb was
164. 5:50basically using the pre-fill stage and
165. 5:54then doing prediction off that. And it
166. 5:55seems that many people converged on that
167. 5:57same idea. So, basically you process the
168. 5:59prompt once, read off the scores, and
169. 6:01answer the tokens. So, you can see here,
170. 6:03we're going to put this in. If we run
171. 6:06both, we can see, okay, the generated
172. 6:08JSON that we get back, and then the
173. 6:11actual just scores that we get back
174. 6:13here. So, if we want to we can just
175. 6:15generate the scores out and then see,
176. 6:17okay, the probability distribution
177. 6:19that's coming off the softmax there. And
178. 6:21this allows you to basically do a lot of
179. 6:23different sort of things where you've
180. 6:24got the yes-no questions, where you've
181. 6:27got some kind of policy, etc. And you
182. 6:30can see that, okay, if I change that to
183. 6:32basically only modify some things, now
184. 6:35you see, okay, it's definitely getting
185. 6:37confused, right? Now, if we totally
186. 6:39change it, so we're asking it to
187. 6:41actually make a change, you can see now
188. 6:43it really understands that, okay, it's
189. 6:45required there. So, this is the typical
190. 6:47kind of use that we would use Jeb for.
191. 6:49And that actually brings us to Bespoke
192. 6:51Nimble. It's the same sort of readout
193. 6:53trick, but they've basically put a LoRA
194. 6:56on a Qwen 3.5 9B model and built the
195. 7:00whole thing with under 3,000 training
196. 7:01examples. And just to make it clear,
197. 7:03they pointed out that they didn't
198. 7:05distill from Jeb. The really interesting
199. 7:07thing here is actually how they made the
200. 7:08data. They call it contrastive data
201. 7:11curation. You write two examples that
202. 7:13are almost identical, and you change one
203. 7:16fact so that the right answer flips. So,
204. 7:19their example is a refund rule. Only
205. 7:21Mira can authorize refunds on this
206. 7:23account. In one version, the refund was
207. 7:25signed by Mira, so it's authorized. In
208. 7:27the other one, it was signed by Noah.
209. 7:29Everything else is exactly the same. And
210. 7:31what that teaches the model is which
211. 7:33piece of evidence should actually change
212. 7:35the decision. Now, on their own holdout
213. 7:38set, this is getting around 90% where
214. 7:40their base model gets around 66, and Jev
215. 7:42is getting around 93. But actually on
216. 7:45the Jev bench hard tier, it was only
217. 7:46getting around 44. So again, this is
218. 7:48strong on what it's been trained on,
219. 7:51which totally makes sense. But the other
220. 7:52thing I would say is the fact that
221. 7:53they've only sort of done some training
222. 7:55on 3,000 examples. Probably we shouldn't
223. 7:58expect it to generalize that much. And
224. 8:00just to make it clear, they pointed out
225. 8:02that they didn't distill from Jev. The
226. 8:04really interesting thing here is
227. 8:05actually how they made the data. They
228. 8:07call it contrastive data curation. You
229. 8:09write two examples that are almost
230. 8:12identical and you change one fact so
231. 8:15that the right answer flips. So their
232. 8:17example is a refund rule. Only Mira can
233. 8:19authorize refunds on this account. In
234. 8:21one version, the refund was signed by
235. 8:23Mira, so it's authorized. In the other
236. 8:25one, it was signed by Noah. Everything
237. 8:27else is exactly the same. And what that
238. 8:29teaches the model is which piece of
239. 8:31evidence should actually change the
240. 8:33decision. Now, on their own holdout set,
241. 8:36this is getting around 90% where their
242. 8:38base model gets around 66 and Jev is
243. 8:40getting around 93. But actually on the
244. 8:42Jev bench hard tier, it was only getting
245. 8:44around 44. So again, this is strong on
246. 8:47what it's been trained on, which totally
247. 8:49makes sense. But the other thing I would
248. 8:50say is the fact that they've only sort
249. 8:52of done some training on 3,000 examples.
250. 8:55Probably we shouldn't expect it to
251. 8:57generalize that much. All right, so the
252. 8:59demo here really shows how they sort of
253. 9:01flipped the facts for making the
254. 9:04training data in here. So you can see
255. 9:06here that okay, we've got this signed by
256. 9:09Mira in here. If we run the score, is it
257. 9:11authorized? Yes, cuz remember they only
258. 9:13should be authorized if they're signed
259. 9:14by Mira. Okay, we've got Mira is the
260. 9:17only one to authorize account 42. This
261. 9:20was signed by Mira, so it should be
262. 9:22true, right? It should process the
263. 9:23refund. And we can see that okay, we've
264. 9:25got our nice JSON response back there.
265. 9:28If we flip the fact here, now you can
266. 9:31see we've actually changed it to it was
267. 9:33signed by Noah, right? And you can see
268. 9:36it comes back false there. If I change
269. 9:38it to Sam, it should still be false,
270. 9:40right? Cuz I've basically done that. And
271. 9:42remember, this is done with a Laura
272. 9:44adapter on the 9B model. We can see that
273. 9:46actually Qwen 3.5 seems to be doing
274. 9:48quite well with this question at least,
275. 9:51both with the Laura on and with the
276. 9:53Laura off. Okay, if we look at this
277. 9:55second one here, we can see Priya bought
278. 9:58a conference ticket. She needs basically
279. 10:00pre-approval from the manager. Priya's
280. 10:02manager sent a written approval for the
281. 10:05ticket on March 1. Actually, on this
282. 10:07one, it should be true, but we can see
283. 10:09in this case that the adapter is
284. 10:10actually getting it wrong. The normal
285. 10:13Qwen model gets it right for this part,
286. 10:15but it doesn't get it right for the
287. 10:17second part. Big thing here, it seems,
288. 10:19is that you can see that the model has
289. 10:21jumped on pre-approval rather than just
290. 10:23approval. So, if we change it to
291. 10:25pre-approval, both of them get approved
292. 10:28now. It's basically both scored as true.
293. 10:31The untrained model gets it right, but
294. 10:33only just, and the other one is still
295. 10:35getting it wrong. And you see, to
296. 10:37actually get it to be both right, I
297. 10:39actually had to change this to
298. 10:41pre-approval for the conference ticket
299. 10:43before she bought. And then, straight
300. 10:45away, both of them can get it right in
301. 10:47here. So, these things can be a little
302. 10:49bit hit and miss here. Okay, next up is
303. 10:51the decider from Mapika. So, this is
304. 10:54built on the Qwen 3.5
305. 10:57to base model. So, this is actually a
306. 10:58much smaller model. This one's a little
307. 11:00bit different in that, rather than
308. 11:02reading the sort of normal output
309. 11:04logits, it reads the hidden state at
310. 11:06each answer slot and projects that onto
311. 11:09the options. So, it does the same three
312. 11:11types as Jev, so you've got choice,
313. 11:14score, null. It can take up to a 32K
314. 11:17context window. And the cool thing with
315. 11:19this one is it actually serves back the
316. 11:22same wire format as the Type Safe API.
317. 11:25So, you can see here, we're basically
318. 11:26putting in I was charged twice for an
319. 11:28order last night, and And duplicate is
320. 11:30still pending. I need this fixed before
321. 11:33my card closes this Friday, etc. You can
322. 11:35see that what it's doing is we've got
323. 11:37choice for basically multiple criteria
324. 11:40being billing, technical, sales, other.
325. 11:42Urgency is in there. Escalate, whether
326. 11:46this should sort of escalate up. And
327. 11:48then actually measuring the sort of
328. 11:49frustration level. So, if we run it, you
329. 11:51can see one, it's actually very quick.
330. 11:54And I find actually if I run it a few
331. 11:55times, yeah, it gets actually quicker
332. 11:57like that. Here you can see sure enough
333. 11:59it's got that was billing, that should
334. 12:01be handled this week, the urgency. It's
335. 12:03probably should be handled today, but
336. 12:06cuz the person's saying by the end of
337. 12:07the week. And you can see it's really
338. 12:08not sure too much whether it's this
339. 12:10week, whether it's today. Should it be
340. 12:13escalated? Yes. And then a frustration
341. 12:15score is sort of halfway between very
342. 12:18frustrated and frustrated. So, it sided
343. 12:20on very frustrated here. The cool thing
344. 12:22with this one is just the speed. So, if
345. 12:25we run like 10 of them, you can see
346. 12:26we're averaging 33 milliseconds per
347. 12:29answer here. Now, one of the things I
348. 12:31notice about this model is that it's
349. 12:32often on the fence. So, with Jeb in my
350. 12:36testing, it was pretty decisive in its
351. 12:39sort of confidence about something being
352. 12:42one thing or the other thing. With this
353. 12:44model, it's very indecisive and it tends
354. 12:47to be sort of in the middle for a lot of
355. 12:49the different tasks. And while for
356. 12:51certain things it does get it totally
357. 12:52right, I got to say one of the things
358. 12:54that is nice about Jeb is just how the
359. 12:56majority of classifications that I made
360. 12:58in it seem to have a very high
361. 13:00confidence. It was confident of its
362. 13:02choices, not just a model hedging in
363. 13:04there. Okay, so the other three that
364. 13:06I've put in here, the first is Alex
365. 13:08Wartega's OpenJeb. And this one really
366. 13:11builds on the whole entailment idea from
367. 13:13before. One of the cool things that this
368. 13:15is doing here though is that we can
369. 13:17actually put images into it quite
370. 13:19easily. So, you can see this is an image
371. 13:22going in and we can run these through
372. 13:24and get scores for all of these. So, you
373. 13:27can see sure enough the payment failed.
374. 13:29So, basically got each of these
375. 13:31statements about this here. And it can
376. 13:33go through and give us the sort of true
377. 13:35or false for each of those. And you can
378. 13:37see I can come in and just add another
379. 13:39statement there. And we can see that
380. 13:41okay, the try button is green. No, it's
381. 13:43not. It's blue, right? And sure enough
382. 13:45it changes it there. If I change it to
383. 13:47actually asking that the try button is
384. 13:49blue, now it's saying yes. And you can
385. 13:51use this idea here on images as well.
386. 13:54And the cool thing for this one is
387. 13:55they've actually got like a playing
388. 13:57Doom, right? So, you can see that if you
389. 13:59pass in these things, you give it the
390. 14:01choice of turning left, turning right,
391. 14:03shooting, whatever it actually is, the
392. 14:06model can actually start to play some of
393. 14:08this stuff. Another demo that's really
394. 14:10cool in here is the whole sort of
395. 14:11Minecraft. So, if you basically have it
396. 14:15move around, they can get it to actually
397. 14:18make different kinds of decisions via
398. 14:21choices, right? Based on what's there.
399. 14:24So, I've got to say that this one I
400. 14:25think is really kind of cool of how it's
401. 14:27able to do it with images as well. Next
402. 14:30up is Diffusion Gemma. So, this is a
403. 14:31totally different class of model. Rather
404. 14:33than just doing the auto regressive
405. 14:35thing of calculating the prefill etc. on
406. 14:37the attention heads, this is basically
407. 14:39doing it all through diffusion. So, it
408. 14:42runs a a denoising step in here, and
409. 14:44then it's reading the log probs of those
410. 14:46slots, and just benefits from the fact
411. 14:48that it's just so quick in here. So, you
412. 14:51can see we've got a whole bunch of
413. 14:52different tickets here, and we're trying
414. 14:55to basically work out which team should
415. 14:56handle this ticket. So, this is the
416. 14:58choice kind of thing in here. We've got
417. 15:00the null of does the customer need a
418. 15:02reply within the hour? We've got how is
419. 15:05the customer feeling? And then we've got
420. 15:06some choices there. So, with this we can
421. 15:09run just one ticket at a time. You can
422. 15:12see the speed on this is kind of insane,
423. 15:13which is definitely true with Diffusion
424. 15:15Gemma. I never got around to making a
425. 15:17video about this, but in my testing, you
426. 15:19could get super fast tokens from this.
427. 15:22The only thing I felt with this model
428. 15:24was that it had kind of been under
429. 15:26trained compared to say the Gemma 4 31B
430. 15:31or the Quen models or things around that
431. 15:32size. But speed wise, this thing's
432. 15:35insane. So, you can see this is how many
433. 15:37decisions it can make per second. If I
434. 15:40just refresh it and give it all the
435. 15:42tickets here, we've got 16 tickets. I
436. 15:44run all of them. You can see, boom, it's
437. 15:47just done, right? So, in that case it
438. 15:49was basically able to fly through these.
439. 15:52And you can see it's done a pretty good
440. 15:53job of some of these. I don't know, like
441. 15:55this one I probably think I would maybe
442. 15:57say it's urgent. I don't know. I guess
443. 15:58it really depends on how you want to
444. 16:00evaluate these. And just passing more of
445. 16:02a prompt in is probably going to
446. 16:04influence these things as well. Because
447. 16:06if sure enough things like this, it
448. 16:07knows that it's technical, it knows that
449. 16:09the user is annoyed because of the
450. 16:11latency times coming back here. I can
451. 16:13see that we can actually get really high
452. 16:15concurrency here as we go through this.
453. 16:17So, this one is definitely the one to
454. 16:20try. If you can run this model and you
455. 16:22want to do sort of insane amounts of
456. 16:24questions on this, it'd be very
457. 16:26interesting to see the Gemma diffusion
458. 16:28doing some of the other tasks in here.
459. 16:30Now, you should remember of course that
460. 16:31this is not a small model. This is a 26B
461. 16:34model here. And in their test they were
462. 16:35basically running it on a DGX Spark.
463. 16:38Here I'm running it on the RTX Pro 6000.
464. 16:40So, lastly one that I would say is kind
465. 16:42of like a bonus in here is Nano Jev. So,
466. 16:45this is a tiny 600 million parameter
467. 16:47model and you can see that here we're
468. 16:50basically giving it a game of snake
469. 16:53where it's given choices. So, you can
470. 16:55see that if I just play this, you can
471. 16:58see that it's got these choices whether
472. 17:00it goes east, west, north, south, and
473. 17:03then it's given this sort of grid format
474. 17:05of where things are so that it can
475. 17:08actually do it. So, if I speed it up to
476. 17:10fast, you can see that actually it does
477. 17:12a really good job at being able to to
478. 17:15this game. So, it just shows you I
479. 17:17think, this one that if you can
480. 17:19repurpose ideas to just be simple
481. 17:21classification tasks, like this one
482. 17:23where it's just basically given a bunch
483. 17:25of different options in here, that can
484. 17:28actually let this model do a whole bunch
485. 17:29of different things going through. You
486. 17:31can see at the end it basically was
487. 17:33running out of options and was going to
488. 17:34basically hit its tail no matter what.
489. 17:37Okay, so one more that I wanted to add
490. 17:38in here is Leila.
491. 17:41So, if you remember back earlier I
492. 17:42talked about the whole idea of sort of
493. 17:44BERT models as being the old way to do
494. 17:46this. Well, Leila actually is a BERT
495. 17:49model. It's built on modern BERT, so
496. 17:51this is actually a sort of much newer
497. 17:53version, not the original 2018 version
498. 17:55of BERT. It's actually built on modern
499. 17:57BERT large, so it's about 420 million
500. 18:00parameters, which remembering back to
501. 18:022018, that was huge then, but now it's
502. 18:05just absolutely tiny. And the cool thing
503. 18:07is this one can do multilingual as well.
504. 18:09So, I think it covers around about 100
505. 18:10different languages in here. It takes
506. 18:12the three same types of questions, so
507. 18:15got choice, score, null, and then it
508. 18:17answers them all in a single forward
509. 18:19pass. So, the cool thing here is they've
510. 18:21actually put all of this into a Google
511. 18:23Colab, so you can come in here and
512. 18:25actually try it out yourself. And
513. 18:28because the model is actually so small,
514. 18:29it's actually just running on its T4 GPU
515. 18:31in here. Along with the actual Leila
516. 18:33model, there's also a really nice
517. 18:35write-up of a person who actually
518. 18:37created this over a year ago, and in
519. 18:39many ways is quite understandably bitter
520. 18:41about no one paying attention to his
521. 18:43research, yet a very well-funded lab
522. 18:46gets all the credit for these kinds of
523. 18:48things. And while I do think Leila is
524. 18:50quite impressive, when we change the JAB
525. 18:52bench weightings to just taking account
526. 18:54to accuracy here, we can see that JAB is
527. 18:57really competing with other large
528. 18:58language models that a lot of these
529. 19:01models below it, and especially
530. 19:03something like Leila, are actually doing
531. 19:04a lot worse, most likely because they
532. 19:07just weren't trained on the sort of
533. 19:09generalization. And I will say that a
534. 19:12well-funded lab should be expected to be
535. 19:14able to train a model that's going to
536. 19:15generalize a lot better than someone
537. 19:17who's just making a fine-tuned version
538. 19:19of an older book model, etc. All right,
539. 19:22so which one of these should you
540. 19:23actually use? If you want to do this
541. 19:25today with no training, you could do the
542. 19:27Sim if style readout on a model that
543. 19:29you're already using already. If you
544. 19:31want a small self-contained server that
545. 19:34speaks the Jeb format, then probably
546. 19:36Decider might be worth trying out. And
547. 19:38then especially if you've got some kind
548. 19:40of narrow policy decision, take Nimbles
549. 19:44recipe and run it on your own data. The
550. 19:46whole sort of idea of swapping something
551. 19:48in, swapping something out to actually
552. 19:50do the contrastive thing. And lastly, if
553. 19:53you need images, check out what Tegas
554. 19:55one or the actual Decider vision one
555. 19:57there. The challenge is going to be that
556. 19:59for really hard stuff, the sort of
557. 20:01multi-hop stuff, the date arithmetic,
558. 20:04really none of these are working that
559. 20:06great. And so what might be your best
560. 20:08option there is to actually sort of
561. 20:09cascade those things. Let a fast model
562. 20:12take everything first, and then when
563. 20:14it's not very confident in something,
564. 20:16pass that one up to a reasoning model on
565. 20:18a low or sort of medium effort. And I
566. 20:20think that can be a really powerful
567. 20:22trick of sort of being able to get the
568. 20:24benefits of prices and speed of system
569. 20:27one on most of what you're doing, and
570. 20:30then only go to system two where you
571. 20:32really need it. Overall, I would say the
572. 20:34open versions are closer than I expected
573. 20:36on sort of the easy and standard stuff,
574. 20:38but there's definitely a real gap on
575. 20:40some of the hard stuff. So as always,
576. 20:42I'll put the links to all of these in
577. 20:43the description. You can go and check
578. 20:44them out. If you haven't seen the
579. 20:46original Jeb video I made, check that
580. 20:48out over there as well. And then I'm
581. 20:49really conscious that as I've been
582. 20:51recording this, there are probably more
583. 20:52of these things coming online all the
584. 20:54time. So let me know in the comments if
585. 20:56you found some that worked really well
586. 20:58for you, or if any of these are breaking
587. 21:00on certain things that you can't get to
588. 21:02work at all. And I mean, looking at this
589. 21:04myself, I'm I'm to think about the mini
590. 21:06CPM 5 2B model could actually be a
591. 21:09really nice model to try for some of
592. 21:12these tasks and for doing your own sort
593. 21:15of Laura's and stuff for this kind of
594. 21:16thing. So, I'm going to probably have a
595. 21:18play with that. If people are really
596. 21:19interested in that, let me know as well
597. 21:21and maybe it's something that I look at
598. 21:22doing a video about over the next few
599. 21:24days. Anyway, as always, if you found
600. 21:26the video useful, please click like and
601. 21:27subscribe and I will talk to you in the
602. 21:29next video. Bye for now.