Jev - The Ultimate Classification Model? — Transcript

Full transcript

1. 0:00Okay, so if you pretty much look at what
2. 0:02every Frontier lab has been doing over
3. 0:04the past 2 years, it's all been in one
4. 0:07direction. They've all been focused on
5. 0:10reasoning, and to get that reasoning,
6. 0:13mostly they've been focused on longer
7. 0:14chains of thought and thinking budgets.
8. 0:16And while those models are great,
9. 0:18they'll have you sit there for multiple
10. 0:20minutes before they even give you an
11. 0:21answer back. And of course, if you
12. 0:23wanted to see that chain of thought, the
13. 0:25Frontier labs are not going to let you
14. 0:26see it even though you're paying for it.
15. 0:28So if you read Daniel Kahneman's book,
16. 0:30Thinking Fast and Slow, you know that
17. 0:33all this kind of reasoning stuff is
18. 0:35system two thinking. It's slow,
19. 0:37deliberate, effortful, whereas system
20. 0:40one on the other hand is fast,
21. 0:41intuitive, sort of like a gut call that
22. 0:44you make really quickly without any
23. 0:46deliberation. And here's where the
24. 0:47subject of today's video, I think, is
25. 0:49really interesting. Most decisions that
26. 0:51actually sit inside of software aren't
27. 0:53system two problems. They're often just
28. 0:55really simple classification problems.
29. 0:57What kind of support ticket is this? Is
30. 1:00this message urgent? Did an agent's
31. 1:02output actually break a rule?
32. 1:04You shouldn't even need 30 seconds for
33. 1:07that kind of thinking, let alone the
34. 1:09sort of minutes that some of the models
35. 1:11end up taking. And while it's great that
36. 1:13you can ask one of these reasoning
37. 1:14models for a one-word answer, and then
38. 1:17wrap it in JSON, you're paying a huge
39. 1:19latency cost for generating all that
40. 1:22text before you get your one-word label
41. 1:24back. And so this week, a new lab that's
42. 1:26just come out of stealth has gone the
43. 1:28complete opposite way of the reasoning
44. 1:30models. They've put the model that you
45. 1:32can't chat with at all, let alone have
46. 1:35it reason and deliberate over things for
47. 1:36a long time. It just doesn't generate
48. 1:39text in the standard auto regressive
49. 1:40way. And the funny thing actually
50. 1:42enforcing that is that the output tokens
51. 1:45of this model are actually free. But
52. 1:47actually working out why those tokens
53. 1:49are free might be the thing that
54. 1:51actually tells us a lot about how this
55. 1:53thing is built. All right, so the
56. 1:54company is called Type-Safe AI, and the
57. 1:56model is called Jev. So, the founder is
58. 1:59Diogo Almeida and actually he was at
59. 2:01OpenAI for quite a while before and
60. 2:03funny enough was one of the top authors
61. 2:05on the InstructGPT paper. That was the
62. 2:08paper that actually led to the whole
63. 2:09sort of instruction tuning of models,
64. 2:11which has just progressed onwards from
65. 2:14that to the reasoning models that we see
66. 2:16today. So, if you don't know that paper,
67. 2:18basically took a raw language model,
68. 2:20basically like what we would call a base
69. 2:22model now, and actually made it follow
70. 2:24instructions. And out of all that
71. 2:27research came ChatGPT. So, this is
72. 2:29someone who clearly helped make models
73. 2:31good at talking to people. And the
74. 2:33question that he says that's been
75. 2:35bugging him for the last four years is
76. 2:37that as models have become superhuman at
77. 2:39chat, where's all the automation here?
78. 2:41And interestingly, he's saying that chat
79. 2:44is the wrong interface for software.
80. 2:46Software doesn't want a paragraph, it
81. 2:48wants a value that it can basically use
82. 2:50straight away. Now, Diogo and the rest
83. 2:52of the team at TypeSafe AI have spent
84. 2:54the last two years in stealth working on
85. 2:57this model called Jev. All right, so
86. 2:59what is Jev? So, they specifically refer
87. 3:02to this as a system one model and
88. 3:04probably the easiest way of looking at
89. 3:06this is it's kind of like a function
90. 3:08call. You pass in two things. The first
91. 3:11is what they call the state, which is
92. 3:13just your unstructured text or data and
93. 3:15that could be anything that you want
94. 3:16classified, could be a support ticket,
95. 3:19could be an agent trace, it could be a
96. 3:21log file, etc. And then the second thing
97. 3:23that you pass in is a set of typed
98. 3:25questions. And there are only three
99. 3:27kinds of questions that you can ask
100. 3:29here. There's choice, where you give it
101. 3:31a list of options and it picks one.
102. 3:34There's score, where it rates something
103. 3:36on a scale that you defined. And then
104. 3:38there's one called null, which is a yes
105. 3:40or no question, where it basically comes
106. 3:42back with a probability that the answer
107. 3:45is yes. So, if you look at what comes
108. 3:47back here, there's no text to parse. For
109. 3:49the the question, you get the option it
110. 3:51picked, and also a probability of every
111. 3:54other option. And I think this is really
112. 3:56kind of interesting because this is one
113. 3:57of the things that a lot of us have kind
114. 3:59of been trained to fake with JSON, where
115. 4:02we ask models to do stuff and return
116. 4:04JSON output. But in many ways, the
117. 4:06number that you're getting back there is
118. 4:08just generated text. And if you look at
119. 4:10things as LLM as a judge, you'll often
120. 4:13see that that text skews in a very
121. 4:15certain way. I.e., it's just a model
122. 4:18writing the character 0.9,
123. 4:21rather than actually a real
124. 4:22representation of probability. The way
125. 4:24they want you to use this model is to
126. 4:26think about it as being a smart if
127. 4:28statement. You don't ask one big fuzzy
128. 4:31question like rate this startup pitch.
129. 4:33You ask small gut questions in there.
130. 4:36What's the feasibility of this? What
131. 4:38type of market are they going after? And
132. 4:40of course, if you're asking a choice
133. 4:42question, you have to actually give it
134. 4:43the options to choose from. Then the
135. 4:46cool thing is that you can actually
136. 4:47combine all of those in normal code. So,
137. 4:50when something changes, you can just
138. 4:52change a number in your code. You don't
139. 4:53need to go and tweak any prompts or
140. 4:55anything. All right. So, what I've done
141. 4:57is basically just code up a little demo
142. 5:00app to try this out. I'm using the model
143. 5:03on OpenRouter. So, this is a crazy sort
144. 5:06of price model, right? You're looking at
145. 5:074.2 cents per million tokens in and 0
146. 5:12cents per million tokens out. And you
147. 5:14can see if we come in here, they've got
148. 5:16some code to actually how to call it on
149. 5:19using the Open AI API endpoints. And so,
150. 5:22we've been playing around for a couple
151. 5:23of hours with different kinds of demos
152. 5:25and trying out different things. So, the
153. 5:27first key thing here is that there are
154. 5:29three kinds of things that you can do,
155. 5:31right? One of them is a choice. So, you
156. 5:33can see here, I've got some text that
157. 5:35I'm going to pass in, and then it's got
158. 5:37a choice of five different classes out
159. 5:40here. And it's very good at being able
160. 5:43to detect which of these is the right
161. 5:45one. And so, what it will do, if we look
162. 5:47at the raw output here, is that it
163. 5:50basically has a type choice. In this
164. 5:52case, the choice was French, and then it
165. 5:54gives us the probabilities for each of
166. 5:57these and a confidence score. And you
167. 5:59can see that it did that pretty quickly.
168. 6:00So, I'm probably on the opposite side of
169. 6:02the world to where the model is actually
170. 6:04being hosted, but you'll see that it's
171. 6:07very quick at being able to respond.
172. 6:09Even for things like this, where I
173. 6:12wanted to try it on Thai, but using the
174. 6:14Thai characters, it's going to give it
175. 6:16away straight away that it's the Thai
176. 6:17language. So, I asked it to basically
177. 6:19make like a romanized version of that.
178. 6:22And you could see that even that has no
179. 6:24problems being able to get that right.
180. 6:27The second thing that you can do with
181. 6:29this is you can get it to give a score.
182. 6:32So, here you can see that we're giving a
183. 6:34score between zero and two. We're sort
184. 6:37of doing sentiment here. We're doing a
185. 6:38classification task again. We've got
186. 6:40sentiment coming out, and if we go for
187. 6:43positive sentiment, you'll see it comes
188. 6:45out two of two. If we go for negative,
189. 6:48zero. If we go for mixed, it's pretty
190. 6:51good at being able to do that. And you
191. 6:53can see each time this is costing me
192. 6:550.001
193. 6:59for of a cent, right, in here. If you
194. 7:02can break down what you're trying to do
195. 7:05to lots of different classifications,
196. 7:08you really can extremely cheaply do a
197. 7:11lot of stuff with this. And you can see
198. 7:13in this case where I tried to make it a
199. 7:14little bit more positive than in the
200. 7:16middle, I it does change coming back
201. 7:20each time. So, it still is a stochastic
202. 7:22process going on here, and we can see
203. 7:24the confidence is less than before. Each
204. 7:27time I ping it, I'm getting something
205. 7:28slightly different, but pretty close to
206. 7:31the same sort of ballpark scores there.
207. 7:34When we look at the API for that, you
208. 7:35can see that, okay, the probability that
209. 7:37it's coming back. So, you can see in
210. 7:39this case we've passed in a score. We've
211. 7:41got that back coming out of there. So,
212. 7:43scoring is a really nice function in
213. 7:45there. And then lastly, we've got the
214. 7:46true or false or the yes no kind of
215. 7:49probabilities here. So, this is a null
216. 7:52and it's basically going to give back a
217. 7:53probability for what we put in. Now, if
218. 7:57I ask it something simple like that, it
219. 7:59comes back 87% yes.
220. 8:01Is this text asking a question? Okay,
221. 8:03the answer would be no here. So, you can
222. 8:05see that it's 2% yes. So, basically it's
223. 8:08no, right? You could kind of think of
224. 8:10this as being flattened out by a sigmoid
225. 8:12function. So, you've got zero to one
226. 8:14coming out of this. And even if we do
227. 8:16things like where okay, we've got a
228. 8:17question, but we've got no question
229. 8:19mark, it's very confident that that is
230. 8:22still a question without a question
231. 8:23mark. So, it's not like it's just
232. 8:25looking for a question mark there. And
233. 8:27you can see different kinds of questions
234. 8:29will get different responses. So,
235. 8:31interestingly, if I just ask it, "Can
236. 8:33you help?" it's a lot less sure if it
237. 8:35can cuz I haven't been specific. But if
238. 8:38I ask it, "Can you help find my cat?"
239. 8:41well, then it's 95% sure that it can do
240. 8:44it. And if I just change one word in
241. 8:45there to "Can you help me feed my cat?"
242. 8:48it goes back down a bit. So, it is
243. 8:51interesting how it actually does this.
244. 8:53You will see that like again, this is
245. 8:55still a stochastic process. It comes
246. 8:57back slightly different each time, but
247. 8:59it is pretty consistent of you either
248. 9:02being able to say one thing or the other
249. 9:04thing. And you can see if I just give it
250. 9:06two words, then it starts losing its
251. 9:08confidence about this. So, it does seem
252. 9:11to me the longer that the input that you
253. 9:13put in, the more confident it gets with
254. 9:16its responses out. All right. Now,
255. 9:18looking at some sort of more practical
256. 9:20kinds of things that we could do here,
257. 9:22you can see that we could put in like a
258. 9:24message of I was charged twice for an
259. 9:26order. Please refund the duplicate. Get
260. 9:28it to work out which team it's going to
261. 9:30route it to. Here, it's billing, kind of
262. 9:33obvious. Sales one, it gets 100% sales.
263. 9:36What if we do something a little bit
264. 9:38more ambiguous? You can see now it's not
265. 9:40very confident. It's actually coming
266. 9:42back that it's unclear as as the class,
267. 9:44but it's still thinking that this is
268. 9:46more of a technical kind of thing. And
269. 9:48you can see that each time we run that,
270. 9:50we do get a slightly different response,
271. 9:52you know, going through this. Now, on
272. 9:53top of these, we're actually doing
273. 9:55multiple things at the same time. So,
274. 9:57you can see that we've also got a null
275. 9:59in here for was a refund requested.
276. 10:02Obviously, if I select this, the answer
277. 10:04is yes. Is it sort of time sensitive,
278. 10:07right? In this case, it's saying time
279. 10:09sensitive 7%. If I change that to please
280. 10:12refund the duplicate charge right now
281. 10:15instead of just right now, you can see
282. 10:17the time sensitive is jumping up to sort
283. 10:19of like 60 70% in this case. We can even
284. 10:22try it with things like injecting stuff
285. 10:25in there. So, doing any sort of thing
286. 10:27where someone's trying to do a prompt
287. 10:29injection or something like that, this
288. 10:31seems to do pretty well at being able to
289. 10:34get that kind of things. Sarcasm is also
290. 10:37something that it seems to be doing an
291. 10:39okay job at it. It would be interesting
292. 10:40also to test it on humor and other kind
293. 10:43of things, but it is good that it's not
294. 10:45being tricked by a lot of different
295. 10:47things that normally would trick this
296. 10:49kind of thing. Other tasks that you can
297. 10:51get it to do is things like code
298. 10:53reviews, like where you can basically
299. 10:55ask it is something safe, is something
300. 10:57not safe. It seems to do a good job at
301. 10:59that. Doing things like classification
302. 11:02on different types of content, that
303. 11:04seems to work really well. You can see
304. 11:06here it's able to discover PII
305. 11:08information, personal identifiable
306. 11:10information here. Does a good job at
307. 11:12that kind of thing. Does a good job at
308. 11:14sort of spam detection as well. So, the
309. 11:16fourth example here is looking at agent
310. 11:19tool selection. So, this is a little bit
311. 11:21like the cactus model that we looked at
312. 11:24a while ago in that it's doing some kind
313. 11:26of sort of function calling, but it's
314. 11:28important to understand here that it's
315. 11:30not actually extracting anything out and
316. 11:33passing it to the tool, right? It's just
317. 11:35telling us which tool to use as opposed
318. 11:39to getting the right arguments out to
319. 11:41pass to the tool. So, in that sense,
320. 11:44function calling models are still able
321. 11:46to basically generate out what should be
322. 11:48the input going to the function. Last
323. 11:50up, one of the things that I thought was
324. 11:51really interesting to test it is
325. 11:53stringing actions together, right? So,
326. 11:56where we basically give it something and
327. 11:59we then want to basically run a number
328. 12:02of different tasks over this to see how
329. 12:05long does it take, how does it actually
330. 12:07process these.
331. 12:09So, here's 20 different tasks going
332. 12:11through, and you can see that it's
333. 12:13flying along. Now, I could have done
334. 12:14them in parallel for some of them at
335. 12:17least, but you can see they're just
336. 12:18going through it like that, we've gone
337. 12:20through all different 20 tasks in here,
338. 12:24gotten the answers out, gotten the
339. 12:26results back for these, and it's cost
340. 12:28just over 1/20th of 1 cent. I really
341. 12:32feel like this is where you're going to
342. 12:33find really interesting things, right?
343. 12:35When you can string lots of different
344. 12:38classifications together to be able to
345. 12:41get some kind of response out that's
346. 12:44really valuable for you in a really
347. 12:45quick time. Okay, so how does this all
348. 12:48actually work? Well, they don't really
349. 12:50tell us a lot here. There's no paper,
350. 12:53there's no architecture diagram. They
351. 12:55don't give us a lot of information, but
352. 12:56what they do give us is three pieces
353. 12:58that they mentioned that there's a new
354. 13:00model architecture, a parallel sampler,
355. 13:03and then perhaps the most interesting
356. 13:05bit is that they're training with a new
357. 13:06method they're calling RLCD. This is
358. 13:09reinforcement learning for calibrated
359. 13:11decisions. Now, how RLCD actually works,
360. 13:15they don't really tell us, but they
361. 13:16compare that to the existing LLMs using
362. 13:20both RLHF, so that's reinforcement
363. 13:22learning from human feedback, and also
364. 13:24reinforcement learning from verifiable
365. 13:26rewards, which is a whole sort of GRPO
366. 13:29and a lot of the ways that the modern
367. 13:30models are being trained at the moment.
368. 13:32So, it does seem probable to me that
369. 13:34this is some kind of transformer, but
370. 13:36perhaps what it's actually doing is just
371. 13:38using the pre-fill stage to calculate
372. 13:41heads, etc. And then doing a prediction
373. 13:43out, which is just a classification or a
374. 13:45regression depending on the different
375. 13:47tasks that you've got there. One of the
376. 13:48cool things here is definitely the speed
377. 13:51because there are no tokens being
378. 13:52generated one after another. Everything
379. 13:55comes back in a single pass. And that
380. 13:57means you're done in about 70 to 500
381. 14:00milliseconds, which you can add to the
382. 14:02basically round trip time and it's still
383. 14:04extremely fast for doing these kinds of
384. 14:07tasks.
385. 14:08Now, one of the things I find
386. 14:09fascinating is that they claim it can't
387. 14:11hallucinate. And it seems what they're
388. 14:13really saying there is that this is not
389. 14:15going to break the schema that you give
390. 14:17it to basically come back with things.
391. 14:19So, like no broken JSON, no invented
392. 14:21tool names. It can only deal with what
393. 14:24you've actually given it. It's not going
394. 14:25to actually generate sort of anything
395. 14:26new there.
396. 14:28Now, it does seem that it can still pick
397. 14:29the wrong option and like a wrong
398. 14:31answer, meaning that it could have the
399. 14:33right type of question, but actually
400. 14:36give you a wrong answer for that. And it
401. 14:38seems that this is where they're relying
402. 14:39on that RLCD to be able to deliver
403. 14:42high-quality answers back. Now, we don't
404. 14:45know if this is just like a smaller LLM
405. 14:47that's been trained in a different way
406. 14:49or if this is actually a different
407. 14:51architecture, etc. It is kind of amazing
408. 14:53though that it can do things like play
409. 14:56Doom where it can do it so quickly that
410. 15:00you can basically have it responding to
411. 15:03these inputs. And the thing that I find
412. 15:04fascinating here is that doing that and
413. 15:06running sort of 10 queries per second,
414. 15:08even at that rate, ends up only costing
415. 15:11about $7 per hour. So, overall, I would
416. 15:14say if you're doing any kind of
417. 15:15classification with any kind of BERT
418. 15:18model or things like that, this is
419. 15:21definitely worth trying out to see how
420. 15:24it goes for your particular use cases.
421. 15:26And I do wonder if this really delivers
422. 15:28like they say it does, is this going to
423. 15:31be the end of sort of fine-tuning small
424. 15:34bird models for doing purely
425. 15:36classification tasks? Anyway, they make
426. 15:38the point here that this is early days,
427. 15:40that they're really focused on any sort
428. 15:42of decision that you need to be able to
429. 15:43automate. And my guess is that they've
430. 15:45probably kicked off a whole bunch of
431. 15:47people trying to make open-source
432. 15:49versions of this. So, over the next
433. 15:51month or so, we might see some really
434. 15:53interesting open-source models that come
435. 15:55along with similar ideas to this.
436. 15:58Anyway, let me know in the comments if
437. 15:59you've actually tried Jev, where you
438. 16:01could see yourself actually using it.
439. 16:03And I'd love to hear from people who've
440. 16:05actually tested it and found that it
441. 16:07didn't work for certain things. What
442. 16:09were those use cases that it didn't work
443. 16:11for? And as always, if you found the
444. 16:12video useful, please click like and
445. 16:14subscribe, and I will talk to you in the
446. 16:15next video. Bye for now.