How to Build Things with Jev & OpenJevs — Transcript

Full transcript

1. 0:00All right, so in this video, I want to
2. 0:02jump in and actually show building
3. 0:04something with Jev. Some of the actual
4. 0:06decisions you need to make about how
5. 0:08you're going to use it, some of the
6. 0:09techniques of how you can use it, and
7. 0:11some of the benefits and challenges that
8. 0:12you're going to see along the way. So,
9. 0:14I'm going to start off with a very quick
10. 0:15recap, but then I'm going to jump in to
11. 0:18building a model router. So, the whole
12. 0:21idea here is that this is something that
13. 0:23will be an endpoint that you run
14. 0:24locally. It can then interpret what you
15. 0:27send to that endpoint via Jev or via an
16. 0:31open Jev like we'll look at later on.
17. 0:33And then it can decide what sort of
18. 0:35category is this, and what model should
19. 0:37it be sent to. Can also decide whether
20. 0:40you've got PII, private identifiable
21. 0:43information in there, so that we could
22. 0:45have it set up to not send to the cloud
23. 0:47for that, so that always gets sent to
24. 0:49something locally. So, I'm going to try
25. 0:50and focus on a few different models in
26. 0:52here, but locally I'm going to run a lot
27. 0:54of this with just the mini CPM 52B. Now,
28. 0:57I'm running this on my machine with an
29. 0:59endpoint, but you could be running this
30. 1:00with LM Studio, you could be running
31. 1:02this with Ollama, totally up to you. And
32. 1:05then for the cloud models, I'm going to
33. 1:06focus on a few models there, but mostly
34. 1:08I'm going to be using the DeepSeek V 4.1
35. 1:10Flash there. To make it a bit more fun,
36. 1:14also, what I'm serving locally is the
37. 1:17latest Qwen image 2.1. So, this
38. 1:20literally just came out yesterday as I'm
39. 1:22recording this. It's a really nice model
40. 1:24that can do a whole bunch of different
41. 1:25kinds of things like image generation,
42. 1:27image editing, et cetera. Probably the
43. 1:30only thing that sucks about it at the
44. 1:31moment is the license currently that
45. 1:33it's basically a non-commercial license.
46. 1:35But, I'm going to show you how our
47. 1:37system will be able to make images,
48. 1:39interpret images, do different automated
49. 1:42calls to lots of LLMs that are out
50. 1:44there. All right, so a quick recap. If
51. 1:46you missed my first video on it, Jev is
52. 1:48from Type Safe AI, and they call it a
53. 1:51system one model. It doesn't generate
54. 1:53text at all. You give it a state, which
55. 1:55is basically just whatever you want to
56. 1:57have judged, and then you give it a set
57. 1:59of typed questions. And what comes back
58. 2:02are typed answers with probabilities as
59. 2:05well. The cool thing is that then you
60. 2:06can use that for your code to basically
61. 2:08make conditional logic decisions on. So,
62. 2:11that's basically just all your sort of
63. 2:12if-else statements, right? If the choice
64. 2:15comes back this, then I do this. If the
65. 2:17probability of this is higher than this
66. 2:19percent, then I'm going to do this. So,
67. 2:21we don't have to worry about a model
68. 2:23outputting good JSON and basically
69. 2:25getting the prompt right to have it
70. 2:27comply with everything that we want. And
71. 2:29it really should come back in somewhere
72. 2:31around the 70 to 500 ms range. And
73. 2:35remember the cool thing for this is it
74. 2:36doesn't charge you anything for output
75. 2:38tokens at all, and it only charges you 4
76. 2:41cents per million input tokens. A nice
77. 2:44way of thinking about this is that you
78. 2:45can kind of think of it as an if
79. 2:47statement that understands language.
80. 2:50Now, in Jev and most of the open Jev
81. 2:52clones, there are three kinds of
82. 2:54questions you can ask it. The first is
83. 2:57choice. So, you just basically give it a
84. 2:59set of options, it picks one, and you
85. 3:01get a probability for each option plus a
86. 3:04confidence value. So, in many ways
87. 3:06that's like your categories, right? You
88. 3:08use it to basically categorize things
89. 3:10into different classes. The second is
90. 3:12score. This is where you've got some
91. 3:14kind of ordered levels from sort of low
92. 3:15to high and up to 10 of them, and it can
93. 3:18then basically give you a position on a
94. 3:20scale that you've determined. Again,
95. 3:22that's with confidence. The third type
96. 3:25is a null, which is really just like a
97. 3:27yes or no question, right? You basically
98. 3:29just ask it a question, you give it your
99. 3:32input, it will come back saying yes or
100. 3:34no. Can also be thought of as true or
101. 3:36false going on here. And yet again, that
102. 3:39will also give you a probability that
103. 3:41that answer is yes, so that if you take
104. 3:43the inverse of that, you've got the
105. 3:45probability of no. All right, so how is
106. 3:47all of this going to map out onto a
107. 3:49router? Turns out pretty neatly. So, you
108. 3:51can think of choice being the
109. 3:53categorization, what kind of task is
110. 3:56this? Is it chit-chat? Is it code? Is it
111. 3:58asking for an image? And we could have a
112. 4:01whole bunch of different things like
113. 4:03that. We've then got it basically where
114. 4:05it picks a lane, right? And then Jev is
115. 4:07going to basically pick the lane. So,
116. 4:09chit-chat is going to go into a small
117. 4:11model running locally. If it's code
118. 4:13though, that's going to go out to the
119. 4:15deep seek model on Open Router. If it's
120. 4:17an image request, that's going to go to
121. 4:19Quen image 2.1, which is also running on
122. 4:22my machine. The second one for score is
123. 4:25we can actually ask something like how
124. 4:27hard is this? So, within a lane, that
125. 4:30decides whether the little model can
126. 4:32handle it. So, remember the little model
127. 4:34that we're using is MiniCPM 52B, right?
128. 4:37A tiny 2B model. So, that will decide if
129. 4:40that can handle it or if it needs to go
130. 4:42up a tier to a much better model. And
131. 4:44then thirdly, we've got the null, which
132. 4:46is the gate. Does this contain private
133. 4:48or client data in any way? If that comes
134. 4:51back as high, the whole thing has to
135. 4:54stay local, no matter what. And at the
136. 4:55start, I'm going to be using Jev to
137. 4:57actually do all of these. But later on,
138. 4:59we can switch to one of the open Jevs,
139. 5:01so that literally if it is PII, it never
140. 5:04actually leaves our machine even to go
141. 5:06to Jev. But at the start, I'm going to
142. 5:07say, "Okay, we trust Jev. We just don't
143. 5:10trust the big frontier lab model
144. 5:12providers." The other cool thing is that
145. 5:14all three questions are going to go in
146. 5:16one request against the same state. And
147. 5:19Jev evaluates
148. 5:21So, asking three things, and we could
149. 5:22actually even add in more, is going to
150. 5:24cost you pretty much the same time as
151. 5:26asking one. But because you get a
152. 5:28confidence back, the router can know
153. 5:31when it's unsure if the confidence type
154. 5:33is low. So, you don't guess, you just
155. 5:35fall back to a safe default. Now, as you
156. 5:38see, I'm going to point out right from
157. 5:39the start that there is one thing in
158. 5:41here, if I'm sending your prompt to
159. 5:43TypeSafe to ask whether it's private,
160. 5:45I've kind of already leaked it, right?
161. 5:47So, later on in the video, I'm going to
162. 5:49swap out Jev for one of the open
163. 5:51versions running locally, so that the
164. 5:53whole router is actually just running on
165. 5:55my machine. So, let's jump in and have a
166. 5:57look at how this actually gets built.
167. 5:59Right, so let me give you a little demo
168. 6:02of what this can do. You can see here,
169. 6:04if I basically just fill out a prompt
170. 6:07and I send that off, this is going to
171. 6:10basically have Jev decide whether this
172. 6:14should go to a local model, whether it
173. 6:16should go to DeepSeek 4.1, whether it
174. 6:18should go to an image model. So, if we
175. 6:20come over here and look, we can see the
176. 6:22models that I've got set up at here is
177. 6:24we've got a MiniCPM 5-2B,
178. 6:27that's fully local, that's running here
179. 6:29at the moment. I've then got the Qwen
180. 6:32Image 2.1, which again is fully local
181. 6:35running here. And then I've got the
182. 6:37DeepSeek 4.1 Flash model. Now, that's on
183. 6:40OpenRouter, so any of the ORs are
184. 6:42OpenRouter. I've also got it set up if
185. 6:44we want to use something like a DeepSeek
186. 6:46Flash plus web in there. And if we want
187. 6:48to, we can even turn on Opus 5 in here
188. 6:51for something that's going to be really
189. 6:52hard. Now, if I wanted to, I could run
190. 6:54DeepSeek 4.1 Flash locally. I am
191. 6:56actually sort of halfway through
192. 6:57recording a video all about DeepSeek 4.1
193. 7:01and the DeepSeek harness and how all
194. 7:03those things go together and how you can
195. 7:04do it all locally. I could also run a
196. 7:06local VLM if I wanted to in here. But
197. 7:09the idea is at the moment we're using
198. 7:11Jev to basically split all these up. I
199. 7:14can come in here and actually, you can
200. 7:15see I can pick the models separately.
201. 7:18So, if I wanted to run this just to this
202. 7:19model all the time, no problem. You can
203. 7:22see super fast there, it's guaranteed to
204. 7:24go local. If I wanted to do that and
205. 7:26just run it to DeepSeek, I can do that,
206. 7:29right? And then obviously it's going to
207. 7:31take longer cuz it's going to the cloud.
208. 7:33It's going to decide if it needs some
209. 7:34thinking, it's going to then stream the
210. 7:36response back. But really what we want
211. 7:38is auto, right? This is where Jev is
212. 7:41going to basically monitor the whole
213. 7:44thing and decide what it is that we
214. 7:48want. So now you can see, okay, if I do
215. 7:50the hey, how's it going? Really this is
216. 7:52a simple request, right? We would expect
217. 7:55that this is going to basically go to
218. 7:57the local model. And sure enough, it
219. 7:59went to the local model. Now, if we come
220. 8:02in here and actually look at the
221. 8:03requests, you can see that the
222. 8:04difficulty here here was basically zero,
223. 8:07that it was really there was no private
224. 8:09information in there and that the lane
225. 8:12or the category that it decided that
226. 8:14this response should go to was the small
227. 8:16model, right? And it did all of that and
228. 8:19Jev basically did all of that in 331
229. 8:23million there. All right, now if I give
230. 8:24it something a little bit harder, you
231. 8:25can see, okay, I'm going to get it to
232. 8:27write a Python code to deduplicate
233. 8:30stuff. So this is actually still a
234. 8:31pretty kind of simple thing, but we've
235. 8:33got this now being routed to Deep Seek
236. 8:36as opposed to doing it locally. And this
237. 8:38is basically because everything that
238. 8:40I've put in here around code should
239. 8:43actually go to the Deep Seek model. So
240. 8:44now if I come in and actually look at
241. 8:46Jev, I can see for this last run what
242. 8:50actually happened. So the difficulty
243. 8:52went up, so this was more difficult than
244. 8:55chit-chat because it's not like off the
245. 8:57charts sort of difficult. So if we look
246. 8:59here, we can actually see what is
247. 9:01actually getting sent to Jev. Here we've
248. 9:04got the state with the actual prompt in
249. 9:05there, etc. You can see the questions.
250. 9:08So for choice, uh choice is going to
251. 9:11have the instructions of what is the
252. 9:13latest user message asking the assistant
253. 9:15to do, judge the latest message in the
254. 9:18context of the conversation. And then
255. 9:20the criteria basically tells us, okay,
256. 9:22we've got one class that's chit-chat,
257. 9:25one class that's simple question, one
258. 9:27class that's rewrite or summarize, one
259. 9:29class that's code, one class that's
260. 9:31reasoning or analysis, and then one
261. 9:34class that's image for doing image
262. 9:36generation. At the same time, we've also
263. 9:38got score. How much model capability
264. 9:41does a good answer need? And we can see
265. 9:43we've got trivial, we've got easy, we've
266. 9:45got moderate, we've got hard. And you
267. 9:47can see you kind of got definitions for
268. 9:48those. So, trivial is a small 2B model
269. 9:51handles it easily, right? Greetings,
270. 9:53one-line facts, tiny edits, that kind of
271. 9:56thing. And then it goes right up to sort
272. 9:58of like a frontier, right? Needs the
273. 9:59best model available, research-grade
274. 10:02reasoning, large systems, et cetera. For
275. 10:04the privacy stuff, we've got a null,
276. 10:07right? So, the instruction here is does
277. 10:09this conversation contain personal,
278. 10:11confidential, client, financial,
279. 10:13medical, credential data, such as API
280. 10:16keys, passwords, anything like that? And
281. 10:18then we've basically got true that it
282. 10:20contains real secrets, and false, which
283. 10:23it's it's not sensitive data at all. So,
284. 10:26that's one that we'll sort of come back
285. 10:27and look at. We've got another null in
286. 10:29there for deciding whether it needs web
287. 10:31stuff or not. If we look at what comes
288. 10:33back, that you can see from this the
289. 10:35choice was code, which makes sense cuz
290. 10:37we asked for code, right? And we can see
291. 10:40that it really has come back like a 100%
292. 10:42for code in there. By the looks of it,
293. 10:45it's kind of decided that it's worked
294. 10:47out its probabilities there and its
295. 10:49confidence. And then we can see for
296. 10:50private and for web there. Let's try one
297. 10:53that actually is sort of private. So,
298. 10:56okay, here I've put a fake API key in
299. 10:58and it's not going to do anything if you
300. 11:00want to use it. So, here we can see
301. 11:02straight away it detected that this was
302. 11:06local and was private. If we come across
303. 11:08and we look at the actual message here,
304. 11:10we can see straight away now difficulty
305. 11:12is not super difficult, but we've got
306. 11:15private. So, this is determined that
307. 11:17okay, this is going to have to stay
308. 11:19local for this case in here. And you can
309. 11:22see the same thing if we put in like a
310. 11:24person's name or something or their
311. 11:25phone number or something. Again, it
312. 11:27straight away picks up PII, right? So,
313. 11:30PII is your private identifiable
314. 11:32information, extracts that, it assigns
315. 11:35it to the local model. And we can see
316. 11:37that if we come in here, we've got the
317. 11:38private coming through at like 0.96.
318. 11:42So, that's what's driving the code at
319. 11:44the back end to actually check all this
320. 11:45out. So, this is the architecture of how
321. 11:48all this works, right? We've got our
322. 11:50browser, so what you're seeing here is a
323. 11:52Next.js UI app, right? Just kind of
324. 11:55simple app. It's then sending to a fast
325. 11:59API server. So, this is a server that's
326. 12:01running locally on my machine. So, fast
327. 12:04API is just a nice way to build a simple
328. 12:07sort of Python API in here.
329. 12:10You can see that with one call, it will
330. 12:12basically do the deciding, and then
331. 12:14we'll do the routing. It can then stream
332. 12:16back to us from the winner. We've got a
333. 12:18fallback in there. We can log everything
334. 12:20to SQLite if we want to have everything
335. 12:22log in there, and we've got health
336. 12:24checks going on.
337. 12:26And then you can see we've got
338. 12:26compatible endpoints. So, that's just
339. 12:28the standard OpenAI API for calling
340. 12:31models on OpenRouter, etc. And we're
341. 12:33using that for both Jev, but also for
342. 12:36MiniCPM, which is running on SG Lang as
343. 12:39a local model here. I'm pretty sure
344. 12:40that's the DFlash version I've got going
345. 12:42there. And then we've got for the
346. 12:44DeepSeek model in the cloud, and we've
347. 12:46got the Quen image model in here. So,
348. 12:48the way it works is it basically asks
349. 12:51Jev, right? It gets a decision. It works
350. 12:54out what it needs from that, and then it
351. 12:55decides a lane. And these lanes are sort
352. 12:58of pre-configured. They've got an order
353. 13:01of preference that they go through this.
354. 13:02Now, if I turned on the actual DeepSeek
355. 13:05in the spark, we could actually run that
356. 13:07locally. You can see for hard here, if
357. 13:10I've got Claude Opus on, that will route
358. 13:12to Opus first up, as opposed to just
359. 13:14routing to DeepSeek. That's the key
360. 13:16thing that I've got there. And then you
361. 13:18can see we've just got our sort of
362. 13:19conditional logic in here of that what's
363. 13:22going to actually happen as we go
364. 13:24through this. So, if it's greater than
365. 13:26.6, we're going to go for a web-enabled
366. 13:28model. If the prompt is over 16,000
367. 13:31tokens, we can set it to go for a
368. 13:33general model. We can do a whole bunch
369. 13:35of different rules in here. We've got
370. 13:36the privacy flag. If it's above 0.5,
371. 13:39we're going to go for a local model,
372. 13:41etc. And you can see at the same time
373. 13:42I'm logging everything out and storing
374. 13:44everything. So, I'm storing the Jev
375. 13:46decisions in here. I'm storing the the
376. 13:48SQLite messages, and I'm even storing
377. 13:51images in here. So, you can see that now
378. 13:53if I ask it to make an image, it's
379. 13:56worked out that, okay, it should be
380. 13:57using a Qwen image in here. It's going
381. 14:00off. It's generating on the machine
382. 14:03here, and we're going to get an image
383. 14:04back showing in here. And you can see
384. 14:06sure enough, there's our sandy-colored
385. 14:09mancoo. Now, actually the way this is
386. 14:11set up, we're actually doing some prompt
387. 14:13rewriting in here. So, this is obviously
388. 14:15one of the things that all the different
389. 14:17sort of image services use is that
390. 14:20they'll take your simple prompt, which
391. 14:22in this case was this, and they'll
392. 14:24rewrite it, right? Now, in this case
393. 14:26we're using just the MiniCPM model to
394. 14:29rewrite it. Could actually probably use
395. 14:30a better model to rewrite it and to get
396. 14:33a nicer rewriting. You can see, okay,
397. 14:35this is basically run it. You can see
398. 14:37that it's added some stuff to it like a
399. 14:39sunlit meadow, some other stuff in there
400. 14:41to the actual prompt. So, that's how it
401. 14:43all sort of comes together like that.
402. 14:45The other thing too is that I put in
403. 14:47here a tools registry. So, you can see
404. 14:50I've actually set it up so that the Qwen
405. 14:52image was actually a tool thing in here.
406. 14:54And you can see also that we've got our
407. 14:57judge, right? Which is what we've got
408. 14:59here. We can actually make our judge
409. 15:02both the web version, or we can even try
410. 15:05doing it with one of the open-source
411. 15:07versions. So, in a second we'll try that
412. 15:09with Sem if for being the sort of local
413. 15:12Jev as opposed to the web one. You can
414. 15:15see here we can also change out the
415. 15:17routing thresholds. So, remember Jev is
416. 15:19delivering back probabilities, etc., and
417. 15:22stuff like that. So, that's something
418. 15:23that we can play with in here. And I've
419. 15:25got some other little sort of simple
420. 15:26things for like putting in a system
421. 15:28prompt in there. All right, we've moved
422. 15:30now to the open Jev one, and we can see
423. 15:32this here, right? So, let's try again
424. 15:36now with our privacy one and see, okay,
425. 15:39is the Sem if model able to do it? And
426. 15:42you see, it was so quick. It's fully
427. 15:44local. It's worked it out. Let's come in
428. 15:46and actually look at our judge. And you
429. 15:48see that the format going in and coming
430. 15:50back is slightly different, but it's
431. 15:52still doing the exact same kind of
432. 15:54thing. We've got this difficulty, we've
433. 15:56got the private, we've got the lanes
434. 15:58going on there. And you can see in this
435. 16:00case, it was 88 milliseconds running
436. 16:02through all of this. Let's ask it to do
437. 16:04a search on the web for this. So, this
438. 16:07should be using the open router with the
439. 16:10web version of this, and you can see,
440. 16:13sure enough, it's using DuckDuckGo in
441. 16:14there. We've basically got this going
442. 16:16through, and you can see, sure enough,
443. 16:18we've got our response back going
444. 16:20through this. So, the idea here was just
445. 16:22to show you a real-world example of
446. 16:24using Jev, and you could use either the
447. 16:27web-based Jev, which obviously is going
448. 16:30to be better at generalization most of
449. 16:33the time, especially for harder kinds of
450. 16:35things, or you could use one of the open
451. 16:37ones. In this case, I was using Sem if
452. 16:40in here. You can see that this is
453. 16:41basically been gating all the models
454. 16:43that we have here. Now, if I come in
455. 16:45here, you can see I can change the
456. 16:47thresholds and stuff like that, but we
457. 16:48can look at some of the stats just from
458. 16:50a little bit since I restarted this
459. 16:52last. That it's done 53 requests. It's
460. 16:55answered 75% of them locally, right? We
461. 16:58can see that, yes, sure enough, we've
462. 17:00spent money, but this is what we've
463. 17:01spent both, not just on Jev, but also on
464. 17:04Deep Seek with these requests. And we
465. 17:07can see that this is how much we've
466. 17:08saved by having 75% of the requests be
467. 17:11local in this case. So, depending on
468. 17:14whether we've got Jev configured or Sem
469. 17:16if configured, we can see what the
470. 17:18actual response times are going to be
471. 17:20coming back here. And we can see how
472. 17:22many of the prompts were actually
473. 17:24private in here. I've been showing you
474. 17:26these are sort of one-shot prompts, but
475. 17:27we can actually do a full sort of chat
476. 17:29in here as well. Now, you could
477. 17:31configure this that if you started out
478. 17:33with something really hard, and we were
479. 17:35using Opus, if we were actually doing
480. 17:38sort of a long conversation going here,
481. 17:41we'd probably want to use the caching
482. 17:43and stay on the Claude Opus. So, that
483. 17:46would be pretty easy to take this and
484. 17:48configure this up. The idea that I'm
485. 17:49trying to show you here is just building
486. 17:52a sort of real-world use case, either
487. 17:54using the Jev model itself, or using one
488. 17:56of the open versions of this. So, let me
489. 17:59know in the comments if there are other
490. 18:00sort of use cases that you would like to
491. 18:02see me do something like this. And also
492. 18:04sort of what you're actually looking at
493. 18:06using this for yourself. My plan is
494. 18:08actually to get a second channel up in
495. 18:10the next couple of days, which is going
496. 18:12to be probably a lot more code
497. 18:13intensive. So, what I might do is just
498. 18:15record a walk-through of showing the
499. 18:17actual code behind these, and I'll put
500. 18:20the code up on GitHub and stuff like
501. 18:21that. If people are interested to play
502. 18:23with it and try it out, etc. For me, the
503. 18:25biggest takeaway I would say is whether
504. 18:27you're using the Jev model in the cloud,
505. 18:29or whether you're using one of the open
506. 18:31local versions of this, this really
507. 18:33enables you to do a lot of rethinking of
508. 18:36how you actually build and structure
509. 18:38these kinds of different agentic use
510. 18:41cases now, these kinds of things where
511. 18:43you want to be able to save on tokens,
512. 18:45but you want to make sure that you've
513. 18:47got these decisions happening really
514. 18:49quickly, and can be scaled for a tiny
515. 18:52amount of money here. The funny thing is
516. 18:54when I went back and looked at the demos
517. 18:56that I did the other day in one of the
518. 18:57videos, the entire demos that I did in
519. 19:01the video cost less than 1 cent. So,
520. 19:04this is a huge way to basically save
521. 19:06money and compute, whether you're
522. 19:07basically just doing everything fully
523. 19:09local, or whether you're using the cloud
524. 19:11version as well. Anyway, as always, if
525. 19:13you found the video useful, please click
526. 19:14like and subscribe, and I will talk to
527. 19:16you in the next video. Bye for now.