Using Jev In Your Agent Harness — Transcript

Full transcript

1. 0:00Okay, so most of the model calls in an
2. 0:03agent harness aren't actually writing
3. 0:05anything. They're making decisions.
4. 0:08They're making decisions like which tool
5. 0:09should I use here, what skill should I
6. 0:12load, is this command safe to run, and
7. 0:15even decisions for things like rag where
8. 0:17it's asking which of these 20 search
9. 0:19results actually answers the question.
10. 0:22Up until recently, everybody has
11. 0:24approached those decisions pretty much
12. 0:27in the same way. They've used a full LLM
13. 0:30call to get the answer back. So in this
14. 0:32video, I want to show you how you could
15. 0:33use Jev and the other open decision
16. 0:36models to actually improve an agent
17. 0:38harness from the inside. I'm going to
18. 0:41talk about some of the things that you
19. 0:42can actually do with it. I'll show you a
20. 0:44couple of cool examples that I've been
21. 0:45playing with for doing progressive
22. 0:47disclosure on lots of skills and for
23. 0:49doing re-ranking of rag results. But
24. 0:51I'll also talk about some general
25. 0:53principles and perhaps more importantly,
26. 0:55what are the use cases that you don't
27. 0:56want to use a model like this. All
28. 0:58right, so let's start with the loop that
29. 1:01every agent runs. A request comes in and
30. 1:03it goes to an LLM. Along with that, you
31. 1:06basically have some kind of tool
32. 1:07registry going in there, a system
33. 1:10prompt, perhaps a set of skills, some
34. 1:12guidelines of how the agent's going to
35. 1:13work, etc. And the LLM decides straight
36. 1:16away what action should it take, whether
37. 1:18it needs to use a tool. And if it does,
38. 1:21it writes out a function call giving us
39. 1:23the tool name plus the arguments as
40. 1:25JSON. Your scaffolding your agent
41. 1:27harness then runs that code as a
42. 1:29function. It gets the results and it
43. 1:32appends that to the conversation. And
44. 1:34the whole thing goes back to the LLM.
45. 1:36And then round and round this goes until
46. 1:39eventually your LLM's decided it's done
47. 1:41and it writes its final answer. Along
48. 1:44that road, it's probably going to use
49. 1:45many tools. It's basically going to get
50. 1:48information to expand its context and
51. 1:50it's going to make a lot of decisions
52. 1:52along the way. Now if you look at those
53. 1:54LLM steps to see what's actually going
54. 1:57on at each step. The whole sort of thing
55. 1:59of asking itself which tool it should
56. 2:01use, is it safe to actually run that?
57. 2:03And often because it's making those
58. 2:05decisions itself, it never really stops
59. 2:07to ask any sort of verification step or
60. 2:10any kind of safety step in there to ask,
61. 2:13is this safe to run? And for most
62. 2:15agents, if they don't have any sort of
63. 2:17reflective step in there, they're really
64. 2:19never even asking themselves, is this
65. 2:21result good enough or should I go around
66. 2:23again and try again? Each of those
67. 2:26things are clear decisions that you can
68. 2:28use a decision model like Jev or one of
69. 2:31the open Jevs to actually improve the
70. 2:33quality of the agent harness. On top of
71. 2:36that, a lot of the times the decisions
72. 2:38that the LLM's actually making are just
73. 2:40a small set of options. Do I go to this
74. 2:43node next or this node? Which skill
75. 2:45looks like the best one to use in this
76. 2:47situation? Which skill do I ask for more
77. 2:50information in this situation? And each
78. 2:53time you're paying for a full call to a
79. 2:55large LLM there. And that's costing you
80. 2:58both in money for tokens, but often even
81. 3:01more it's costing you in time of you
82. 3:03actually having to go through and do
83. 3:05these things. And that's why a lot of
84. 3:07the time when people are building custom
85. 3:08agents, they often don't put anywhere
86. 3:10near as many verification steps in there
87. 3:13as they should simply because that's
88. 3:16just going to slow everything down. So
89. 3:18if you've watched any of my other Jev
90. 3:19videos, you know that I've basically
91. 3:21talked about the whole concept of Jev as
92. 3:24being like a smart if statement. You
93. 3:26give it a state which is just the thing
94. 3:28that you're evaluating, and then you
95. 3:30give it a set of typed questions about
96. 3:33that state and it gives you back typed
97. 3:35answers with probabilities. No text at
98. 3:38all. Choice will pick one option from a
99. 3:40list. Score rates the state on a scale
100. 3:43that you define. A null is a yes or no
101. 3:46or a true or false that's returned back
102. 3:48as a probability. And the cool thing is
103. 3:50you're not limited to one question at a
104. 3:53time. You can ask many questions about
105. 3:55the one state all in a single call and
106. 3:58get answers back in parallel and it
107. 4:00almost doesn't change the response time
108. 4:02at all. So, if you're looking at how to
109. 4:05improve your agent harness, one of the
110. 4:07key things you can do is look at that
111. 4:10loop of where it's full of a bunch of if
112. 4:12statements that are dressed up as
113. 4:14function tools or dressed up as asking
114. 4:16the model for structured JSON responses
115. 4:18back and you can convert those kind of
116. 4:20calls to something that's much quicker
117. 4:22like a decision model. All right, so
118. 4:24where does this actually belong in your
119. 4:26agent harness? I'd say that there's six
120. 4:28main places that you want to be thinking
121. 4:30about here. So, the first one is model
122. 4:32routing. I did a whole video about how
123. 4:34you can look at this. You got a request
124. 4:37coming in, Jev decides whether it goes
125. 4:39to a cheap fast model or expensive model
126. 4:42that's going to do much better reasoning
127. 4:43etc. If anyone watched the video that I
128. 4:45made about that, you can see the whole
129. 4:47thinking process for actually putting
130. 4:48that together. The second use is risk
131. 4:51gating. So, before a tool call actually
132. 4:54runs, Jev can actually check whether
133. 4:56it's going to be something destructive
134. 4:58like deleting a database or wiping a
135. 5:00folder or whether what it's actually
136. 5:02going to be doing is quite benign. Now,
137. 5:04we know that a lot of the coding agents
138. 5:06have had this kind of [clears throat]
139. 5:07check for a while, but it's been hidden
140. 5:09generally in their harness. And when you
141. 5:11were using it, you were basically paying
142. 5:13for a full tool call with the amount of
143. 5:15tokens that it actually gets used there.
144. 5:17So, being able to configure your harness
145. 5:19so that when certain tools get called,
146. 5:22perhaps like a bash tool, you actually
147. 5:24look at what the arguments are going to
148. 5:26be going into that and you send that to
149. 5:28Jev to basically ask it, "Hey, is this
150. 5:31going to be malicious or benign?" Very
151. 5:33quickly, you've got a safety check that
152. 5:35can take milliseconds. And of course,
153. 5:37with the open decision models, that can
154. 5:38be running fully locally as well. Third
155. 5:41up, you've got tool and skill selection.
156. 5:43So, generally what people have done is
157. 5:45they inject their entire tool registry
158. 5:48that they're going to be using into the
159. 5:50system prompt. And they can chew up
160. 5:51thousands of different tokens. So, the
161. 5:54whole idea of skills has reduced that a
162. 5:56lot by using progressive disclosure.
163. 5:59This is one of the key things that I'm
164. 6:00going to show you in the demos a bit,
165. 6:01that you can actually have lots of
166. 6:03skills going on, categorize them, have
167. 6:06your decision model decide which kind of
168. 6:08category it's going to need, and then
169. 6:10just load the appropriate skills into a
170. 6:12system prompt at that point. Fourth up,
171. 6:15you've got things like ranking and
172. 6:16judging. So, this is another one that I
173. 6:18want us to demo here. I think the whole
174. 6:20idea of scoring retrieve trunks and
175. 6:23grading on answers against a rubric,
176. 6:25this can be a huge win for anything that
177. 6:28you're doing with rag or anything you're
178. 6:30doing where you're injecting context
179. 6:32into the context window. Fifth up, we've
180. 6:34got things like triage. So, this is all
181. 6:36your things like support tickets, email
182. 6:38issues, and one call here can give a
183. 6:41proper probability of how urgent
184. 6:43something is. I start to help giving
185. 6:45things an actual probability of priority
186. 6:48here. And this one can be really useful
187. 6:50for when to basically get the model to
188. 6:51go back to the human for things like
189. 6:53human in the loop as well. And the sixth
190. 6:55use that I'm not going to go into too
191. 6:57much here is the whole idea of just
192. 6:58real-time filtering. This can be really
193. 7:01useful for anytime you're basically
194. 7:02dealing with APIs, polling requests,
195. 7:05anything where it's good to just filter
196. 7:07what comes back from that before you
197. 7:10inject that into the context window. So,
198. 7:12the rule of thumb that I use for all of
199. 7:14these actually comes from Type Safe
200. 7:16themselves. If a panel of smart people
201. 7:19could answer the question in just a few
202. 7:21seconds, then it's the kind of question
203. 7:23that you probably should be using a
204. 7:24decision model for. If they need to go
205. 7:27away and formulate an answer and then
206. 7:29write a complicated answer back, that's
207. 7:31when you should be sticking with the
208. 7:32LLMs. In a little bit, I'll actually go
209. 7:34through the use cases of where you
210. 7:36shouldn't be using these decision models
211. 7:38at all. So, how do you actually wire
212. 7:40this in? So, if we go back to that loop
213. 7:43that we had there, there are four places
214. 7:45to hook a decision in. And really,
215. 7:47they're going to be the same in any
216. 7:49framework, whether you're talking about
217. 7:50LangChain, whether you're talking about
218. 7:52ADK, whether you're just using custom
219. 7:54Python code to write your own loop here.
220. 7:56You've got before the prompt is built,
221. 7:58so decide which skills or tools to load,
222. 8:02before the model called, decide which
223. 8:04actual model you're going to call,
224. 8:06before a tool runs, decide whether it's
225. 8:08safe, and then after a result comes
226. 8:11back, decide whether it's good enough to
227. 8:13use or whether you should go back and
228. 8:15actually go through the loop again. So,
229. 8:17each hook here is the same small piece
230. 8:18of code. Build a state from what the
231. 8:21loop already has, ask your questions,
232. 8:24and then read the probabilities and
233. 8:25answers that come back, and then branch
234. 8:28or make your decision for the path going
235. 8:30forward based on those. Now, there's
236. 8:32also a second pattern that you can use,
237. 8:34where Jev is the loop's decider and the
238. 8:37LLM is the fallback. So, Jev picks the
239. 8:40tool or the answer, and only hands it to
240. 8:42the LLM when it's unsure or when
241. 8:44something needs text to be written. Now,
242. 8:46this pattern can actually cut more LLM
243. 8:47calls, but it's definitely a much bigger
244. 8:49change to your agent. But so far I found
245. 8:51with this one, it just doesn't seem to
246. 8:53work as well when you've got a lot more
247. 8:54sort of complicated things going on in
248. 8:56there. Anyway, for either of these
249. 8:58patterns, you don't need a framework.
250. 9:00That said, the first pattern that I
251. 9:01talked about here, LangChain has already
252. 9:03shipped that as middleware, and Pydantic
253. 9:06has shipped their own version by making
254. 9:07the type safe models something that you
255. 9:09can just plug in and use in the agent
256. 9:11loop. All right, so before I show you
257. 9:13the demos, let me just go through
258. 9:14quickly where I think this doesn't fit.
259. 9:17Clearly, anything where you need text
260. 9:19back, anything that needs any sort of
261. 9:22multi-step reasoning, and anything where
262. 9:24you're going to need to connect several
263. 9:26things together to generate an answer
264. 9:28out, all of those are still an LLM's
265. 9:31job. And in many ways, a lot of that
266. 9:33multi-hop stuff is still going to be the
267. 9:34stuff that you're going to use the LLM
268. 9:36for and then have your Piranhas process
269. 9:38that and keep passing it back to the
270. 9:40large LLM. The second thing you want to
271. 9:42think about here is size. So, if your
272. 9:45state is just getting too big, and then
273. 9:48you're trying to pass in 100,000 tokens
274. 9:51to get some kind of response back from
275. 9:53that context window, then Jev is not
276. 9:56going to be a good fit here. Type-safe's
277. 9:58own docs say that accuracy drops a lot
278. 10:00on long inputs. And so, it generally
279. 10:03seems to fit whether you're using the
280. 10:05Type-safe Jev decision model or one of
281. 10:07the open decision models. On top of
282. 10:10that, another thing, obviously, is that
283. 10:12Jev itself doesn't use images. Now, some
284. 10:14of the open models actually do use
285. 10:16images and use images really well. So,
286. 10:18at the moment, that's definitely a win
287. 10:20for the open decision models over the
288. 10:22proprietary models. All right, another
289. 10:24thing that doesn't work well are
290. 10:25compound questions. So, compound
291. 10:27questions are often something that the
292. 10:28decision models don't handle that well.
293. 10:31If you've got a yes-no question, okay,
294. 10:32that's fine, but if you've got something
295. 10:34that requires multiple questions in
296. 10:36there to be answered, then often the LLM
297. 10:39is going to do better for that kind of
298. 10:41thing. Another thing you want to be wary
299. 10:42about in here is, while the decision
300. 10:45models can be really good at spotting
301. 10:46sort of weak spots, you want to be
302. 10:48careful about how you always rely on
303. 10:50them. It does seem currently that a lot
304. 10:52of these models can be prompt injected
305. 10:54to basically have instructions that
306. 10:57actually move the results out of that.
307. 10:59So, often the way to get around that is
308. 11:01to basically have a question first that
309. 11:03asks, are there prompt injections in
310. 11:05here before you actually pass it on to
311. 11:07the real question that you wanted to
312. 11:08actually ask. The other thing I find,
313. 11:10too, is that you really want to think
314. 11:12about criteria for making the decisions.
315. 11:15So, if you've got very clear decision
316. 11:18spots where you're asking the same
317. 11:19questions, you can control the actual
318. 11:22criteria very well in situations like
319. 11:25that. If you're just asking some vague
320. 11:27question about the output from the LLM,
321. 11:30and you haven't got a nice clear
322. 11:32in there, I wouldn't necessarily be
323. 11:34trusting the LLM to write that in a way
324. 11:36that's going to make the whole system a
325. 11:38lot better. You kind of want to define
326. 11:40the instructions and the criteria
327. 11:41yourself, and then benchmark any changes
328. 11:44that you make to that over time. And
329. 11:47then of course lastly, if you're
330. 11:48actually running your agent locally, but
331. 11:50then you're using Jase in the cloud,
332. 11:52you've got to think about the whole fact
333. 11:53that it's a hosted API. And do you
334. 11:56really trust it for every state that
335. 11:58you're going to send to the cloud? Is it
336. 12:00okay that that leaves your machine? And
337. 12:02I think this is where the open decision
338. 12:03models have really come along so far
339. 12:05now. So, let's jump into the demos and
340. 12:07actually look at some examples of how
341. 12:09you could do this for a couple of these
342. 12:11key tasks and some of the challenges
343. 12:13that you might face along the way.
344. 12:15All right. So, in the demo here, I've
345. 12:17basically made a UI just so you can see
346. 12:19what's going on a lot easier, but these
347. 12:22are exactly the kind of decisions that
348. 12:23are going on in your agent. So, if we've
349. 12:26got a bunch of skills, more and more now
350. 12:28people are starting to use large amounts
351. 12:30of skills files. And remember the whole
352. 12:33thing that's really cool about skills is
353. 12:35that you've got progressive disclosure
354. 12:37in there. So, the idea here is that if
355. 12:40this is the actual skill file, and it
356. 12:42might actually be a lot longer going
357. 12:44through this, you only give the model
358. 12:46this after it's decided that okay, it
359. 12:50needs to basically use a coding write
360. 12:53tests skill for this. Then you would
361. 12:56actually go and give it this in the
362. 12:58context window. So, the different thing
363. 13:00that we're doing here is that rather
364. 13:02than just give it 50 skills in one shot,
365. 13:06we can see for each of these 10
366. 13:08different categories, we've got five
367. 13:10different skills in here. And if we're
368. 13:12injecting that in the system prompt for
369. 13:14every time we call the LLM, that's going
370. 13:16to burn a lot of tokens. So, the idea is
371. 13:20that the thing that we inject into the
372. 13:22LLM is just the categories of skills.
373. 13:27And then the Jev model itself will
374. 13:29basically take the task that we've got.
375. 13:32It will then decide which category to
376. 13:35pick from, then it will be given the
377. 13:37list of skills for that category, and it
378. 13:41will then choose the most appropriate
379. 13:43one to actually do this. So, let's look
380. 13:45at this. So, if the task is basically
381. 13:47someone saying, "Someone got an email
382. 13:48pretending to be our CEO asking for gift
383. 13:51cards, is this a hacking attack?" And
384. 13:54you can see I'm running this through one
385. 13:56of the bigger Jev models in here. But
386. 13:57actually, you can try it out on a lot of
387. 13:59the different Jev models. You can see if
388. 14:01I run this and route it, it goes through
389. 14:04and it basically first decides that the
390. 14:07category here is security and access.
391. 14:10So, it's done that as a choice. It's
392. 14:12basically returned this back. And then
393. 14:14in this category, it has a bunch of
394. 14:17these in here. And we can see it decided
395. 14:20investigate a security alert in here.
396. 14:23And sure enough, the description for
397. 14:24that is triage a suspicious login,
398. 14:26phishing report, or security alert in
399. 14:29there. So, you can see in this case what
400. 14:31we've actually done is a sort of cascade
401. 14:33classifier, where we've done one sort of
402. 14:35classifier first to basically pick out
403. 14:38the class of the category, and then
404. 14:40we've done a second one to actually pick
405. 14:42out the skill in this case. Because
406. 14:44we're running this locally, it's done it
407. 14:45super quick going through this. All
408. 14:48right, you can see that if I give it a
409. 14:50sort of general question here, we've got
410. 14:52certain things that we can actually do.
411. 14:54Let's route it through. Sure enough, in
412. 14:57this case, it's basically saying do web
413. 14:59research, fact-check a claim, right? It
414. 15:02probably could have also gone for search
415. 15:04the web, although I guess this does
416. 15:06actually make more sense in that it's
417. 15:08actually stating a fact and then asking
418. 15:10if it's true here. And from there, we
419. 15:12could load the skill file or use any
420. 15:14sort of API, etc., that we actually
421. 15:16wanted to there. All right, one more in
422. 15:18here. You can see, "Okay, make me a
423. 15:19short voiceover of this product
424. 15:22description for a launch video. Now it's
425. 15:24deciding that okay, that relates to
426. 15:26images, audio, and video, and it needs
427. 15:28to be a text-to-speech skill in there.
428. 15:30So, this is a way of using Jev for sort
429. 15:32of progressive disclosure with skills,
430. 15:35especially if you've got lots of skills.
431. 15:38By classifying them first into
432. 15:39categories and then putting them into
433. 15:42actually which skill is being chosen,
434. 15:45you're able to get a very fast result
435. 15:47here, and then the skill itself can
436. 15:48actually be sent to the large language
437. 15:50model to work out any arguments that
438. 15:52need to go into that skill, those kinds
439. 15:55of things. But, this is able to help us
440. 15:57reduce on token spend, but also just
441. 16:00speed things up like this. Another one
442. 16:02that you can use in the same sort of
443. 16:04vein of this is doing sort of
444. 16:05verification steps of things that come
445. 16:08back from APIs. To basically have the
446. 16:10whole thing make a decision of whether
447. 16:12whatever's coming back from the API is
448. 16:15really good and can just go through
449. 16:16straight through as a result, whether
450. 16:18you need to change the arguments that
451. 16:20you gave to the API, or whether you just
452. 16:22need to call it again because you got a
453. 16:23certain type of error, etc. The next
454. 16:26demo is combining this with rag. So, one
455. 16:28of the patterns in rag that works really
456. 16:30well is go and get a lot of information,
457. 16:33and then re-rank that information. And
458. 16:36generally, there are ways to do that
459. 16:37where you've got a separate re-ranker
460. 16:38model. And that can work really nice as
461. 16:40a pattern. The only challenge is that
462. 16:43for certain projects, you just can't do
463. 16:45that because of time, you can't do that
464. 16:47because you're going to need more
465. 16:48models, and especially if you're already
466. 16:50using an embedding model on an API, you
467. 16:53probably don't want to have lots of
468. 16:55calls to the API for re-ranking, for
469. 16:58that kind of thing. So, the idea here is
470. 17:00that we can ask it a question, it will
471. 17:02then go off, get a lot of different
472. 17:05passages, then give those to the actual
473. 17:09Jev model, which can decide, okay, these
474. 17:12are the top five passages that I'm going
475. 17:14to send to the full-size LLM. And there
476. 17:16are different ways you could do this.
477. 17:17You could basically do it on a sort of
478. 17:18yes no thing. You could do it via sort
479. 17:21of scoring kind of things. Or if you
480. 17:23really wanted to just sort of pick one,
481. 17:25you could actually just have a choice of
482. 17:27one to see, okay, what's actually going
483. 17:29to go through in there. Now, for me
484. 17:31playing around with this, I find the
485. 17:32scoring one seems to work quite nicely
486. 17:34here. We then basically get a score. So,
487. 17:37in our question here is, why are my
488. 17:38webhook requests failing with an error?
489. 17:41And then we can see, sure enough, we're
490. 17:42getting back webhooks verifying
491. 17:43signatures, second up webhooks raw
492. 17:46bodies, third up was proxies, fourth up
493. 17:49was webhooks signing secrets. And you
494. 17:51can see that we've got the five things
495. 17:53now that we can basically send back. And
496. 17:56this is something that you could even
497. 17:57use if you weren't using an embedding
498. 17:59model. If you're just using something
499. 18:00like BM25 to basically get different
500. 18:04sections out, you can kind of get a bit
501. 18:06of the best of both worlds by having
502. 18:08sort of a broad sweep that gets you 25
503. 18:11different passages back, and then using
504. 18:13the Jev model, again in this case using
505. 18:16one of the open local ones here, to
506. 18:19actually go through and pick out the top
507. 18:21five that you then send to your LLM. You
508. 18:24can see if I try it with one of the
509. 18:25other ones that it's then able to
510. 18:27basically go through, get out different
511. 18:29parts of the context, pick out stuff
512. 18:32that's most relevant, and then pass that
513. 18:34through. Now, these are certainly larger
514. 18:37chunks of text than what I had with the
515. 18:39skills before. So, it's perhaps not
516. 18:41quite as fast as the cascading system
517. 18:43that we had for the last one, but
518. 18:46fundamentally the same idea of where
519. 18:48you're doing scoring and classification
520. 18:50using the world knowledge that's in that
521. 18:52particular decision model to help it
522. 18:55make simple but useful decisions for
523. 18:58you. All right, so just to finish up,
524. 18:59you can see that there are lots of uses
525. 19:02of where you can use either the Jev
526. 19:04model itself or the many open versions
527. 19:07of this in building any sort of custom
528. 19:10harness. And again, if you're interested
529. 19:12to basically think of these things more,
530. 19:14come through and read some of the Jev
531. 19:16tutorials. This OpenRouter has got one
532. 19:18up here. And when I'm actually using the
533. 19:20Jev model, I'm going through OpenRouter.
534. 19:22So, for those of you who are asking, if
535. 19:24you haven't been whitelisted for the
536. 19:26actual type-safe API on their site,
537. 19:29anyone can just come and use the
538. 19:30OpenRouter one here. Another useful
539. 19:32resource that's worth checking out is
540. 19:34LangChain has got a nice article about
541. 19:36using Jev in LangChain, about different
542. 19:39uses and stuff like that. They've
543. 19:41actually made some middleware for doing
544. 19:43this. But overall, I would say just try
545. 19:45it out. If you're doing any sort of
546. 19:47custom harness or any sort of
547. 19:49modifications to an harness, it's
548. 19:51definitely worth it looking at putting
549. 19:53this in your pipeline to be able to get
550. 19:56these really fast and really cheap
551. 19:58decisions made for improving the results
552. 20:00that your harness gets overall. Anyway,
553. 20:02as always, let me know in the comments
554. 20:03what you've tried out. I'm sure there
555. 20:05are plenty of examples that I haven't
556. 20:07put into this video. So, if you've got
557. 20:08some interesting use cases, please share
558. 20:10them cuz that would be really to check
559. 20:12out and see what other interesting ideas
560. 20:15people have got for how to actually use
561. 20:16this kind of thing. And as always, if
562. 20:18you found the video useful, please click
563. 20:20like and subscribe, and I will talk to
564. 20:22you in the next video. Bye for now.