YouTube2Text

Jev + Claude = INSANE Results (Every Concept Explained) — Transcript

by Simon Scrapes · 8,870 words · 1,256 segments · language en · Watch on YouTube

Full transcript

  1. 0:00The most talked about AI model this
  2. 0:01month can't even write a sentence. You
  3. 0:03can't chat with it, and it can't explain
  4. 0:05itself. And this is what people built
  5. 0:07with it in the first week it was
  6. 0:09released. We've got a browser that fills
  7. 0:10in Google Flights in about 7 seconds.
  8. 0:12And if you've ever tried to use Claude
  9. 0:14with a browser, by the way, you'll know
  10. 0:15that it's incredibly slow. We've got 3
  11. 0:17million website sessions capturing lost
  12. 0:20customers, abandoned checkouts, and more
  13. 0:22sorted in 40 seconds for about $2. And
  14. 0:25then we've got someone else driving
  15. 0:26Chrome with their voice, where it even
  16. 0:28takes actions before they finished their
  17. 0:30full sentence. So, it's called Jev, and
  18. 0:32I've been through all the official docs
  19. 0:34so that you don't have to. And by the
  20. 0:36end of this, you'll know what it is, why
  21. 0:38it matters if you use Claude already,
  22. 0:40how to run it from Claude code and ask
  23. 0:43it the right questions, as well as how
  24. 0:45to get the most out of it. So, let's
  25. 0:47start by talking about what's different
  26. 0:48about Jev, because some of this isn't
  27. 0:50new. So, every piece of software you
  28. 0:53already use is already full of if
  29. 0:55statements. For example, if an order is
  30. 0:57over $10,000, we're going to flag it for
  31. 0:59review. The catch, though, is that an if
  32. 1:01statement can only check things a
  33. 1:03computer can measure. So, we're
  34. 1:04comparing it to a number, a date, a tick
  35. 1:06box, a criteria. But, it can't check
  36. 1:08whether an email sounds angry or whether
  37. 1:10an order looks dodgy, because those
  38. 1:12aren't numbers. They aren't checkable
  39. 1:13things. They're not deterministic. So,
  40. 1:14Jev is doing exactly that. It's turning
  41. 1:16those messy inputs from customers into
  42. 1:19numbers. So, the if statement then
  43. 1:20becomes, "If this order looks
  44. 1:22suspicious, okay, we're 95% sure, flag
  45. 1:25it for review." So, now it can be read,
  46. 1:27and a decision can be made by a
  47. 1:29computer. And we'll talk about how you
  48. 1:30determine what counts as suspicious,
  49. 1:32because we're going to use Claude and
  50. 1:33you to write the questions, and you are
  51. 1:35going to set the criteria for yes, the
  52. 1:3795% confidence level, for example. So,
  53. 1:39we're going to send a bunch of questions
  54. 1:40to Jev. Jev is going to answer each one
  55. 1:42with how sure it is, and your code only
  56. 1:45flags the order when all of them come
  57. 1:47back above that threshold that you've
  58. 1:48chosen, so that 95%. And this might feel
  59. 1:51a little bit different to your
  60. 1:51traditional LLMs, like your Anthropic
  61. 1:54chat models. And that's why Type-Safe,
  62. 1:56the company behind Jev, call it a
  63. 1:58judgment model. So, it was built by a
  64. 1:59founder, Diogo, who helped build the
  65. 2:01methods behind ChatGPT at OpenAI. So,
  66. 2:04this is a massive announcement. And if
  67. 2:06you compare it to something like Claude,
  68. 2:07on the surface, you can absolutely ask
  69. 2:09Claude, "Does this order email look
  70. 2:11suspicious?" And you're going to get a
  71. 2:12text response from Claude like, "You're
  72. 2:14absolutely right. This does look
  73. 2:16suspicious. Would you like me to draft
  74. 2:17an email to the customer?" It's going to
  75. 2:18take about 10 to 15 seconds to get to
  76. 2:20that point. Versus Jev, on the other
  77. 2:21hand, who's given a bunch of questions
  78. 2:24to look for up from. The answers to
  79. 2:25those questions are going to determine
  80. 2:27how suspicious it actually is. Like, is
  81. 2:29there an address mismatch between the
  82. 2:31billing address and the delivery
  83. 2:32address? And then it's given the
  84. 2:33information it's actually processing.
  85. 2:35This is called the state, and in this
  86. 2:37case it would just be the individual
  87. 2:38order email that we're assessing. And
  88. 2:39then the response from Jev is not going
  89. 2:41to be a text response. It's only ever
  90. 2:43going to give you a number from the
  91. 2:44options that you've defined. And the
  92. 2:46number is going to tell you the
  93. 2:46likelihood of it being suspicious
  94. 2:49against the criteria that you defined.
  95. 2:51So, it's never going to write an output
  96. 2:53like Claude. And there are certain
  97. 2:54advantages to this that we'll cover in a
  98. 2:56moment. And this is totally different
  99. 2:57from the chat interfaces that we're used
  100. 2:59to when we see things like LLMs, large
  101. 3:01language models. This is closer to these
  102. 3:03deterministic machine learning models
  103. 3:05that is another style or type of AI. And
  104. 3:07Type Safe specifically call this what
  105. 3:09they call a system one model, after the
  106. 3:11book Thinking Fast and Slow. And system
  107. 3:13one is basically a quick name for a fast
  108. 3:15gut call. Where system two is sitting
  109. 3:17down and working out the reasoning,
  110. 3:19right? So, Claude, ChatGPT, and every
  111. 3:21other reasoning model are system two,
  112. 3:23but Jev sits in system one. So, why is
  113. 3:25it so special then compared to
  114. 3:26everything else that we've seen today
  115. 3:28with our traditional LLM chat models?
  116. 3:31So, there's a few reasons why not
  117. 3:32writing as the output is advantageous in
  118. 3:35some use cases. Firstly, it's cheap and
  119. 3:38extremely fast. So, it's suggested that
  120. 3:40it's 42,000 times cheaper than
  121. 3:42traditional LLMs and 20 to 400 times
  122. 3:45faster. It's about 4 cents per million
  123. 3:48tokens in, and the output is practically
  124. 3:51free. The second reason it's
  125. 3:52advantageous is it can't make up an
  126. 3:54answer. So, if you give it three
  127. 3:55options, it comes back with one of those
  128. 3:57three options every time. So, this is
  129. 3:59actually a pro and con, which we'll come
  130. 4:00to later in the specific rules that you
  131. 4:02need to give Jev when you're operating
  132. 4:03with it. The third advantage then is it
  133. 4:05that gives the same answer or almost the
  134. 4:07same answer twice in a row. So, it's
  135. 4:09much more consistent than LLMs in
  136. 4:11answering the same question over the
  137. 4:12same data set. So, the same test, they
  138. 4:14ask the same questions over and over,
  139. 4:15and the answers or the output from Jev
  140. 4:17moved less between runs than any of the
  141. 4:20chat models they tried. Even with the
  142. 4:21prompt set to be as repeatable as
  143. 4:23possible with the other models. And then
  144. 4:24the entire point of this is the number
  145. 4:26that Jev gives you as the output makes
  146. 4:28something that's usually uncertain, like
  147. 4:31a fuzzy customer email that's messy,
  148. 4:33it's got lots of text, and it makes it
  149. 4:35actually usable by software. So, Type
  150. 4:37Safe have trained this model so that the
  151. 4:38probabilities are calibrated. So, in
  152. 4:40plain English, it basically means it
  153. 4:42works like a weather forecast. So, when
  154. 4:44Jev says 0.8 is the output, it should be
  155. 4:46right about eight times out of 10. And
  156. 4:48based on that eight times out of 10 or
  157. 4:49the confidence in the output, that's
  158. 4:51what lets the software decide whether to
  159. 4:53act on it or not, whether to route that
  160. 4:54query somewhere else. The next actions
  161. 4:56are decided based on that number, and
  162. 4:58Jev gives you the probability of that
  163. 5:00specific question being true or false.
  164. 5:02And we'll see the different types of
  165. 5:03options we're able to give to Jev in a
  166. 5:05little while. Compare that to a chat LLM
  167. 5:07model which are trained to sound right
  168. 5:10to a specific person, and that's why we
  169. 5:11have that bias where it agrees with
  170. 5:13everything we said. Jev is not built in
  171. 5:15the same way. Now, super important, if
  172. 5:17you already use Claude or something like
  173. 5:19GPT, it does not replace it. It's going
  174. 5:22to work alongside it, and it's going to
  175. 5:23add advantages to certain workflows. So,
  176. 5:25Jev replaces one entire category of work
  177. 5:28that Claude does today. And whenever I
  178. 5:30say Claude, you can, you know, replace
  179. 5:31that with any of the LLM providers that
  180. 5:33you want here. And that category of work
  181. 5:35is going to be the fast decisions that
  182. 5:37we need to make in bulk inside software.
  183. 5:39So, the code that you're going to pass
  184. 5:40to Jev hands Jev that information, like
  185. 5:43our customer email, and a set of tightly
  186. 5:45defined questions which we want to know
  187. 5:46the answer to, and Jev is going to send
  188. 5:48back a probability, a category, or a
  189. 5:50score. And then your code, off the back
  190. 5:52of that, is going to use that to route
  191. 5:54it, to rank it in order, to flag it to a
  192. 5:57human, or to carry on. So, it's going to
  193. 5:59take actions off the back of that
  194. 6:00potentially. But, everything that needs
  195. 6:02a conversation, a chat back and forth,
  196. 6:04or reasoning, or writing is still going
  197. 6:06to stay within our Claude interface. And
  198. 6:08Diogo, the founder, specifically said on
  199. 6:10the Latent Space podcast that Claude is
  200. 6:12the conversation, and Jev is the
  201. 6:14plumbing underneath that. And it's
  202. 6:16important to actually reflect on the
  203. 6:17fact it cannot do everything, and in
  204. 6:18fact, it can do very few things. But,
  205. 6:20the things that it can do, it can do
  206. 6:22them really, really well. So, we
  207. 6:24mentioned earlier that each question it
  208. 6:25answers can be split into one of three
  209. 6:27types. Now, let's run through each of
  210. 6:29those types using that angry customer
  211. 6:30example again for ease. So, first, we
  212. 6:33have yes-no questions. These are called
  213. 6:35a null, and this might be, "Does this
  214. 6:37customer sound angry?" Let's say Jev
  215. 6:39gives us a 0.9 back. So, it reckons it's
  216. 6:42a 90% chance of yes, this customer is
  217. 6:44angry. And then your code, your script,
  218. 6:47important to note this isn't Jev, this
  219. 6:48isn't your LLM, it's a script that we
  220. 6:50can get Claude to write, then does the
  221. 6:52rest with a normal if statement. So, if
  222. 6:55the result is above, let's say 0.8,
  223. 6:57we're going to escalate that to a
  224. 6:58manager. So, the manager can then
  225. 6:59directly respond to that customer. Or if
  226. 7:01it's below 0.2, i.e., they're definitely
  227. 7:03not angry, or Jev has deemed them
  228. 7:04definitely not angry, then we're not
  229. 7:06going to escalate it, we're going to
  230. 7:07deprioritize it. But, it's important to
  231. 7:09understand that Jev never actually
  232. 7:10escalates anything, it just gives you
  233. 7:12the number, which represents the
  234. 7:14probability of it being a yes in this
  235. 7:16case. Where 0.9 is a confident yes, the
  236. 7:18customer is angry, and 0.5 means it has
  237. 7:20no idea, it's like a coin flip. And then
  238. 7:22the code or the script is going to take
  239. 7:24the decision based on that number it
  240. 7:25gets back. The second type we can give
  241. 7:27it is called a choice. It's exactly as
  242. 7:29it sounds. You give it categories that
  243. 7:31you predefine, and in the angry customer
  244. 7:33case, it's like, "Which team should
  245. 7:34handle this query? Should it be billing,
  246. 7:37technical, sales?" And it's going to
  247. 7:39come back with a probability for every
  248. 7:40single one of those options. It's not
  249. 7:42going to say yes, it's billing. Yes,
  250. 7:43it's sales. It's actually going to say
  251. 7:45billing was an 85% chance of being
  252. 7:47correct. Technical was a 10% chance.
  253. 7:50Sales with a 5% chance. And the code off
  254. 7:52the back of those results is then going
  255. 7:54to route the email to the winner. So in
  256. 7:56this case it's going to be billing. So
  257. 7:57you're going to use choice when you're
  258. 7:59routing the output somewhere. And when
  259. 8:01the options do have an order, we're
  260. 8:02going to use the third type, which we'll
  261. 8:03talk about in a second. Now because
  262. 8:05we've got more than one option there,
  263. 8:06we've got billing, technical, and sales,
  264. 8:08you actually get a second number
  265. 8:09returned by Jev called confidence. And
  266. 8:11that's worked out from how spread out
  267. 8:13those probabilities are. So the 85% for
  268. 8:15billing is confident. But let's say
  269. 8:17billing was at 40, technical at 35,
  270. 8:19sales at 25, then we're less confident
  271. 8:21about that or Jev is less confident.
  272. 8:23Billing is still going to be the chosen
  273. 8:24one, but the confidence is going to come
  274. 8:25back low. And that's the number your
  275. 8:27code is going to check before it trusts
  276. 8:28that pick in the first place. So it's
  277. 8:30going to check that confidence number
  278. 8:31too. And that's exactly what the browser
  279. 8:32demo that we showed at the start is
  280. 8:34doing. Every button and box on that page
  281. 8:36for the flight booking is going to get a
  282. 8:37number. And Jev answers two choices at
  283. 8:40once. What to do with it? So do we
  284. 8:42click, type, or scroll based on the
  285. 8:45action or the input from the voice? And
  286. 8:46which number to do it to? And we've
  287. 8:48basically ordered everything on the page
  288. 8:50to get a number. And we're going to pick
  289. 8:51element three or four or seven. And
  290. 8:53there's no order to either of those. And
  291. 8:55so we know that this type is going to be
  292. 8:56a choice. And the confidence then is
  293. 8:58going to be that safety catch. So if Jev
  294. 9:00is confident, i.e. puts it above that
  295. 9:02confidence level, then the click is
  296. 9:04going to happen. With below it, the
  297. 9:05agent is going to stop and ask the user.
  298. 9:07And you can raise or lower the threshold
  299. 9:08for confidence and what is acceptable
  300. 9:10based on a specific action. If it's
  301. 9:12something that's going to be detrimental
  302. 9:13or highly costly, then we need a higher
  303. 9:15confidence level to permit an action or
  304. 9:18a higher safety catch effectively. And
  305. 9:20the third one then is a score. So it's
  306. 9:21the same idea as choice, but the options
  307. 9:23have an order to them. And you describe
  308. 9:25each one in words. So the question might
  309. 9:26be how angry is this customer? And the
  310. 9:29results are going to be a score. And the
  311. 9:30score is going to be between calm,
  312. 9:32annoyed, and furious. Those are the
  313. 9:34three options. So Jev is going to come
  314. 9:35back with a position on that scale
  315. 9:37between calm, annoyed, and furious. And
  316. 9:39it can actually land between the two
  317. 9:40levels. So, 2.4 out of 3 means they're
  318. 9:42mostly annoyed because that was the
  319. 9:43second option, but they're edging
  320. 9:45towards furious. So, when we're sorting
  321. 9:46an entire inbox of customer service
  322. 9:48emails, we can actually rank them. And
  323. 9:50this is what our code does or our
  324. 9:51script. The inbox by the worst first.
  325. 9:54So, the three out of threes on top, the
  326. 9:55null out of threes or one out of threes
  327. 9:57right at the bottom. So, you'd use the
  328. 9:58score then on anything you'd put on a
  329. 10:00scale. This is why it's important
  330. 10:01between choice and score that there is
  331. 10:03an order to things here. So, it's like
  332. 10:05how urgent is this ticket? How good a
  333. 10:06fit is this lead for what we actually
  334. 10:08sell? How senior is this candidate on
  335. 10:10this scale? How bad is this bug? Is it
  336. 10:12cosmetic bug? Is it an annoying bug? Or
  337. 10:14is it a blocking bug for somebody taking
  338. 10:16an action on our page? So, use it when
  339. 10:17you want to rank or sort and not root a
  340. 10:20response necessarily like choice. And
  341. 10:22because it's got several different
  342. 10:23levels, it also comes with a confidence
  343. 10:25the same as choice does, but null does
  344. 10:27not come with a confidence. It's either
  345. 10:28yes or no. So, the website session demo
  346. 10:30I showed at the start of the video is
  347. 10:31mostly that first type, a yes or no per
  348. 10:34event. Was this person rage clicking?
  349. 10:36Was this person dead clicking? Did
  350. 10:38something error on the page? Then we
  351. 10:40actually add this score as a layer on
  352. 10:42top of that. So, how bad was this whole
  353. 10:44session for this user? So, that when
  354. 10:46we're sorting those sessions, we can
  355. 10:47come back and actually sort them from
  356. 10:49worst first, and a person then can only
  357. 10:51open up the top 10. Now, you can of
  358. 10:53course do this whole process in a large
  359. 10:55language model, but it's not optimized
  360. 10:57for making these probabilistic
  361. 10:58decisions, and therefore we can do an
  362. 11:00astounding amount of them in a very
  363. 11:02short space of time for very, very
  364. 11:04cheap. And the way to choose between
  365. 11:06these three different methods, the null,
  366. 11:07the choice, or the score, is working out
  367. 11:10what your code does with the answer
  368. 11:12afterwards. So, if it needs to act or
  369. 11:14not act based on the answer, that's
  370. 11:15going to be a yes or no or a null. If it
  371. 11:17needs to root the response somewhere,
  372. 11:18that's going to be a choice. Or if we
  373. 11:20need to sort or rank by it, then that's
  374. 11:22going to be a score. And most
  375. 11:23importantly, all three of these types
  376. 11:24can actually go into one request to Jev.
  377. 11:26So, here's an example of one email going
  378. 11:28in with all three questions attached,
  379. 11:30and this is what comes back. So, we've
  380. 11:32got a null first, a question that says,
  381. 11:33"Does this customer sound angry?" We've
  382. 11:35got the null 0.9. So, basically it's
  383. 11:36saying for the code to escalate this to
  384. 11:39a member of staff. Then we've got the
  385. 11:40choice, which team is this? And it says,
  386. 11:42"Billing is 85 likely, technical 10,
  387. 11:44sales five." And we've got a high
  388. 11:46confidence level. So, it's going to
  389. 11:47route to the billing team. And then the
  390. 11:49score, how calm, angry, annoyed, or
  391. 11:51furious on a scale of four here. And
  392. 11:53we've got a 2.4 with confidence of 80%
  393. 11:56or 0.8. So, it's going to go near the
  394. 11:58top of the pile for the emails that the
  395. 11:59billing team are going to answer. And
  396. 12:01they're only going to see the ones where
  397. 12:02the customer actually sounds angry. So,
  398. 12:04they've got a short list of answers
  399. 12:06because Jev has actually processed a
  400. 12:07huge amount that don't need to go to a
  401. 12:09customer service rep. So, we've got one
  402. 12:10email in, but three answers out in a
  403. 12:12very, very short space of time. So, now
  404. 12:14let's move on to how to actually pair
  405. 12:16this with Claude code so that you can
  406. 12:18use both tools to their full advantages.
  407. 12:20But first, I'm about 1,000 subs from
  408. 12:22that magic 100k and that plaque on my
  409. 12:25wall. So, if you're enjoying this so
  410. 12:26far, make sure to sub down below. So,
  411. 12:28anyway, there's two ways to pair it with
  412. 12:30Claude to get the best out of both
  413. 12:31tools. And Type Safe are really clear in
  414. 12:33their docs about what it's bad at and
  415. 12:35therefore where you should use Claude.
  416. 12:37And that's reasoning, specialist
  417. 12:38knowledge, and anything that effectively
  418. 12:40needs writing. So, the first way in
  419. 12:42which you can pair it with Claude is
  420. 12:43actually using it around Claude's
  421. 12:45features for checking things. So, before
  422. 12:47Claude reads something, you're going to
  423. 12:49get Jev to decide whether Claude should
  424. 12:50actually read it, whether you should
  425. 12:51spend tokens on that thing. So, Type
  426. 12:54Safe use a guardrails example, which
  427. 12:56screens every single message that comes
  428. 12:58into Claude for jailbreaks and harmful
  429. 13:00requests on the way into Claude. It's
  430. 13:02then going to screen Claude's reply on
  431. 13:04the way out before the user sees it. So,
  432. 13:07that's a good example of Claude in the
  433. 13:08middle, but using Jev as guardrails on
  434. 13:10the before and after. And you can also
  435. 13:11use the same idea after the LLM
  436. 13:13processing. So, in their citation
  437. 13:14example, an LLM wrote a document with
  438. 13:16eight citations in it, and Jev checked
  439. 13:18each quote against the source. Four were
  440. 13:20found to be fine, one quote was a
  441. 13:23non-existent citation, and one said the
  442. 13:25actual opposite, and two were actually
  443. 13:26rooted to a person because Jev wasn't
  444. 13:28sure. So, that is like post-processing
  445. 13:30of Claude and LLM content gone through
  446. 13:32Jev to make sure that everything they
  447. 13:34were putting out was factually accurate.
  448. 13:36And the second way you can use it with
  449. 13:38Claude is actually using it instead of
  450. 13:40Claude for the bulk work, for the tasks
  451. 13:42that are larger in scale, but also very
  452. 13:45deterministic. So, don't use Claude to
  453. 13:47sort 5,000 rows, tagging every ticket in
  454. 13:49your inbox, scoring every single lead.
  455. 13:52Claude should only get the rows that Jev
  456. 13:53wasn't actually sure about or the ones
  457. 13:55that need something written. If it's
  458. 13:56that deterministic scoring, then we
  459. 13:58should use Jev instead of Claude for
  460. 14:00that stage in our workflow. And this is
  461. 14:02the one that you've probably seen all
  462. 14:03over X or Twitter. Here's someone
  463. 14:05scoring 700 sales leads in 40 seconds
  464. 14:08for around 9 cents. So, something that
  465. 14:10is bulk work would take significantly
  466. 14:12longer using Claude and also cost a lot
  467. 14:14more, too. And actually both of those
  468. 14:15examples we're going to build together
  469. 14:17in the next 5 or 10 minutes using a
  470. 14:18customer service example. So, let's get
  471. 14:20that set up inside Claude Code now. So,
  472. 14:23Type Safe actually shipped a skill for
  473. 14:25use in LLMs like Claude Code or
  474. 14:26harnesses like Claude Code, and it's two
  475. 14:28lines to install. So, you can either do
  476. 14:29this through the console once you've
  477. 14:31signed up on typesafe.ai,
  478. 14:33you can copy this agent prompt, or you
  479. 14:35can actually head to the
  480. 14:36docs.typesafe.ai/agent-skill,
  481. 14:39and you're going to run these two
  482. 14:40commands in your terminal to get it
  483. 14:42inside to Claude Code. And you can also,
  484. 14:43you know, take these commands directly
  485. 14:45from here or use it with other agents
  486. 14:46there, too. I recommend to actually use
  487. 14:48this that you need to make sure that you
  488. 14:49name the skill in your prompt. So, when
  489. 14:51you're sending a command to Jev, for
  490. 14:53example, we're going to say use the Type
  491. 14:54Safe skill, and you'll see that in all
  492. 14:56of our prompts. So, we're going to run
  493. 14:57those two commands. It's going to add
  494. 14:58the marketplace. We're then going to
  495. 14:59reload to make sure that the plugin's
  496. 15:01installed. And you can see that that's
  497. 15:02now a success. So, once we reload or
  498. 15:04load up a new session with Claude Code,
  499. 15:05we can run /plugins, and we can see, if
  500. 15:08we go to installed, that we've got the
  501. 15:10Type Safe plugin now enabled. You're
  502. 15:12going to head into the API key section,
  503. 15:14create a new key, and you're going to
  504. 15:15put that in a .env file under the name
  505. 15:17Type Safe API key. Then to check it's
  506. 15:19working, we're going to say use the Type
  507. 15:20Safe skill. That's really important.
  508. 15:22Check my key works. Ask Jev one yes or
  509. 15:24no question about and this is our input
  510. 15:27as if it's a customer email, right?
  511. 15:28We've been charged twice. Please fix
  512. 15:30this ASAP. And we're going to ask Jev to
  513. 15:32assess it on the question, the customer
  514. 15:34is reporting a billing problem, and to
  515. 15:36tell us the raw answer from Jev that we
  516. 15:38get back. So, with this, we should just
  517. 15:39get a yes or no in the form of a number
  518. 15:41that that deems is this a billing
  519. 15:44problem or is it likely to be a billing
  520. 15:45problem? Now, before we run the output,
  521. 15:47if you're not working in Claude code
  522. 15:48directly, and you actually want to work
  523. 15:49in the desktop app, I haven't tested
  524. 15:51this yet, but it should work exactly the
  525. 15:52same. All you need to do is go to find
  526. 15:55the skill.md on GitHub. You're going to
  527. 15:56open this page and then effectively
  528. 15:57download and upload the skills in the
  529. 16:00customize section inside your skills.
  530. 16:02So, you're going to add upload the file
  531. 16:03and then you should also have access
  532. 16:04inside the Claude desktop app to using
  533. 16:06Jev as well. So, let's run this prompt.
  534. 16:08We're going to use the type-safe skill
  535. 16:09and let's see what comes back. So, you
  536. 16:10can see here, this is the request based
  537. 16:12on the type-safe skill that it's
  538. 16:13actually determined. So, the state is
  539. 16:14the information we're sending, charged
  540. 16:15twice, please fix this ASAP. That's the
  541. 16:17email we're sending it to Jev latest.
  542. 16:19The question is, is it a billing
  543. 16:20problem? It's deemed that it's a yes or
  544. 16:21no and given it a descriptive
  545. 16:23instructions. So, the customer is
  546. 16:24reporting a billing problem, exactly the
  547. 16:26criteria that we wanted to assess
  548. 16:28against. The true criteria is the
  549. 16:29message reports a problem with charges,
  550. 16:31payments, invoices, refunds, or
  551. 16:32subscription billing. The false is the
  552. 16:34message does not report a billing
  553. 16:35problem. So, you can see that these are
  554. 16:37very descriptive criteria that Jev can
  555. 16:39therefore determine is a message or the
  556. 16:41state likely to be a yes or no against
  557. 16:44that criteria. Claude has created this
  558. 16:46criteria based on the type-safe skill.
  559. 16:48So, it's come back with an answer then
  560. 16:49and the answer from Jev was effectively
  561. 16:51this one number, 0.99, where one means
  562. 16:54it was absolutely a billing problem and
  563. 16:56zero means it was not a billing problem
  564. 16:58at all. You can see that this is the
  565. 17:00response that we get directly from Jev.
  566. 17:02The input tokens were 324 tokens. The
  567. 17:04output tokens were 24. I also asked
  568. 17:06Claude to extract the time it took as
  569. 17:08well as the cost. Jev's own share of
  570. 17:10that was 0.3 seconds and that's probably
  571. 17:12an overestimate based on the fact that
  572. 17:13Claude actually had to make the
  573. 17:15connection in the first place. And 324
  574. 17:17input tokens is effectively 0.0000136
  575. 17:21dollars. So, you can see how it could
  576. 17:22have done this for thousands of emails,
  577. 17:24and it barely scratched the surface on a
  578. 17:26couple of cents. You'd need 73,000 calls
  579. 17:28like this to spend a dollar. Extremely
  580. 17:30cheap, extremely fast. And then the 24
  581. 17:32output tokens, which was effectively
  582. 17:34just telling us the number, cost nothing
  583. 17:36to us. And by the way, if we jump back
  584. 17:37to what this skill actually does, the
  585. 17:39type safe skill, it doesn't actually run
  586. 17:41anything itself. It's basically a rule
  587. 17:43book that Claude can use or another
  588. 17:44agent can use with the three question
  589. 17:46types. So, it's able to determine which
  590. 17:47question type it should be in, how to
  591. 17:49write a question Jev won't misread. So,
  592. 17:51it's about specificity of a question,
  593. 17:53when to batch questions together versus
  594. 17:55when to keep them separate, and we'll
  595. 17:56cover that later, and where to keep the
  596. 17:58thresholds, the confidence levels. So,
  597. 18:00if you're working from Claude, you never
  598. 18:01talk to Jev directly. You describe the
  599. 18:03job in plain English. Claude is going to
  600. 18:05read the skill. Claude's going to write
  601. 18:06the scripts to assess it against, and
  602. 18:08the script is going to send your rows to
  603. 18:09Jev, and Claude is going to read you the
  604. 18:11results back. So, this is how you use
  605. 18:13them in conjunction. And alternative
  606. 18:15right is you could use something like
  607. 18:16Open Router, skip Claude code as a
  608. 18:18harness here, and directly use the model
  609. 18:21from Open Router, the Jev Lab model from
  610. 18:23Open Router. But that would effectively
  611. 18:24mean that actually you need to make up
  612. 18:25the questions, you need to make up the
  613. 18:27descriptions, you need to effectively
  614. 18:29structure that input to Jev there. So,
  615. 18:31you've seen now that the key works.
  616. 18:32You've seen it's really quick. You can
  617. 18:33see we can scale this up. So, let's
  618. 18:35actually deconstruct how you would use
  619. 18:37this for yourself using an example of 50
  620. 18:39support emails that come in overnight.
  621. 18:41So, I've got this CSV or Google Sheet
  622. 18:43here of 50 different support emails,
  623. 18:45when they were received, the ID of the
  624. 18:46email, when that customer has been a
  625. 18:48customer since, which plan type they're
  626. 18:49on, what invoice they last paid, the
  627. 18:51subject, the body. So, everything that
  628. 18:53we'd need to pass into Jev. We're going
  629. 18:55to deconstruct on exactly how you would
  630. 18:57turn this into a workflow that sorts
  631. 18:59them so that the easy ones are answered,
  632. 19:00the risky ones are sent to a member of
  633. 19:02customer support, and that when they're
  634. 19:03queued, they're actually in priority
  635. 19:05order so they're ranked when somebody
  636. 19:07sits down in the morning to read these
  637. 19:08customer service messages. So, these
  638. 19:10emails are in a emails.csv that we made
  639. 19:12earlier, but equally you could if you're
  640. 19:14using a harness like Claude code, use
  641. 19:15the MCP to plug this directly into your
  642. 19:17email inbox and process this in the same
  643. 19:20way. So, you can see we've got examples
  644. 19:21like "Hi, I've just seen two charges of
  645. 19:22$99. I signed up for a trial and the
  646. 19:24magic link never arrived." A feature
  647. 19:26request here, "Can you add perplexity to
  648. 19:28it?" Some customer complaints here like
  649. 19:30"Metrics haven't been shown." or
  650. 19:32"Metrics have gone down." Some questions
  651. 19:33about GDPR compliance. All of that is
  652. 19:36effectively data that we can go through
  653. 19:38and get Jev to assess extremely quickly
  654. 19:40against the set of questions. Now, when
  655. 19:41deconstructing it, the most important
  656. 19:44thing before we write a single question
  657. 19:46is to decide what the code should do
  658. 19:48with each answer. So, the answers we're
  659. 19:50going to get back from Jev, the
  660. 19:51probabilities, the numbers, should all
  661. 19:52be a trigger for an action. So, every
  662. 19:55question that we make should be a
  663. 19:57trigger to a specific action. And we're
  664. 19:59going to talk about how to work out what
  665. 20:00questions to ask in the first place, and
  666. 20:02that's why we're going to go through and
  667. 20:03deconstruct this step-by-step. And when
  668. 20:05we consider the different actions that
  669. 20:06we will take off the back of the results
  670. 20:08in this customer support case, we've got
  671. 20:10four different actions effectively.
  672. 20:11We're going to answer the email with
  673. 20:13Claude directly or draft it with Claude.
  674. 20:14We're going to put it in the engineering
  675. 20:16queue as something that's a bug that
  676. 20:17needs fixing. We're going to log it as a
  677. 20:19feature request for the future. Or we're
  678. 20:21going to escalate it or send it to a
  679. 20:22person. So, that's then what I need to
  680. 20:24give to Claude, the job I want it to do,
  681. 20:26the four different actions or routes it
  682. 20:28can take, and one instruction, which is
  683. 20:30"Show me the plan before you run
  684. 20:32anything." And therefore, we're going to
  685. 20:33see effectively the structured plan that
  686. 20:35it's going to send to Jev, and we can go
  687. 20:37through and read before we do this on a
  688. 20:39bulk set of data piece by piece what the
  689. 20:42questions are, what set of data it's
  690. 20:43going to run on, etc. So, let's talk
  691. 20:45about the prompt, and I've been quite
  692. 20:47specific here, but you can actually get
  693. 20:48Claude, like I did, to write a similar
  694. 20:51prompt for your exact use case. So,
  695. 20:52again, we've got "Use the TypeSafe
  696. 20:54skill." Very important. "I've got 50
  697. 20:56support emails in emails.csv." I'm going
  698. 20:58to save it in my downloads. "Every
  699. 20:59morning I want them sorted into four
  700. 21:00piles, and these are our four actions,
  701. 21:02right? Reply with Claude, engineering
  702. 21:03queue, send to person, etc. Each pile
  703. 21:06sorted with the angriest customer first.
  704. 21:08So, from that and using the type safe
  705. 21:10skill, it's going to understand the list
  706. 21:12of priorities and what questions we need
  707. 21:13to ask. And then it's going to
  708. 21:15understand that we need a scoring
  709. 21:16question to score it by, you know,
  710. 21:18anger, for example. And because I've got
  711. 21:19some idea of exactly what I want to
  712. 21:21understand for each, I've actually
  713. 21:22specified it here, but you can leave
  714. 21:24this open for Claude to actually
  715. 21:25determine the questions and criteria,
  716. 21:26too. But for each email, I want to know
  717. 21:28what it's about. So, this might be a
  718. 21:29choice, billing, technical, feature
  719. 21:31request, account, or none of those. And
  720. 21:32it's really critical you put none of
  721. 21:34those in there because if it's not one
  722. 21:35of those, then Jev is effectively going
  723. 21:38to just try and pick the most likely
  724. 21:40outcome. Whereas if none of those is in
  725. 21:42there, it effectively gives it an out. I
  726. 21:44want to know if they're asking for money
  727. 21:45back or a refund. I want to know if
  728. 21:46something is broken because therefore we
  729. 21:48want to send it as the action to put it
  730. 21:51in the engineering queue. I want to know
  731. 21:52on a scale, probably, how angry they are
  732. 21:54from calm, annoyed, or furious, so like
  733. 21:56one to three. And I want to know whether
  734. 21:57it's just something quite routine and
  735. 21:59standard, needs some judgment, or is
  736. 22:00unusual enough that a person should
  737. 22:01handle it. And then I've added some
  738. 22:03rules for what the code does or what the
  739. 22:05code should do. Claude's code is going
  740. 22:06to create the code that's going to
  741. 22:07process Jev's answers. Anything about
  742. 22:10money should go to a person, never
  743. 22:11Claude. And I basically want a high
  744. 22:13confidence level on that one. If Jev
  745. 22:14isn't confident about the category or
  746. 22:17the case is unusual, send it to a
  747. 22:18person. If the product is broken, so
  748. 22:20they can't use it, it should go to
  749. 22:21engineering. Feature requests are
  750. 22:23getting logged. Everything else Claude
  751. 22:24drafts a reply using a a given reply
  752. 22:26guide, a separate brand resource,
  753. 22:28effectively. And we're going to sort
  754. 22:29every pile by a priority score that's
  755. 22:32mostly anger, partly severity. So, it
  756. 22:34needs to score based on multiple
  757. 22:36questions. And Claude, working with the
  758. 22:38type safe skill, is going to understand
  759. 22:39that it needs to break these down into
  760. 22:41individual deterministic or more
  761. 22:43deterministic questions that Jev can
  762. 22:44then give us a probability score on and
  763. 22:46a confidence score on. And then we
  764. 22:48specified this because we want to do an
  765. 22:49extra check. So, before any Claude draft
  766. 22:51is saved, check it with Jev in another
  767. 22:53request, a separate request, asking
  768. 22:55different questions against these
  769. 22:57questions. Does it actually answer the
  770. 22:58customer's question? And does it promise
  771. 23:00a refund or discount? So, we're doing an
  772. 23:01extra validation pass after Claude
  773. 23:05for the ones that are going to be
  774. 23:06written by Claude. And then we're only
  775. 23:07going to keep drafts that pass both.
  776. 23:09Write the plan in three sections, the
  777. 23:10state which is, you know, the input
  778. 23:11information, the questions with their
  779. 23:13criteria, and the routing with a
  780. 23:15threshold for every rule. So, this is
  781. 23:16the confidence level that passes that
  782. 23:18given rule. Keep that in one file. Show
  783. 23:20me the plan before you run anything. And
  784. 23:21you could just ask to see the scripts
  785. 23:22that it's going to send to Type Safe or
  786. 23:24Jev, but this way we can copy this,
  787. 23:26paste it into our Claude code instance
  788. 23:28with Type Safe, and actually start
  789. 23:29running that.
  790. 23:31And what it's going to do is effectively
  791. 23:32come back with a plan, right? So, at
  792. 23:34this point we're less concerned with how
  793. 23:36quick it's going to run because what
  794. 23:37we're actually just trying to do is come
  795. 23:38up with a good plan with a good set of
  796. 23:40questions so that we can run it
  797. 23:42effectively using Jev. Jev is only going
  798. 23:44to be really powerful for these bulk
  799. 23:45workflows and useful if you can ask good
  800. 23:47questions and Claude or an LLM is able
  801. 23:49to break down our requirements into
  802. 23:51those good questions. So, we've now got
  803. 23:53Claude's plan back in the questions.py
  804. 23:56file. And the first thing that we can
  805. 23:58see in Claude's plan is the state or
  806. 24:00what each email looks like when it
  807. 24:01reaches Jev. So, it's used the skill to
  808. 24:03determine what this state should look
  809. 24:05like. Now, you can see that we've got
  810. 24:06ID, received at, plan, subject, body,
  811. 24:08all these different fields. So, what it
  812. 24:10said is actually we need one object per
  813. 24:12email rather than a flat string where
  814. 24:15each part gets a descriptive name so the
  815. 24:16relationship stay clear. So, it's going
  816. 24:18to include customer facts like the plan
  817. 24:20or the customer they've or when they've
  818. 24:22been a customer since because the
  819. 24:24founding customer on the top plan is a
  820. 24:25completely different scenario from a
  821. 24:27day-old trial and Jev will want to treat
  822. 24:29that differently. So, it's basically
  823. 24:30mapped out the CSV columns here
  824. 24:33and given it some scripts. So, it's
  825. 24:34going to turn one CSV row into a state
  826. 24:36or input for a triage request. So, every
  827. 24:38single row is going to have information
  828. 24:41about the email which is going to be
  829. 24:42passed into Jev and information about
  830. 24:43the customer. So, it's going to be
  831. 24:44broken down into these specific
  832. 24:46sections. And that's because the skill
  833. 24:47told it to so that a question can point
  834. 24:50at a field by name and it's left
  835. 24:52everything that's irrelevant out of that
  836. 24:54like the previous thread history because
  837. 24:55the more irrelevant text it could
  838. 24:57potentially bring into the the
  839. 24:58decision-making process. And that brings
  840. 25:00us to kind of the limitation or the hard
  841. 25:02limit of 64,000 tokens per request that
  842. 25:05you send to Jev, but you should never be
  843. 25:07near that. It's not designed like an LLM
  844. 25:09where you have a million tokens in the
  845. 25:11input. It's it's very different. These
  846. 25:12structured requests are, as they sound,
  847. 25:14very structured, to the point, and
  848. 25:16minimal. They ask very defined sets of
  849. 25:17questions. We've also got a state for
  850. 25:20the second pass, which is actually based
  851. 25:22on the subject and body of the email
  852. 25:24that's been drafted. And we've also got
  853. 25:25a second state for that second pass that
  854. 25:28we're going to do when Claude drafts a
  855. 25:29reply. So, it's going to contain
  856. 25:30information about the email and the
  857. 25:32actual draft reply to see if we've
  858. 25:33answered the question, which was the
  859. 25:35second pass that we were going to do.
  860. 25:37The second thing that it's broken down
  861. 25:38here are the questions. So, we've got
  862. 25:40five questions per email in one single
  863. 25:43request. So, every single email in that
  864. 25:46CSV is going to get all five questions
  865. 25:48answered in one single request, and
  866. 25:50that's fine because every single
  867. 25:51question we add to that, we could have
  868. 25:5310, 15 questions, a basically free. So,
  869. 25:56we can continue to add these defined
  870. 25:58questions for free, and that's why we
  871. 25:59send them all at once so we can have
  872. 26:00this parallel processing of all of the
  873. 26:02different criteria. And in each
  874. 26:03response, we're going to get a score
  875. 26:05level
  876. 26:06and a confidence level back from Jev.
  877. 26:09And you can see it's built out these
  878. 26:10different triage questions. So, the
  879. 26:11first one we've got is a choice
  880. 26:12question. It says, "What is the support
  881. 26:14email primarily about?" If it's about
  882. 26:15billing, then the criteria is about
  883. 26:17charges, invoices, payments,
  884. 26:19subscription cost, but it's not for
  885. 26:20things like login or access problems.
  886. 26:22So, we're being very specific what it is
  887. 26:23and what it isn't, and it's given some
  888. 26:24examples in there. Then we've got
  889. 26:26technical, feature request, account, and
  890. 26:28none.
  891. 26:29And like we said, in every request, we
  892. 26:30want to give it an out. We want to give
  893. 26:31it a none so that it can actually choose
  894. 26:33that rather than trying to just
  895. 26:34pigeonhole it into one of these
  896. 26:35categories. And then we've broken it
  897. 26:37down for things like asking for money
  898. 26:38back. And again, we've got the criteria
  899. 26:40here for a null. If it's true, the email
  900. 26:42asks for money to be returned or
  901. 26:43credited. If it's false, the email makes
  902. 26:45no claim on money already paid. And it
  903. 26:47says, "Asking to cancel without
  904. 26:48mentioning a refund is false." So,
  905. 26:49they're not going to get that money back
  906. 26:50for that. We've broken down how badly
  907. 26:52broken and that is a score. So, the
  908. 26:54score gives the different criteria from
  909. 26:56effectively one to four. Nothing is
  910. 26:58broken, it's a cosmetic damage, it's
  911. 27:00annoying, and they cannot use the
  912. 27:01product. Now, what's really important
  913. 27:03here is we're really specific with the
  914. 27:04criteria. So, what does annoying
  915. 27:06actually mean? It means a feature is
  916. 27:07degraded or fails sometimes and there is
  917. 27:10a workaround. So, because we specified
  918. 27:11that, Jeff can be more sure in its
  919. 27:14determination of whether it's annoying
  920. 27:16or not annoying. And it's important to
  921. 27:17remember that every single email gets
  922. 27:20put through the same criteria. So, for
  923. 27:21example, a billing email will still go
  924. 27:23through and get asked these breakage
  925. 27:24severity questions. So, how badly is the
  926. 27:27product broken for this customer,
  927. 27:28judging by what the email describes? It
  928. 27:30will basically come back and say, "It's
  929. 27:32not at all." and therefore it will get
  930. 27:33ignored. But by sending all the
  931. 27:35questions at once, we can speed up,
  932. 27:36they're all parallel processed, we can
  933. 27:38basically just understand that it's
  934. 27:39nothing to do with breaking and
  935. 27:41therefore it's zero. And the actions are
  936. 27:43not taken off the back of that score
  937. 27:45then for that email. And then for anger,
  938. 27:47how angry is the customer, we've got the
  939. 27:48score. So, how angry does this customer
  940. 27:50sound? We've got the score from calm
  941. 27:51annoyed to furious. Then we've also or
  942. 27:53it Claude has also broken out how much
  943. 27:55judgment it needs. How much human
  944. 27:56judgment does this answer need? Is it a
  945. 27:58standard case? Is it a judgment or is it
  946. 28:00unusual? And that is a choice there. So,
  947. 28:02it's an option to choose any of those
  948. 28:03three. And the second part, we've got
  949. 28:04the verification questions around does
  950. 28:06it answer the question? Does it promise
  951. 28:08a refund or discount? And Claude is
  952. 28:09going to basically use the type safe
  953. 28:11skill to turn all of this information
  954. 28:12into a request that then gets sent to
  955. 28:14Jeff in the JSON structure that we saw
  956. 28:16earlier. But also inside here, inside
  957. 28:18the plan, we've got the routing rules.
  958. 28:19So, is it going to get sent to a person,
  959. 28:21engineering team? Is it going to be put
  960. 28:24into a
  961. 28:25pile for Claude to actually go and
  962. 28:26draft? It's got the different confidence
  963. 28:28levels that we're measuring this against
  964. 28:30and you can come in and change these.
  965. 28:31And it defines those confidence levels.
  966. 28:32So, if it's below this category
  967. 28:34confidence on rule number two, then it's
  968. 28:36not going to be trusted as a result and
  969. 28:38actually it's going to be routed to a
  970. 28:39person. And then we've actually got down
  971. 28:41here all the different routings in terms
  972. 28:43of actual scripts and Claude has gone
  973. 28:45ahead and written those scripts based on
  974. 28:46the type safe skill. You can see, like
  975. 28:48we said, there's no LLM determination
  976. 28:50here. It's purely a script. It will
  977. 28:52literally send all of the results from
  978. 28:53Jeb through these routes to actually put
  979. 28:55them into the right piles. And these are
  980. 28:57all if statements that are verifiable
  981. 28:59against the numbers that Jeb has
  982. 29:01returned as the output. So, you've seen
  983. 29:02now how to write this using Claude, and
  984. 29:04that's going to be a lot quicker than
  985. 29:05writing it yourself. But, the harder bit
  986. 29:07actually is knowing which questions to
  987. 29:09ask in the first place about your
  988. 29:10process. And this is the bit that I
  989. 29:12couldn't find online. Nobody has written
  990. 29:14up. So, I decided to make my own
  991. 29:16checklist that you can grab for free in
  992. 29:18the description below that you can go
  993. 29:19through to try and work out what
  994. 29:20questions you need to ask of your
  995. 29:22process or how to come up with the
  996. 29:23requirements that you can input as that
  997. 29:25initial prompt. So, this isn't even
  998. 29:26available in this skill that they
  999. 29:28provide for you. That's a very specific
  1000. 29:29set of criteria of how to structure the
  1001. 29:31requests to Jeb. But, here's the
  1002. 29:33checklist that I've used for the
  1003. 29:35different five question areas. And I've
  1004. 29:37kind of pulled this together from the
  1005. 29:38docs that I've read and the videos I've
  1006. 29:40seen around the use cases of Jeb here.
  1007. 29:42So, always think about it from what the
  1008. 29:44software is going to do next, not from
  1009. 29:46the data point of view. So, every
  1010. 29:48question that you ask Jeb exists because
  1011. 29:50an action or it needs to trigger an
  1012. 29:52action. An action depends on it. So,
  1013. 29:54you're going to list the actions first.
  1014. 29:55That's number one. So, ours, if you
  1015. 29:57remember, were Claude replies, uh going
  1016. 29:59to put it into an engineering queue,
  1017. 30:00going to put it into a feature log, or
  1018. 30:02send it to a person, route it to a
  1019. 30:03person. And then for each action, you're
  1020. 30:05going to write the trigger as a
  1021. 30:06sentence. And that sentence is going to
  1022. 30:07form your question effectively. Let's
  1023. 30:09use the example of send it to a person
  1024. 30:12when it's about money. And that sentence
  1025. 30:13is the question, and it also tells you
  1026. 30:14the action, which tells you the type.
  1027. 30:16So, if it's going to act or not act,
  1028. 30:18it's a null or yes/no. If it's going to
  1029. 30:20choose a different destination, then
  1030. 30:22it's a choice. It's like a routing
  1031. 30:23problem. Or if it's sorting, then it's a
  1032. 30:25score. Thirdly, we're going to write
  1033. 30:26down what a person would look at before
  1034. 30:28making that call. I.e., what is the
  1035. 30:30input? What's the state? And in this
  1036. 30:31case, it was an individual email or a
  1037. 30:33list of emails. And then number four,
  1038. 30:35we're going to add what else could be
  1039. 30:36true that would change the routing. So,
  1040. 30:38this These are your edge cases, even if
  1041. 30:40they're super rare. And that's where bug
  1042. 30:42severity came from. We're going to
  1043. 30:43re-route depending on bug severity and
  1044. 30:45we're going to actually then score that.
  1045. 30:46And then number five is deciding what a
  1046. 30:48wrong answer is going to cost you so
  1047. 30:50that you can set those confidence levels
  1048. 30:52or those tolerances. So that's going to
  1049. 30:53set the threshold for each action. And
  1050. 30:55you saw that we had all of our different
  1051. 30:56confidence thresholds. Now I've gone
  1052. 30:58ahead and run this on all 50 emails in
  1053. 31:00that CSV. It took 4.2 seconds for all 50
  1054. 31:02to be run using eight concurrent workers
  1055. 31:04so it's done that in parallel. It would
  1056. 31:06have taken 31 seconds sequentially which
  1057. 31:08is how you probably do it with an LLM.
  1058. 31:09It would have taken 31 seconds
  1059. 31:11sequentially so it's able to actually do
  1060. 31:12those all at once. And the cost was
  1061. 31:150.0026 for all 50 which is roughly
  1062. 31:17equivalent to 19,000 emails equaling a
  1063. 31:20dollar. So if we run this every morning
  1064. 31:21for 50 emails it's going to cost us
  1065. 31:23about a dollar a year. So extremely low.
  1066. 31:25It's put it into this markdown file for
  1067. 31:27us to easily digest the results and it's
  1068. 31:29told us it's sorted by priority. So it's
  1069. 31:31basically said 70% anger, 30% severity
  1070. 31:34in terms of the weighting it's given it
  1071. 31:36on priority. You can see that 19 of them
  1072. 31:38rooted to a person based on my anger and
  1073. 31:40severity. And it gives us, you know, the
  1074. 31:41subject, why it's been categorized as
  1075. 31:44that. So this for example is somebody
  1076. 31:45asking for their money back. On a scale
  1077. 31:48of naught to three I think it was they
  1078. 31:49are super angry and it's clearly
  1079. 31:51something severe or it's been deemed
  1080. 31:53severe by Jev. And therefore our
  1081. 31:55customer support person or customer
  1082. 31:56support rep has a prioritized list of
  1083. 31:58cases to go through every single morning
  1084. 32:00and it took less than 10 seconds to work
  1085. 32:02that out. We've then got a bunch of
  1086. 32:04requests in the engineering queue that
  1087. 32:06have been rooted there, some feature
  1088. 32:07requests and 19 to actually just be
  1089. 32:10replied to by Claude. Stuff that is no
  1090. 32:11longer a task for our customer service
  1091. 32:13rep. So now you've seen how you compare
  1092. 32:16Jev with Claude, here are the rules you
  1093. 32:18need to follow to get the most out of
  1094. 32:20it. These are the patterns taken
  1095. 32:22directly from the docs. So number one is
  1096. 32:24actually kind of counterintuitive but
  1097. 32:26super interesting insight which is ask
  1098. 32:27everything at once. We're going to ask
  1099. 32:29all five questions like we did earlier
  1100. 32:31in one request. So the problem this is
  1101. 32:33going to solve is waiting for it and
  1102. 32:34this is why it's so incredibly quick. So
  1103. 32:36with a chat model you're going to ask
  1104. 32:37one thing, wait, then decide what to ask
  1105. 32:39next. But, with Javelin, every question
  1106. 32:41in a request runs in parallel, like we
  1107. 32:43saw, and you only pay to read the text
  1108. 32:45once. So, you ask everything up front,
  1109. 32:47including questions that might not
  1110. 32:48apply, like the bug severity for our
  1111. 32:50billing scenario earlier, and then it
  1112. 32:51ignores what you don't need. And this is
  1113. 32:53called speculative fan out. So, for
  1114. 32:55example, if it comes back as a billing
  1115. 32:56query, the bug answers just get
  1116. 32:58completely ignored in the processing.
  1117. 33:00And if it was a bug, you already have
  1118. 33:01the severity, and you don't have to make
  1119. 33:03another call to understand the severity.
  1120. 33:05There's one request, multiple questions
  1121. 33:06in that request, multiple types of
  1122. 33:08request, null, choice, scored. Pattern
  1123. 33:11two is gating decisions based on your
  1124. 33:13confidence level for specific actions
  1125. 33:16will help you build systems that are
  1126. 33:17both reliable and safe. Now, I know that
  1127. 33:19was quite wordy. An example that
  1128. 33:21Typeface used directly in their docs is
  1129. 33:23voice banking. So, let's say under 0.6
  1130. 33:25confidence, so 60% confidence, nearly
  1131. 33:2750/50, it's going to get redirected to a
  1132. 33:29person. So, check my balance at 0.6 is
  1133. 33:32fine because the worst-case scenario is
  1134. 33:34somebody hears their balance if it goes
  1135. 33:35wrong. But, we need to gate different
  1136. 33:37actions differently. So, approve the
  1137. 33:38transfer needs to be a lot more
  1138. 33:40confident, say 0.85, 0.95, and below
  1139. 33:43that it asks you to confirm if that is
  1140. 33:45the action you're wanting to take first.
  1141. 33:46And you can see this in the demo from
  1142. 33:48the voice browser from the start. Scroll
  1143. 33:49down just happens. It just scrolls down
  1144. 33:51immediately. But, anything destructive,
  1145. 33:53like closing a tab, is going to stop and
  1146. 33:55ask you to confirm before it does so.
  1147. 33:56And that's because it falls in below the
  1148. 33:58confidence threshold that they've set.
  1149. 34:00So, you have one confidence threshold
  1150. 34:01per action set by what it costs you to
  1151. 34:04actually be wrong on that action. Now,
  1152. 34:05pattern three is basically scoring the
  1153. 34:08individual parts by breaking it down
  1154. 34:09into into individual questions and
  1155. 34:11weighing them or adding them together.
  1156. 34:13And this rule or pattern exists because
  1157. 34:15you're encouraged to break the judgment
  1158. 34:16of these different things, independent
  1159. 34:18dimensions, into separate questions that
  1160. 34:21we can score each one separately and
  1161. 34:23then combine them with weights so that
  1162. 34:25we can then control the routing in the
  1163. 34:27code. And I'll give you an example to
  1164. 34:29bring this to life, which is big, messy
  1165. 34:31judgments. So, we could have a bunch of
  1166. 34:33CVs, right? And we could ask, is this a
  1167. 34:36good candidate? Score it 1 to 10. And
  1168. 34:38you get one number from Jev that you
  1169. 34:41can't question. So, it's hard to know
  1170. 34:42what criteria is used to determine the
  1171. 34:44outcome there. But what we might do
  1172. 34:46instead is break that down into what
  1173. 34:47good might look like. So, it might be
  1174. 34:49for engineer, Python depth of knowledge,
  1175. 34:52team leadership skills, system design,
  1176. 34:55or how fast they pick up new things. So,
  1177. 34:57for a senior engineer, you could weight
  1178. 34:59the Python skills and the design at 40%
  1179. 35:01each as the threshold. But for a manager
  1180. 35:03position, maybe leadership gets 40% as
  1181. 35:06the threshold. Now, pattern four then in
  1182. 35:09the final pattern is routing it to the
  1183. 35:10best handler for that job. Or this is
  1184. 35:12called intent routing. So, we're going
  1185. 35:14to classify incoming requests and route
  1186. 35:16each to the optimal handler, whether
  1187. 35:18that's deterministic logic inside a
  1188. 35:20script like we saw, a specialist LLM
  1189. 35:22like Claude, or a human. So, TypeSafe
  1190. 35:25can sit in front of all these as a fast,
  1191. 35:28cheap classifier that determines which
  1192. 35:30handler to invoke, whether to send it to
  1193. 35:31our human customer support rep or not.
  1194. 35:33So, like where's my order is going to go
  1195. 35:35to a database look up. That's not going
  1196. 35:37to use AI. It's not going to require a
  1197. 35:38human. But a product question is going
  1198. 35:40to go to Claude who has access to the
  1199. 35:41product docs. And then a complaint is
  1200. 35:43going to get a second question of how
  1201. 35:45complex is this complaint. And then the
  1202. 35:46messy ones are going to go completely to
  1203. 35:48a person from the get-go. And then
  1204. 35:50anything with low confidence is not
  1205. 35:52going to hit the threshold and it's
  1206. 35:52going to go to a person anyway. And when
  1207. 35:54you bring all of those four patterns
  1208. 35:55together, you get the smart home demo
  1209. 35:58that TypeSafe have directly on their
  1210. 36:00docs that uses all of this at once. So,
  1211. 36:02this is about controlling your lights
  1212. 36:04and your gadgets in your house using
  1213. 36:06voice effectively or using typing
  1214. 36:07commands or using an app. So, you type
  1215. 36:09in turn off all the lights in the house.
  1216. 36:10One request is going to ask which kind
  1217. 36:12of request is it, which rooms, which
  1218. 36:14device, and what action needs to take
  1219. 36:16place. And then if you type two commands
  1220. 36:18in one sentence, Jev is going to spot
  1221. 36:20that it's a compound request and
  1222. 36:21actually hand it to an LLM to split
  1223. 36:23that. It's then going to judge the two
  1224. 36:24halves there. So, it understands that it
  1225. 36:25needs to actually split the request into
  1226. 36:27multiple questions there. And then if
  1227. 36:28you ask just a general question that's
  1228. 36:29unrelated, it knows to pass it to an LLM
  1229. 36:31to answer that. And that is effectively
  1230. 36:33pattern four. And the whole point in
  1231. 36:34that is that they put Jev
  1232. 36:36next to the LLM, and you basically Jev
  1233. 36:39works so quick on these deterministic
  1234. 36:41actions where you've got questions,
  1235. 36:42threshold, confidence, etc. That you
  1236. 36:44don't even notice he's there. It can
  1237. 36:46make judgment calls on things so quickly
  1238. 36:48that decide the next most probable step
  1239. 36:50to take depending on score and
  1240. 36:52confidence against your criteria. So, if
  1241. 36:53you can't tell, I am super excited about
  1242. 36:55this because when you pair it up with
  1243. 36:56Claw code or GPT-Astra, you've got one
  1244. 36:59hell of a combination. Jev can do a lot
  1245. 37:01of those deterministic or more
  1246. 37:03deterministic categorizations in rapid
  1247. 37:05time and for record low cost. Whilst
  1248. 37:07Claw can do the chat back and forth, the
  1249. 37:09reasoning and the judgment where extra
  1250. 37:10thinking or looser requirements, for
  1251. 37:12example, are required.
  1252. 37:14Now, if you want to see a video on
  1253. 37:15applying this combination to a domain
  1254. 37:17like GEO or browser use, then drop a
  1255. 37:20comment below and see you in the next
  1256. 37:21video.

About this transcript

This page contains the full transcript of Jev + Claude = INSANE Results (Every Concept Explained) by Simon Scrapes, generated from the public captions YouTube serves with the video. The transcript has 8,870 words across 1,256 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.