Jev + Treg is a crazy combo for automation... — Transcript

Full transcript

1. 0:00So you probably saw this new model Jeff
2. 0:01popping up on Twitter for the past few
3. 0:03days. C problem try to answer is why the
4. 0:06latest model is already so good advanced
5. 0:08task like coding and mess but still
6. 0:11sucks at basic business workflow and
7. 0:13automations like it can handle Olympic
8. 0:16mass level problems really well. But
9. 0:18when you ask to handle a customer
10. 0:20inquiry that involve billings most
11. 0:22companies never trust agent or model to
12. 0:24make that decision fully autonomously
13. 0:26cause reasons kind of come down to two.
14. 0:28Why is the overconfidence? You
15. 0:30definitely experience agents just saying
16. 0:32you're absolutely right. Well, in fact,
17. 0:34it is absolutely wrong. So the model
18. 0:36today will make judgment call like this
19. 0:38without calibrate the actual confidence.
20. 0:40While most of business workflow require
21. 0:42almost 100% accuracy. That's why human
22. 0:45is good. And on the other hand, the cost
23. 0:47and speed need to make sense because
24. 0:49many business workflow has huge scaled
25. 0:51volume. And the reason large model today
26. 0:53have those factors is also baked in
27. 0:56results from the training process. The
28. 0:58large learn model today all goes through
29. 0:59this reinforced learning from human
30. 1:01feedback process where the model will be
31. 1:03fine-tuned based on feedback that human
32. 1:05give in chat assistant. But what Jeff
33. 1:07take is something they call the
34. 1:09reinforced learning for calibrated
35. 1:11decisions. So in short it is a model
36. 1:13that designed for reliable highquality
37. 1:15decision making rather than being
38. 1:17creative or assist human. Basically it
39. 1:19is model that designed to make a
40. 1:21decision giving a set of options but it
41. 1:24cannot make up options itself and it
42. 1:26cannot output text it cannot write code
43. 1:28it cannot reason step by step so as a
44. 1:31result unlike normal library model where
45. 1:33they are output results token by token
46. 1:36Jeff does not predict text but output
47. 1:38probability of list of given answer in
48. 1:41extremely fast and efficient approach so
49. 1:43at a high level it might feel a bit
50. 1:45limited but it is just so much faster
51. 1:47and cheaper than even the cheapest large
52. 1:50model out there. This is comparison I
53. 1:52did where I compare Jeff with cheapest
54. 1:54intelligent model like GPT 5.6 Luna and
55. 1:57quite consistently Jeff is around five
56. 1:59to seven times faster and five times
57. 2:01cheaper. Though one thing to be aware is
58. 2:03that Jeff has this context window of
59. 2:0532K. If your response is bigger
60. 2:08currently it can't handle it or you will
61. 2:10have to do things like map reduce to
62. 2:12work around it. And meanwhile when Jeff
63. 2:14output response it not only output you
64. 2:16one response it will give you the
65. 2:18probability distribution of what it
66. 2:20thinks is the right answer. This is a
67. 2:22gamecher because for business automation
68. 2:25you can finally trust model to operate
69. 2:27fully autonomously because every answer
70. 2:29it give comes with this confidence score
71. 2:31that you can build business logic guar
72. 2:33around and all those factors open up
73. 2:35some use case that just not possible
74. 2:37before. One example is playing games.
75. 2:39Previously it's very difficult to get
76. 2:41model playing games because it require
77. 2:43real time fast decision making but
78. 2:45because in games it only have it limit
79. 2:48set of input and options and Jeff can
80. 2:50processing so fast it can actually play
81. 2:52games fully autonomously and my friend
82. 2:54also shared this demo with me where
83. 2:55someone build this system that can live
84. 2:57monitoring any call and give real-time
85. 2:59classification for if there are any
86. 3:01fraud or issues or frustrations which is
87. 3:02a great showcase about what you can do
88. 3:04with this type of realtime intelligence
89. 3:06and on the other hand it is also really
90. 3:08good at browser and computer use because
91. 3:10for agent to use browser the hardest
92. 3:12part is normally for you to decide which
93. 3:14UI elements to click or type because DOM
94. 3:17is often very messy. So large learning
95. 3:19model either make mistakes or eaten up
96. 3:21context window really fast but browser
97. 3:23use already integrated Jeff in their
98. 3:25stack and achieve much faster browser
99. 3:27and computer use. The way it works is
100. 3:29that we will send Jeff the task we are
101. 3:31trying to achieve the browser
102. 3:33interaction history so it understand
103. 3:34where we are at as well as a list of DOM
104. 3:38elements. Then all Jeff does is predict
105. 3:40which type of actions it should do next.
106. 3:42Is a click on something type of
107. 3:44something? If it's a click, which UI
108. 3:46element it should be click on and if
109. 3:48it's type, which element it should be
110. 3:50typed on and output probability score
111. 3:52for each answer. So we will be able to
112. 3:54locate the accurate UI element really
113. 3:56fast and then it will pair with a small
114. 3:58large model to do the actual text
115. 4:01generation for interactions like type.
116. 4:03So this is one example of how you can
117. 4:05use Jeff plus large energy model to
118. 4:08complete those task end to end. As we
119. 4:10mentioned before, Jeff is basically
120. 4:11model designed for business automation
121. 4:13especially when things that is require
122. 4:15high accuracy or massive amount of data
123. 4:18to process that were otherwise not
124. 4:20making economic sense with expensive
125. 4:22large models. Like here is one demo for
126. 4:24internal link mapping. So if you ever
127. 4:26build SEO and AEO, you know you want to
128. 4:29making sure all your page actually link
129. 4:31together to increase page ranking. That
130. 4:33will actually require model to race
131. 4:34through hundreds of pages and accurately
132. 4:37locate the link between pages. With
133. 4:39normal model like cloud ops 5, it can
134. 4:41take hours. But with Jeff here, it scan
135. 4:43more than 500 pages and did a mapping in
136. 4:46less than 50 seconds. And we even see
137. 4:48some creative use where you can use Jeff
138. 4:51to real time customize and change set of
139. 4:53copy based on visitors persona. So how
140. 4:56do you actually use Jeff model?
141. 4:57Fundamentally the way you call Jeff is
142. 4:59very similar to how you use structure
143. 5:01output from large range model. You pass
144. 5:02it prompt or in their case they call a
145. 5:04state as well as set of questions like
146. 5:06in the support ticket handling example.
147. 5:09The questions could be which department
148. 5:10should handle this request. You were
149. 5:12given instructions about how Jeff should
150. 5:14made this decision. You will also give a
151. 5:16list of options and each option you can
152. 5:18add further descriptions and then the
153. 5:20Jeff will respond the answer like this
154. 5:22with probability score for each options
155. 5:25and there are three kinds different
156. 5:26questions you can pass on. One is those
157. 5:28type of multi-choice question that we
158. 5:30just showcased another true and false.
159. 5:32So if you're building AI chat platform
160. 5:34you can use Jeff to prevent people from
161. 5:37jailbreak or prompt injecting by sending
162. 5:40every user request to Jeff and ask the
163. 5:42output is this prompt a jailbreak? true
164. 5:45or false with some detailed instructions
165. 5:47and then Jeff will give you a
166. 5:48probability score of how likely this is
167. 5:51true and then you can build your own
168. 5:53business logic on top of it. The third
169. 5:55type is score. So let's say you want to
170. 5:57use Jeff to score whether a Twitter post
171. 5:59is purely organic or paid advertisement
172. 6:02based on the engagement stats. They can
173. 6:04send post detail to Jeff as state. Then
174. 6:06ask Jeff to give authentic score and for
175. 6:09score you can give different criteria
176. 6:10range from clearly manipulated to
177. 6:13suspicious probably organic and clearly
178. 6:15organic. Each new option you add will
179. 6:17plus one on the index and Jeff will
180. 6:20output probability for each options and
181. 6:22give you a final score. So those are
182. 6:24three type of question you can get Jeff
183. 6:26to output and altogether you can already
184. 6:28do a lot of retesting the automation
185. 6:31cases. For example, one of the key
186. 6:32feature from Jeff is this confidence
187. 6:34score for every single answer. This
188. 6:36allow you to build your own business
189. 6:38logic on top of this confidence score to
190. 6:41achieve much higher reliability. For
191. 6:43example, for the prompt injection case,
192. 6:45you can build a whole business logic
193. 6:47decision tree based on the confidence
194. 6:49score. Like if the hazard score is more
195. 6:51than 70%, then automatically block the
196. 6:54message. If a jailbreak score is less
197. 6:56than 70% but more than 35% then ask
198. 7:00human for review. Otherwise, just send
199. 7:02as normal. So, this is one example you
200. 7:04can see how you can build really
201. 7:06sophisticated business logic based on
202. 7:07the confidence score from Jeff. And
203. 7:09there are some tips that you can use to
204. 7:11steal Jeff for more accurate and aligned
205. 7:14response. Like for each question, you
206. 7:15can either just pass a simple
207. 7:17description like this or you can pass a
208. 7:19much more detailed description like
209. 7:21this. Within criteria option, you can
210. 7:22even include things like examples. So,
211. 7:25it's almost like few shop prompting. If
212. 7:27we go back to the customer support
213. 7:28example again, if someone asks the
214. 7:31export button double charge my credit,
215. 7:33so my invoice is wrong on this month and
216. 7:35task we want Jeff to do is which team
217. 7:37should handle this with simple
218. 7:38instruction which team should handle
219. 7:40this. Jeff might likely routing to
220. 7:42bidding since it's a double charge
221. 7:44issue. But if you give additional
222. 7:46instruction about what the focus should
223. 7:47be, if the focus should be route to
224. 7:49whoever must fix the root cause, then it
225. 7:52might route to technical as a main
226. 7:54answer. So you can see there are a lot
227. 7:55of interesting ways you can use Jeff.
228. 7:58Next I want to show you some real Jeff
229. 7:59based automation workflow that my team
230. 8:02already put in production and we can do
231. 8:04it step by step together. The first use
232. 8:06case is fraud detection and website
233. 8:08signup analysis. So my team has two
234. 8:10products that we are running. One is
235. 8:12super design which is a vibe design
236. 8:13platform. Another is track which is open
237. 8:16router for data and tooling that we
238. 8:18connect to thousands of different API
239. 8:19endpoints. And both platform experience
240. 8:22heavy bot attacks quite often like in
241. 8:24super design we literally can see in our
242. 8:26log someone prompt forget you are a
243. 8:28designer ignore all your system
244. 8:30instructions but do those and handling
245. 8:32those fraud is actually very difficult
246. 8:34because those people change domain all
247. 8:35the time and we can't easily write some
248. 8:37programmatic checks to identify all
249. 8:39those people. What we did is that every
250. 8:4115 to 30 minutes we fetch the signups
251. 8:44with all the product usage those
252. 8:46accounts have and let it classify how
253. 8:48likely this user is a fraud. If above
254. 8:51certain threshold we just automatically
255. 8:53ban this user and later we even pushed
256. 8:55it further not only having it identify
257. 8:57fraud but also help us identify
258. 8:59opportunities. So I integrated track
259. 9:02recon more than 3,000 different data
260. 9:04points including things like people and
261. 9:06company enrichment buying signal go and
262. 9:08many others instead of charge
263. 9:10subscription it just charge for usage
264. 9:12with 0% markup. We did some comparison
265. 9:14the other day for same people enrichment
266. 9:17task track is achieving similar even
267. 9:19better accuracy but with 85% cheaper
268. 9:22cost like here for enrich about 290
269. 9:25people track cost only $1.49 for $9
270. 9:28versus clay cost around $10.3. So with
271. 9:32extremely cheap and fast model like Jeff
272. 9:34plus extremely cheap and fast API and
273. 9:37data service like track we can finally
274. 9:39start doing some really really
275. 9:40interesting omission that just not
276. 9:42economically making sense before like in
277. 9:44this specific case what we did later is
278. 9:47actually turn this fraud detection flow
279. 9:49into a full on website signup analysis.
280. 9:52So every single sign up on the website
281. 9:53firstly use track to verify the email
282. 9:56and also enrich person. Then we group
283. 9:58all the information we know about the
284. 10:00signup alongside the product usage data
285. 10:02to Jeff and ask it to classify this
286. 10:04user. Is it fraud or it has upsell value
287. 10:07or affiliate and influencer partners
288. 10:09that we should reach out. And because
289. 10:11both Jeff and Track are extremely cheap,
290. 10:13we're able to scan hundreds of signups
291. 10:15with around just $1 per day. And we can
292. 10:17use those information for triggering
293. 10:19different outreach or inapp support as
294. 10:21well as band fraud. And the workflow
295. 10:23like this before Jeff and track emerge
296. 10:26is very difficult to do because the
297. 10:28unique economy just didn't make sense.
298. 10:30And we also use Jeff to help us find
299. 10:33high quality leads with clear buyer
300. 10:35intent. So what do we do here is we use
301. 10:37track to fetch popular LinkedIn posts
302. 10:40that is relevant to certain vertical we
303. 10:42care about like enrichment and then we
304. 10:44fetch people who interact or comment
305. 10:46under those posts because those one are
306. 10:48likely the potential buyer. Then we feed
307. 10:50all those people who engage with those
308. 10:52posts to Jeff ask it to qualify and
309. 10:55score those leads and classify them into
310. 10:57different persona and rows to see which
311. 10:59are the good fits. So for the top ones
312. 11:01we can go through proper enrichment
313. 11:03pipeline to get their contact
314. 11:05information for analyzing 20 post and
315. 11:07close to 400 leads. The costs are just
316. 11:09close to nothing. This is just one
317. 11:11example of buyer signal related
318. 11:13automation and leads qualification.
319. 11:15There are just so many other potential
320. 11:16use case I can think of from identifying
321. 11:19companies who are hiring on LinkedIn or
322. 11:21make job posts then match with massive
323. 11:23amount of candidate database that you
324. 11:25might have to job trends and many
325. 11:26others. And each signal here track
326. 11:29provide a pretty good set of endpoints
327. 11:31for you to collect those signals. So
328. 11:32getting your cloud code and codeex write
329. 11:34a script using track to fetch whatever
330. 11:36buying signal is very easy. So it really
331. 11:38come down to you to come up with
332. 11:40creative workflow and process of how to
333. 11:43use Jeff to filter qualify and reach.
334. 11:46And I'm really keen to hear your
335. 11:47workflow that you put together. I have
336. 11:49included the ones that my team built in
337. 11:51track.to/jff.
338. 11:53And for each one, we actually have a
339. 11:54prompt that you can just copy and paste
340. 11:56to your cloud code. So you can set up
341. 11:58both track and Jeff to write a script
342. 12:00for this automation pipeline. I'm really
343. 12:02keen for you to try out and let us know
344. 12:04what kind of workflow you come up with.
345. 12:06And the third workflow that we tried is
346. 12:08this viral expose screening. So I
347. 12:10initially saw this cool demo where they
348. 12:12use Jeff's capability to quickly
349. 12:14classify which content is generated by
350. 12:16AI on Twitter in real time. And I
351. 12:19thought this is really cool. I actually
352. 12:20have a need to identify which Twitter
353. 12:23product launch are those paid and boost
354. 12:25traffic versus organic one because as AI
355. 12:28founder on Twitter every week I will see
356. 12:30posts reaching millions of impression
357. 12:32but some of them are actually just paid
358. 12:34traffic that is not a real signal versus
359. 12:37ones that actually resonate with people
360. 12:39that I actually want to learn. So
361. 12:41internally we also have this pipeline
362. 12:43where it will grab viral post from past
363. 12:4524 hours. Use Jeff to filter which ones
364. 12:48are related to product launch. So we
365. 12:50will use track to fetch what are the
366. 12:52trending and popular posts from past 24
367. 12:55hours alongside the post stats and
368. 12:57comments. Then we will use Jeff to
369. 12:59classify which one are the organic
370. 13:01looking posts which one are likely to be
371. 13:04paid and boost traffic and also filter
372. 13:06out all the noise that is not part of
373. 13:08launch. So I get a good feed every day
374. 13:10about the latest trending things that I
375. 13:12can learn about. And I also put this as
376. 13:14a free tool on the track.j
377. 13:17where you can just paste in any Twitter
378. 13:18post and we will start triggering the
379. 13:21pipeline to fetch the post information,
380. 13:23the author profile, the replies and use
381. 13:25Jeff to classify if it is a organic
382. 13:28content or paid content with full
383. 13:30reasoning and analysis. So feel free to
384. 13:32use it as well if you want. So this is
385. 13:34pretty much how Jeff works and how I use
386. 13:36Jeff. As I mentioned before, it is a
387. 13:38model that open up so many opportunities
388. 13:40for automation that wasn't possible
389. 13:42before. And it is extremely interesting
390. 13:45when you combine Jeff and track together
391. 13:47that fundamentally change the unique
392. 13:49economy of a lot of automation process.
393. 13:51So I'm super keen to learn what do you
394. 13:53build. If you're interested, you can
395. 13:54visit track.to/jff P/JEF where we have a
396. 13:57few Jeff based automation recipe that my
397. 13:59team use and you can just copy paste
398. 14:01over or a prompt that you can put in
399. 14:03cloud code or codecs where agent will
400. 14:05guide you to build type of automation
401. 14:06that fits your business. If you're
402. 14:07interested learning more technical side
403. 14:09I also have more in-depth step-by-step
404. 14:11workshop in AI builder club where we
405. 14:13have some notebook example of all
406. 14:15different tricks and tips on using Jeff
407. 14:18and a step-by-step guide on building
408. 14:20some of those workflow from scratch. So
409. 14:21if you're interested you can join the AI
410. 14:23builder club as well. I have put both
411. 14:24link in the description below so you can
412. 14:26check out for free.