The OpenSource Managed Agents — Transcript

Full transcript

1. 0:00Okay, so vendor neutral MIT licensed on
2. 0:04your own hardware without a runtime
3. 0:06bill. These are all things that Claude
4. 0:09managed agents doesn't have. Also, the
5. 0:12new OpenAI agent and API or Gemini
6. 0:15agents doesn't have them. But all of
7. 0:17these things are the things that the
8. 0:19agent system I'm going to show you today
9. 0:21actually has. Today I want to talk about
10. 0:24a way to run managed agents fully
11. 0:26locally or in your own private cloud
12. 0:29with any model that you want, adding in
13. 0:31any tools that you want, skills that you
14. 0:34want, MCPs that you want, and being able
15. 0:37to do it all really easy without you
16. 0:39having to go into low-level code and
17. 0:41tinker with every tool call or every
18. 0:43skill that you add. Today, I want to
19. 0:45talk about a self-managed agent
20. 0:47execution harness called True Forge.
21. 0:50Now, before we get into it, this video
22. 0:52is sponsored by True Foundry, the
23. 0:54creators of True Forge, but everything
24. 0:57I'm showing you today is in an
25. 0:58open-source repo. There's no paid tier
26. 1:01in anything that you'll see. And the ask
27. 1:03from them is basically that if you find
28. 1:05this useful and interesting and you
29. 1:07would like to see them work on it more,
30. 1:09just go and start the repo. That's it. I
31. 1:12wor with True Foundry earlier on in the
32. 1:13year and made a video about their
33. 1:15gateway platform and it seems to me that
34. 1:18as they're developing things for
35. 1:19enterprise, they're also looking at how
36. 1:22they can make some of these cool tools
37. 1:23available to the open-source community
38. 1:25as well. In this video, I want to try
39. 1:27and untangle that a little bit. I want
40. 1:29to talk about how you should be thinking
41. 1:31about the different kinds of harnesses
42. 1:33that are out there and how you can be
43. 1:35using them for your different kinds of
44. 1:37projects. So let's start with the word
45. 1:39because I think this is where a lot of
46. 1:41the confusion is. When most people hear
47. 1:43the words agent harness right now they
48. 1:45tend to think of coding harnesses.
49. 1:48That's things like claude codeex pi om
50. 1:52open code a whole bunch of these and
51. 1:55those are harnesses right they wrap a
52. 1:57model they give it a file system a shell
53. 2:00some tools and they run the loop for you
54. 2:02but they're really built for one job
55. 2:04right which is writing code. The second
56. 2:06kind of harness that's out there at the
57. 2:08moment is what I would call the assemble
58. 2:10itself framework. So this is Langraph,
59. 2:12deep agents perhaps agent SDK or ADK and
60. 2:16these give you all the pieces and you
61. 2:19write a loop and you wire up everything
62. 2:21yourself, right? You decide how the
63. 2:23state gets persisted, how approvals work
64. 2:26and so on. And I've done quite a number
65. 2:28of videos on those over the years, but
66. 2:30nowadays can be actually really tedious
67. 2:32to use a lot of those frameworks if
68. 2:34you're trying to build your own harness.
69. 2:36The third kind and the one that's most
70. 2:37recent is what I actually just made a
71. 2:39video about recently, which is the whole
72. 2:41idea of a hosted runtime harness. So,
73. 2:45this is what people are calling managed
74. 2:47agents. So, you've got Claude managed
75. 2:49agents, you've got Gemini managed
76. 2:51agents, and I made a whole video about
77. 2:54this talking about the different kinds
78. 2:55of managed agents that are out there.
79. 2:57But the whole idea with those is that
80. 2:59you define an agent, give it a model,
81. 3:02almost certainly it's going to have to
82. 3:03be the provider's model. You'll have to
83. 3:06give it tools, which again will be often
84. 3:08provided for you, and a system prompt.
85. 3:11And then that company's going to run the
86. 3:13loop. So session, tool calls, sandboxes,
87. 3:17approval, all of that is handled. You
88. 3:19just hit an API. And that third category
89. 3:22can be really nice to use, right? I've
90. 3:24built with it, etc. But here's the
91. 3:26thing. Up until pretty recently, that
92. 3:28category really meant you only had one
93. 3:30choice in that it has to be hosted with
94. 3:33the actual provider. And perhaps even
95. 3:35worse than that, you're often just tied
96. 3:37down to certain models that may just be
97. 3:40not a great fit for what you're trying
98. 3:42to do. So, True Forge is an open-source
99. 3:46version of that third kind. And the
100. 3:49readme puts it really nicely. Building
101. 3:51an agent is easy, but running one well
102. 3:54is not. The running it well part is all
103. 3:56about streaming, session persistence,
104. 3:59tool servers, sandboxes, approvals, a
105. 4:02UI. That's all the stuff that really
106. 4:04honestly most people don't want to
107. 4:06write. And it's the stuff that managed
108. 4:08agents have been charging you a premium
109. 4:11for. True Forge is that layer as a repo
110. 4:15that's fully open- source, MIT licensed
111. 4:18that you can clone and then run with any
112. 4:20particular model that you want to. All
113. 4:22right. So the question should be why
114. 4:24does it matter what kind of managed
115. 4:26agent I'm using? And this is the bit
116. 4:28that I think many people even myself at
117. 4:30times have gotten wrong that you tend to
118. 4:33think of the cost of all of this as just
119. 4:36being related to which model that you're
120. 4:38using and how much you're paying per
121. 4:40token. And then often you don't think
122. 4:43about the harness and all the plumbing
123. 4:45that is actually going on around this.
124. 4:47But when you start running managed
125. 4:49agents at volume, you start to realize
126. 4:52that actually that harness really is
127. 4:55often the key thing because it's that
128. 4:57harness that is actually deciding how
129. 5:00many tools calls you make, what tool
130. 5:03calls you make, in what order, how it's
131. 5:05going to actually decide what goes in
132. 5:07the context window. Remember, really
133. 5:09what we're paying for is whatever's in
134. 5:11the context window. So, if you've got
135. 5:13lots of tool calls in there, if you've
136. 5:15got tool bloat, right, where you've just
137. 5:17got so many tool definitions and stuff
138. 5:19in there, and you're not using
139. 5:21progressive disclosure and things like
140. 5:23that, you're going to run up a much
141. 5:25bigger bill than you would with a very
142. 5:28efficient harness. And this is where the
143. 5:30True Foundry team actually put out a
144. 5:32benchmark that shows all of this. They
145. 5:35took 14 tasks from DevRev's enterprise
146. 5:39bench, the kind where the agent has to
147. 5:41join data across a CRM, an issue
148. 5:44tracker, a document store, multiple MCP
149. 5:47servers, all with the same prompt. And
150. 5:50they ran three trials each and used a
151. 5:52blind LLM judge. And here's the thing,
152. 5:54they didn't even go for cheaper models.
153. 5:56They actually ran it on Opus through
154. 5:59three different harnesses. And you can
155. 6:01see from this diagram here, they
156. 6:03basically showed with the actual true
157. 6:05forge harness that even using the same
158. 6:08model, they were able to get the cost
159. 6:10down by 30% with the same amount of
160. 6:13accuracy. And that's because Claude
161. 6:15managed agents was using 10 million
162. 6:17tokens per run on average versus only
163. 6:203.8 million tokens on the true forge.
164. 6:23And of course, when they moved that to a
165. 6:25model like GLM 5.2, suddenly the model
166. 6:28cost came down. the whole thing now was
167. 6:30down 75% compared to the original claude
168. 6:34managed agents version. So even if we
169. 6:37skip the GLM 5.2 and just look at the
170. 6:40same model, same answers for roughly a
171. 6:43third of the tokens, that was all
172. 6:45harness decided. So the fact that True
173. 6:48Forge is actually open source means that
174. 6:51you can pick any model that you want
175. 6:53here. You can decide if you want to go
176. 6:55for a more expensive clawed model or
177. 6:58open AI model, etc. But you can also go
178. 7:01for much cheaper open models like I'm
179. 7:03going to show you in here. So, if you
180. 7:05just take one thing away from this
181. 7:07video, it's this. The harness, not the
182. 7:09model, is usually what's driving your
183. 7:12agent costs. And the harness is the part
184. 7:15that you've actually got a lot of
185. 7:16control over, even if you're not running
186. 7:18your own models with your own GPUs, etc.
187. 7:21Now for me I would say while the cost
188. 7:23savings here are the headline for me the
189. 7:26lockin thing is probably a bigger deal
190. 7:28than a lot of people are watching just
191. 7:30in the park sweep we've seen a number of
192. 7:32different news items about what user
193. 7:35data these people are actually training
194. 7:37on and how they're actually using user
195. 7:39data and you've got things like data
196. 7:41retention if you want to have zeroday
197. 7:43data retention you cannot use clawed
198. 7:45managed agents if your agent definition
199. 7:48lives inside a hosted runtime then and
200. 7:51the system prompt, the tool wiring, the
201. 7:53approval rules, the session history, all
202. 7:56of that sits on someone else's product.
203. 7:59And perhaps that's fine up until the day
204. 8:02you want to try a different model or you
205. 8:04want to try a different sandbox or you
206. 8:06want to try a different configuration,
207. 8:08etc. You don't want to get locked into
208. 8:10these services where you become
209. 8:12dependent on them for a lot of things
210. 8:14beyond the model. We're already seeing
211. 8:17open models get so good that really high
212. 8:20levels of intelligence are now being
213. 8:22democratized so you can access it over
214. 8:24lots of different providers. Even if you
215. 8:27have objections or legal things around
216. 8:29sending your data to certain countries,
217. 8:31the cool thing now is with open weights
218. 8:34models, there are providers popping up
219. 8:36everywhere. They're running the same
220. 8:38weights with the same API endpoints. And
221. 8:40I hear this a lot from people building
222. 8:42inside larger companies. They often have
223. 8:44a mandate to run things on prem or at
224. 8:46least inside their own cloud account and
225. 8:48a hosted agent system just isn't going
226. 8:50to get through procurement through
227. 8:52compliance no matter how good it is. So
228. 8:55they fall back into a category of trying
229. 8:57to write the entire loop themselves and
230. 9:00that can also become both an inefficient
231. 9:02and expensive option. And then of course
232. 9:04if you do have some local hardware that
233. 9:06you can run some local models on we're
234. 9:09already seeing the smaller models get
235. 9:11better and better and this whole system
236. 9:13allows you to integrate with things like
237. 9:16Olama with things like LM studio to run
238. 9:19models locally. All right so True Forge
239. 9:22is a MIT licensed TypeScript repo from
240. 9:24the True Foundry team with one MPX
241. 9:27command. You can basically spin this up.
242. 9:29It takes care of a local SQL light
243. 9:32database. all your data is staying
244. 9:33locally and it's running on a local
245. 9:35host. There are also hosted modes if you
246. 9:38want to do it with things like Postgress
247. 9:39or Reddus and Docker, but the whole
248. 9:41local mode is a very deliberate step on
249. 9:43True Foundry's part to make this as easy
250. 9:46as possible for you to get started
251. 9:48trying it out. Now, they have a whole
252. 9:50chat UI, but don't be fooled into
253. 9:51thinking that this is another chatbot.
254. 9:53It's not. The chat interface there is
255. 9:56the test bench. It's where you poke
256. 9:58around and play with the agent and test
257. 10:00things out before you actually save it.
258. 10:02The actual main thing here is the
259. 10:04runtime and it's actually exposed a few
260. 10:07different ways. You've got the UI that
261. 10:09you can start off with, but you've also
262. 10:11got an HTT API endpoint and then also
263. 10:15you can use that with a TypeScript SDK.
264. 10:18This means that you can build something
265. 10:20in this very quickly, expose the API
266. 10:23endpoint, and then have other apps ping
267. 10:26it. So, this makes it ideal if you've
268. 10:29got people on your team, for example,
269. 10:31that you want them to basically run
270. 10:33different agent workflows, but you don't
271. 10:36want them having to worry about how to
272. 10:38configure the tools, how to actually
273. 10:41prompt it, right? How to do all of that.
274. 10:43You can actually just bake this into
275. 10:45something like Slack, a mobile app, any
276. 10:48sort of software that you wanted to
277. 10:49write. So, let's jump in, have a look at
278. 10:52how this actually is all put together.
279. 10:54All right, so to get started here, this
280. 10:57is pretty simple. We're just going to go
281. 10:59to the readme docs and see go to the
282. 11:01quick start and then I'm going to just
283. 11:04install this locally to get started. And
284. 11:07you can see this this will basically
285. 11:08install all the components in here.
286. 11:10everything from being able to deal with
287. 11:12sandboxes, MCPs, skills, all those sorts
288. 11:15of things and it's actually now running
289. 11:17locally so that I can get started. Okay,
290. 11:20so once you're in the actual interface
291. 11:22in here, you can come into the settings
292. 11:25and you can set up models, connectors,
293. 11:27skills, and sandboxes. So models here,
294. 11:31you've got your standard sort of
295. 11:33offthe-shelf models. If you do want to
296. 11:35use Claude, you can use Claude. If you
297. 11:37want to use Gemini, you can use Gemini,
298. 11:39etc. You've also got a bunch of open
299. 11:41providers or you can configure something
300. 11:44totally private. Here I've configured up
301. 11:46my own API endpoint to run DeepSeek
302. 11:49Flash 4.1. So this model is just out.
303. 11:52The previous four Flash was one of my
304. 11:54favorite models. So I think like a lot
305. 11:56of people I'm quite excited to see how
306. 11:57this new model goes. But you could
307. 11:59configure lots of different models in
308. 12:01here. Next up you've got connectors. So
309. 12:03you can see I've already gone and
310. 12:05configured Tavilli which is a really
311. 12:07good search tool. You can get a free API
312. 12:09call. I think you get a thousand calls a
313. 12:11month or something for free. Also, Deep
314. 12:13Wiki is one of the ones that I like, but
315. 12:15there are lots of others in here. So, if
316. 12:17I wanted to add my notion, I can go
317. 12:19ahead and do that. If I wanted to add
318. 12:21bright data for doing any sort of
319. 12:22scraping or things like that, I've got
320. 12:24that in here. Or I can just come in here
321. 12:27and totally configure up a personal MCP
322. 12:30server. So, that could be running
323. 12:32locally, that could be running in the
324. 12:33cloud, etc. And then you've got skills.
325. 12:36Now skills is obviously one of the best
326. 12:37ways to actually get started. So I'm
327. 12:39going to put in the Tavi research skill
328. 12:42there. So I'm going to enable that so
329. 12:44that the folder contains the skill.md. I
330. 12:47can then just import that and use that
331. 12:49as well. And then lastly, we've got
332. 12:51sandbox providers. So they're a bit
333. 12:52light on this at the moment. I think
334. 12:54True Foundryy's also got some
335. 12:55interesting things around sandboxes that
336. 12:57you might see coming in the near future.
337. 12:59But once I've got this set up, you can
338. 13:02see I can just come along and I can do a
339. 13:04new chat. Now the chat ID here is really
340. 13:07just to allow you to test that
341. 13:10everything's connected up and stuff like
342. 13:11that. So you can see I've got my
343. 13:13DeepSeek VFL.1
344. 13:15flash model in there. I've put on to
345. 13:18Villi as a tool in there. And if I
346. 13:21wanted to import some files or something
347. 13:23like that, I can use this just like a
348. 13:24normal chat interface. But it's very
349. 13:26important that you understand that this
350. 13:28is not the product, right? This is just
351. 13:31to basically test the product. So the
352. 13:33idea here is that okay, now I can see
353. 13:37that okay, everything looks like it's
354. 13:39running. I'm getting reasoning tokens
355. 13:41back from 4.1 flash. I can see that it's
356. 13:45running through the different tests and
357. 13:47we've got the ability now to run and see
358. 13:50what actually is going on here. You can
359. 13:51see that the agent has gone off and just
360. 13:54me saying that this is basically a test
361. 13:56to wire things up. I actually was just
362. 13:58planning on testing the actual LLM, but
363. 14:01it's gone through the whole system. And
364. 14:03this is the cool thing of this, right?
365. 14:04All of these things are now integrated
366. 14:06as an agent. So, it's acting as an agent
367. 14:10with these tool steps and with this
368. 14:12thinking. So, it's taken this as like my
369. 14:16task that hey, we want to check that
370. 14:18everything's set up. It's gone through.
371. 14:19It says right, yep, we've got the date
372. 14:21time, we've got the MCP setup, we've got
373. 14:24our tool execution, everything on that
374. 14:26is working. We can see it failed on a
375. 14:28sandbox because I haven't set up a
376. 14:30sandbox yet, right? So, it's good to
377. 14:32know that this is already starting to
378. 14:34work. Okay, so if I want to build a new
379. 14:38agent, let's say I want to build an
380. 14:40agent that's going to go and get AI news
381. 14:43for me. So, I'm just going to do a very
382. 14:44quick one here just to sort of show you
383. 14:46how you would do it. Okay. So, you can
384. 14:48see I've got my prompt there. And what
385. 14:52I'm doing here is basically just going
386. 14:54through this, letting it work out the
387. 14:56different things to do. And the cool
388. 14:59thing is it's doing all this in the
389. 15:01background. It's using Tilly. I didn't
390. 15:02need to tell it to do that. It basically
391. 15:04worked out to do that. It's also then
392. 15:07gone to the VengeB site, which is the
393. 15:08link that I gave it. Worked out that
394. 15:10okay, we probably want the category A in
395. 15:12there. Now I don't have a sandbox set up
396. 15:15so it's obviously working out that
397. 15:16that's going to be an issue but you can
398. 15:18see very quickly it's basically worked
399. 15:20out itself what it needs to do like
400. 15:22extracting the homepage trying out
401. 15:24different things etc for doing this and
402. 15:27then the cool thing is it's then used to
403. 15:29villi to actually do searches passing in
404. 15:32info knowing that I'm just looking for
405. 15:34the last 48 hours we can see what it's
406. 15:37actually done there with Tavilli and you
407. 15:40can see that finally it goes through it
408. 15:42finds out like the Deepseek v 4.1 flash
409. 15:45release different releases that have
410. 15:47come up chat GPT for financial services
411. 15:49etc. And then even for other ones it's
412. 15:51then focused on checking the date right
413. 15:53to see if those are right now once we've
414. 15:56got that we can then come over here to
415. 15:58actually save this I then basically put
416. 16:01in my instructions which is like my
417. 16:04system prompt here for doing this. I can
418. 16:06add any skills that I want. So I'm going
419. 16:08to put in the actual Tilly research
420. 16:11skill in there. And then I've got to
421. 16:13decide some things like, okay, do I want
422. 16:15the agent to ask clarifying questions?
423. 16:18Do I want it to have dynamic sub agents?
424. 16:20Do I want it to have basically like
425. 16:22return a generative UI? Actually, I
426. 16:24don't want the generative UI or the
427. 16:26clarifying questions on this. I just
428. 16:28want to sort of ping it and basically
429. 16:30have it return the result. So now I can
430. 16:32save the changes and it will go through
431. 16:36and create this agent for me. And you
432. 16:39can see now if I want to basically just
433. 16:41try this out, I can come in here and put
434. 16:44in the prompt to kick it off. And you
435. 16:46can see sure enough it's going off. Even
436. 16:48though I wrote just get me the news, it
437. 16:50understands that this is going to be AI
438. 16:53news. I actually changed it to be 72
439. 16:55hours in the system prompt there. It's
440. 16:57got today's date. It's got the different
441. 16:59parts that it's going to basically get
442. 17:01there. And you can see that already it's
443. 17:03got a bunch of sort of the recent
444. 17:06stories that have come up. And so now
445. 17:08it's going to basically check the dates
446. 17:09of those. And all of this is running
447. 17:12with the deepse model, right? So this is
448. 17:14a very cheap model to run. And you can
449. 17:16see that its final output is coming back
450. 17:19with the actual stories with a summary
451. 17:22and with the the source URL for each of
452. 17:24these. So this is pretty good, right? If
453. 17:26I check something like this, sure
454. 17:28enough, this was written in the past day
455. 17:29or so. And then it's also put in some
456. 17:32honorable mentions there. Okay. So if I
457. 17:34want to use this with Python, I can
458. 17:36actually just come in here and basically
459. 17:39set up a a little UV project with just a
460. 17:41single Python file. If we look at this,
461. 17:44it's just going to basically call the
462. 17:47local host version of this. This is
463. 17:50where the agent is currently being
464. 17:51hosted. You could actually have it
465. 17:54running in the cloud if you wanted to do
466. 17:55that. Obviously, I think going forward
467. 17:57you'll see True Foundry have their own
468. 17:59sort of managed versions of this which
469. 18:02lets you prototype everything locally
470. 18:03and then ship it to their cloud to run
471. 18:05regularly at scale kind of thing. But
472. 18:08you can see if I want to basically run
473. 18:09this now, all I have to do is UV run
474. 18:12main.py. It will basically set it up. It
475. 18:15will go through setting up the different
476. 18:18tools, etc. And then it will go off and
477. 18:20do the task. And here I'm just streaming
478. 18:22all the things back. But if you wanted
479. 18:24to, you could get it to basically just
480. 18:25give you an output file or to save to a
481. 18:29database or something like that at the
482. 18:31end. And you can see now that it's
483. 18:32finished off here. We've got all the
484. 18:35results back. We've got the Open AI
485. 18:38launches financial services. We've got
486. 18:40Deepseek V4.1 Flash in there. And for
487. 18:43each of these, we've got the Venturebe
488. 18:45article. We've got the date. We've got
489. 18:47the different information in there. So
490. 18:50just to finish up, I would say this is
491. 18:52definitely worth checking out if you
492. 18:53wanted to run agents locally on your own
493. 18:56machine, etc. And the cool thing is once
494. 18:59you've actually created these agents,
495. 19:01they're stored there. So you could
496. 19:03actually install this in the cloud and
497. 19:04import them in there and use that. Now
498. 19:06looking at their road map, it looks like
499. 19:08they've got a bunch of things coming up.
500. 19:10You can see that they've planned to have
501. 19:11a gateway integration, which totally
502. 19:13makes sense. True Foundry has a lot of
503. 19:15sort of gateway features both for
504. 19:17models, for MCPs, for locking things
505. 19:19down like that. It looks like they're
506. 19:21addressing probably my biggest issue
507. 19:23with it is that I would like them to
508. 19:24have a local sandbox execution and it
509. 19:27looks like that's coming. It'd be great
510. 19:28if you could also lock that down with
511. 19:30Docker sandboxes or something like it
512. 19:33just to make it fully safe and secure.
513. 19:35And looking at the rest of the road map,
514. 19:36it all seems very sensible stuff if you
515. 19:39wanted to do this kind of thing. And a
516. 19:41lot of it is perhaps stuff that is in
517. 19:43the lockdown versions of managed agent.
518. 19:46So it's great to see that this is all
519. 19:47coming in the near future. So go give it
520. 19:49a star on GitHub. The links are in the
521. 19:51description. Really this is giving you
522. 19:53all the advantages of a managed solution
523. 19:56without the vendor lockin and without
524. 19:58the downsides etc. So let me know in the
525. 20:00comments what are the biggest features
526. 20:02you would like to see added to this.
527. 20:05Talking to the team at True Foundry,
528. 20:06they do seem very committed to making
529. 20:08this work, both for their customers who
530. 20:12are using it, but also for the general
531. 20:14open-source community. Anyway, as
532. 20:16always, if you found the video useful,
533. 20:18please click like and subscribe, and I
534. 20:20will talk to you in the next video. Bye
535. 20:21for now.