Managed Agents - Don't Get Locked In — Transcript

Full transcript

1. 0:00Okay, have you noticed that over the
2. 0:01past 5 months or so, almost every major
3. 0:05AI company has shipped a product called
4. 0:08managed agents? Alongside people
5. 0:10shipping new models, alongside people
6. 0:12shipping new frameworks, we've got this
7. 0:14whole new area of managed agents. So,
8. 0:17Anthropic was the first one to kick it
9. 0:18off. They started it in April or even
10. 0:21perhaps a little before with some of the
11. 0:22ideas there. And I've got to say, most
12. 0:24of the other companies have just copied
13. 0:26their ideas from the Anthropic offering.
14. 0:29So, Google came out in May with their
15. 0:30anti-gravity agent, but really followed
16. 0:33around the same time with their Gemini
17. 0:35managed agents. And then June brought
18. 0:38AWS to the game with Agent Core.
19. 0:40Microsoft, also June, Foundry hosted
20. 0:43agents. And then LangChain came along in
21. 0:45mid-August with a managed deep agents
22. 0:48public beta. And supposedly, on
23. 0:51September 29th, OpenAI's Dev Day,
24. 0:54they're going to roll out their version
25. 0:55of managed agents as well. So, this is
26. 0:58often on top of other SDKs and other
27. 1:01offerings that the companies are
28. 1:03actually offering. So, these managed
29. 1:05agents are the providers running the
30. 1:06harness and the sandbox, not just the
31. 1:09model. And what I want to do in this
32. 1:10video is open up one of these and show
33. 1:13you actually what's in the box. And then
34. 1:15I'll look at why these six companies,
35. 1:18all with different businesses, all
36. 1:20decided that this was the product they
37. 1:21needed to get into. And it kind of tells
38. 1:24you a lot about how they're thinking
39. 1:26about this entire business strategy.
40. 1:28Okay, so what actually is a managed
41. 1:30agent? You can think of this as three
42. 1:32layers. The bottom layer is the model.
43. 1:34And of course, that started out with
44. 1:36Opus and Claude at the time. Now has
45. 1:39moved to Fable on the Anthropic side and
46. 1:41the latest GPT models on AWS and Azure.
47. 1:45Oh yeah, and then don't forget Gemini.
48. 1:48But on top of the model, the middle part
49. 1:50is the harness. And this basically is
50. 1:52the loop. This is what calls the models,
51. 1:55what picks the tools, what's runs them,
52. 1:58and it feeds all the results back in and
53. 2:00manages that context window. And then
54. 2:02finally on top of these is the actual
55. 2:05runtime. So, where does the loop
56. 2:07actually run? The sandbox that holds
57. 2:10your credentials, that often does
58. 2:12observability or tracing, the thing that
59. 2:14survives and lives in the cloud by
60. 2:16itself. So, the whole managed element in
61. 2:19managed agents here means the provider
62. 2:21owns level two and level three of this
63. 2:23structure. And of course, if you're an
64. 2:25Anthropic or Open AI, you actually own
65. 2:27that model as well. So, the pitch has
66. 2:29been quite simple. People have talked
67. 2:31about the different APIs for messages,
68. 2:34for chats, for completions. They're
69. 2:37simply what allows you to communicate
70. 2:38with the model.
71. 2:40The managed agents is for all the
72. 2:42long-running async stuff. So, you can
73. 2:44kind of think of this as two flavors.
74. 2:46Flavor one, they've got the loop, you
75. 2:48send in your tasks and a config. You
76. 2:51never have to write that loop, you never
77. 2:52have to actually call that model. And
78. 2:54that's basically Claude managed agents,
79. 2:56Gemini agents, Agent Kohanas, and even
80. 2:59the LangChain managed deep agents.
81. 3:02Flavor two on the other hand, you
82. 3:03actually bring the loop. So, perhaps
83. 3:05this is a loop that you've written
84. 3:06yourself with LangGraph or with 80K, or
85. 3:09it's just your own code from scratch.
86. 3:11You put it in a container, they host it
87. 3:13for you with some kind of per session
88. 3:16isolation, but they also provide the
89. 3:18runtime, things like tool execution,
90. 3:20memory, and other server-side tools as
91. 3:22well. So, that's basically things like
92. 3:25hosted agents, Agent runtime. Used to be
93. 3:28called Agent in your own a few weeks
94. 3:29ago, and may change its name again in a
95. 3:31few weeks time. And even LangChain has
96. 3:33LangSmith deployment for your own
97. 3:35LangGraph code. It's kind of worth
98. 3:37noting that both Google and LangChain
99. 3:39are kind of having an each-way bet on
100. 3:41both of these, which certainly for
101. 3:43Google makes a lot of sense because
102. 3:44you've got some things being built by
103. 3:47Google DeepMind, and other things being
104. 3:48built by Google Cloud. Another access
105. 3:50that you should be looking at for some
106. 3:52of these things, too, is for the harness
107. 3:54itself, is it closed like Claude,
108. 3:57antigravity, agent core, etc., or is it
109. 4:00something that's actually open source
110. 4:02like deep agents? All right, so since
111. 4:04Claude managed agents seems to be the
112. 4:06template that started this all off,
113. 4:08let's look at the actual concepts in
114. 4:10there. So, you've got four things here:
115. 4:12agent, environment, session, and events.
116. 4:16So, the agent, obviously, is the model,
117. 4:17the system prompt, tools, ability to
118. 4:20call MCP services, skills, and these are
119. 4:23things that you create once and then
120. 4:25reference by ID. The second thing is the
121. 4:28actual environment. So, this is where it
122. 4:30all runs. The cloud sandbox of where
123. 4:33it's run could either be an Anthropic
124. 4:35cloud sandbox, or they are supporting a
125. 4:38variety of different other sandboxes,
126. 4:40including some self-hosted sandboxes on
127. 4:42your own infra. Next up, you have the
128. 4:44session. This is one running instance
129. 4:46doing one task, and then you have the
130. 4:49actual events. The messages that go back
131. 4:51and forth, the user turns, the tool
132. 4:53results, the status updates. Now, pretty
133. 4:56much all of this maps over to the Gemini
134. 4:58managed agents, where they're giving you
135. 5:00a fresh Linux sandbox as a remote
136. 5:02environment. And one of the cool things
137. 5:03that they've got there is that you can
138. 5:05actually customize that file system and
139. 5:08mount your own skills or your own agents
140. 5:10MD file. So, the managed agents are
141. 5:13literally taking care of all these
142. 5:15sessions and events and memory and tools
143. 5:18and skills and credentials and tracing
144. 5:21and scheduling all for you. So, why are
145. 5:24they all doing this? Well, basically,
146. 5:27it's the economics. And if you think
147. 5:30about it, the economics of one run of a
148. 5:33chat call is just literally a round
149. 5:35trip. You call the model, you get
150. 5:36something back, maybe your agent uses
151. 5:39it, etc. And a good example of this is
152. 5:41even Google's own docs say that a single
153. 5:44antigravity agent interaction typically
154. 5:47burns somewhere between 100k and 3
155. 5:50million tokens. And that basically all
156. 5:52happens because a single interaction can
157. 5:54trigger multiple reasoning and tool
158. 5:56loops. You don't really have control
159. 5:59over that. That's all running
160. 6:00server-side. Now, currently during
161. 6:02preview, Google's not actually charging
162. 6:04you any extra for the session server,
163. 6:07but that certainly hints that that's
164. 6:08going to come. Just like Anthropic with
165. 6:11their Claude managed agents, that's
166. 6:13charging you a session hour fee on top
167. 6:16of the tokens. So, you're now paying for
168. 6:18time, not just tokens. And you got to
169. 6:21think these long-running agents like
170. 6:23this are the highest token product a
171. 6:25model company can sell. It runs while
172. 6:28you're asleep. It manages all the
173. 6:29different turns itself. You just have to
174. 6:32tell it what to do, hope that it
175. 6:33actually does that, because most of
176. 6:35these agents don't have any sort of
177. 6:37built-in verification system, and then
178. 6:39you just have to pay the bill. Of
179. 6:40course, the other thing is with the
180. 6:42companies that actually are using their
181. 6:44own model, this all becomes a lock-in
182. 6:47game to lock you into their suite of
183. 6:50models. You want to use Claude managed
184. 6:52agents, you're going to have to use
185. 6:53Claude Fable, Opus, Sonnet, Haiku. You
186. 6:57want to use Gemini managed agents,
187. 6:58you're going to have to use one of the
188. 6:59Gemini models. Now, if you look at the
189. 7:01cloud companies, whether that's AWS,
190. 7:04Azure, or even Google Cloud, they're
191. 7:07taking the opposite bet. Their managed
192. 7:10agents, whether that's Agent Core,
193. 7:12whether that's ADK, etc., they're making
194. 7:15the model interchangeable when the
195. 7:17runtime isn't. So, of course, you're
196. 7:19running it on their serving
197. 7:20architecture. You're going to have to
198. 7:21probably use it on their memory system
199. 7:23or a memory system that's running in
200. 7:25their cloud. And then you got LangChain
201. 7:27with the deep agents. So, they're really
202. 7:29a framework company. So, they're neither
203. 7:31a model nor a cloud. They're basically
204. 7:34trying to be the control plane. And the
205. 7:35cool thing there is that they allow you
206. 7:37to use any model, any sandbox. This can
207. 7:40be self-hostable if you want it to be,
208. 7:42but their locking is going to be
209. 7:43operational. Their goal is to get you to
210. 7:45use LangSmith. And once your traces,
211. 7:48evals, deployments, etc. are all in
212. 7:50LangSmith, moving from that is going to
213. 7:52be a really big challenge. So, let me
214. 7:54make the point. None of these services
215. 7:56is inherently bad. They all have things
216. 7:58going for them. Running Cloud Managed
217. 8:00Agents is an easy way for you to
218. 8:03basically get something up, get it
219. 8:05hosted, get it running, give it a
220. 8:07sandbox, give it a memory, give it
221. 8:10access to all the different things that
222. 8:12you get with the Cloud Agents SDK. And
223. 8:15in many ways, if you're already using a
224. 8:17lot of the Cloud models, this can often
225. 8:19just end up making your life a whole lot
226. 8:21easier. You might want to be a little
227. 8:22bit more careful with your Cloud
228. 8:23providers, though, because they're often
229. 8:25taking a number of different products
230. 8:27that they had going before, and they're
231. 8:29now bolting them together to make their
232. 8:32version of managed agents. So, how
233. 8:34should you think about whether you want
234. 8:35to go with a managed agent system, or
235. 8:37whether you want to take a harness and
236. 8:39run it yourself? Well, there are five
237. 8:41key questions that you should be asking
238. 8:43yourself. First off, do you need state
239. 8:46that survives a disconnect and can
240. 8:47resume days later? If the answer is yes
241. 8:50there, you probably want to go with the
242. 8:52managed agents. It just makes things a
243. 8:54lot easier. Second, do you need a
244. 8:56sandbox for code executions that's going
245. 8:59to live server-side? Again, if the
246. 9:01answer is yes, you probably want to give
247. 9:02managed agents a good look. But, what if
248. 9:05you're designing your own agent? Or what
249. 9:07if your loop is actually your product?
250. 9:10In cases like that, you definitely want
251. 9:12to look at self-hosting, and perhaps
252. 9:14just using parts of these managed
253. 9:16solutions, like a sandbox in the cloud,
254. 9:18or a memory tool, or an observability
255. 9:20tool. Lastly, one of the key things you
256. 9:22want to make sure that you ask yourself,
257. 9:23too, is can your data actually leave
258. 9:25your infra? And do you actually need
259. 9:27zero data retention? If your data is not
260. 9:30allowed to leave your own infra, or you
261. 9:31do need zero data retention, that rules
262. 9:34out things like Cloud Managed Agents.
263. 9:36It's just not eligible for zero data
264. 9:38retention because session store history
265. 9:40and sandbox state are both server-side.
266. 9:43And that rules out a lot of enterprises
267. 9:45that are actually looking at doing this
268. 9:47kind of thing. So, each of these managed
269. 9:49agents both has their plusses and their
270. 9:51minuses. They each have their own amount
271. 9:53of control that you actually get to run
272. 9:56your agent in their cloud. And of
273. 9:58course, I haven't even touched on local
274. 10:00AI. If you're actually using a locally
275. 10:03hosted model, you want to be able to put
276. 10:05all of this on your own machine anyway.
277. 10:07And then perhaps you want to look at
278. 10:08things like Docker sandboxes, which
279. 10:10allow you to have a safe and secure
280. 10:12sandbox running fully locally. So, over
281. 10:14the next few weeks, I'm planning to have
282. 10:16a look at a number of these different
283. 10:18managed agent solutions, different
284. 10:20harness solutions, both the ones that
285. 10:22I've mentioned here, and perhaps the
286. 10:24OpenAI one if they ship at the end of
287. 10:26September, and also some of the open
288. 10:28agent systems that we're now seeing
289. 10:30start to evolve out there. So, let me
290. 10:32know in the comments what if you've
291. 10:33tried these out, what you actually found
292. 10:35to be the plusses or minuses of the
293. 10:38managed agents, and which one seem most
294. 10:40interesting to you. Anyway, as always,
295. 10:42if you found the video interesting,
296. 10:44please click like and subscribe, and I
297. 10:45will talk to you in the next video. Bye
298. 10:47for now.