Inside OpenAI's Agentic Software Factory — Transcript

Full transcript

1. 0:00Today we're talking about software
2. 0:01factories, and how the best companies in
3. 0:02the world are using them, and also how
4. 0:04you can apply these ideas to your own
5. 0:06workflow. What we're going to look at
6. 0:07today is how agents can be used at every
7. 0:09single part of the development process,
8. 0:10from writing code, testing code,
9. 0:12reviewing code, all the way through to
10. 0:14how you can monitor your applications in
11. 0:15production using AI agents.
12. 0:17>> [music]
13. 0:17>> I think this is where software
14. 0:18engineering is heading, and the best
15. 0:20companies in the world are already
16. 0:21adopting these ideas. So, in this video,
17. 0:23I'm going to break down a recent article
18. 0:24about how OpenAI are using software
19. 0:26factories, and I'll also show you a
20. 0:28bunch of live demos, so you can see how
21. 0:30to apply this in your own workflow.
22. 0:32Everything in this video, including all
23. 0:33of the prompts and the resources, are
24. 0:34linked for free in the description
25. 0:35below. So, let's get into it. Okay, so
26. 0:37this is the article we're going to talk
27. 0:38about today. So, this is from The
28. 0:39Pragmatic Engineer. It's a really great
29. 0:41article talking about how OpenAI are
30. 0:43building their software factory, or how
31. 0:45they're structuring their development
32. 0:47process in the age of AI agents. There
33. 0:50are a couple of interesting ideas, but
34. 0:52probably the main thing we want to focus
35. 0:53on today is the actual architecture of
36. 0:56the software factory.
37. 0:58So, when we take a look at this diagram,
38. 0:59what we can see is that it looks pretty
39. 1:01similar to the way most people are
40. 1:03probably already building code. So, you
41. 1:05have a software builder, this is
42. 1:07essentially the person that describes
43. 1:08the task. Notice that they've used the
44. 1:10word builder, because this could be an
45. 1:12engineer, but it could also be someone
46. 1:14else in the team, maybe a product
47. 1:15manager as well. They send their request
48. 1:18to Codex, which is used to build the
49. 1:19code, and then we have some kind of
50. 1:21continuous integration system to build
51. 1:23and test the code, and then the agent
52. 1:25will iterate around this cycle a few
53. 1:26times until we have a code change that
54. 1:29is ready to go.
55. 1:30The other thing that most companies are
56. 1:32already doing is code review with AI
57. 1:34agents. But what's interesting about the
58. 1:36way OpenAI are doing this
59. 1:38is they have multiple different agents.
60. 1:39So, they have things like a data
61. 1:41specialist, an infrastructure
62. 1:43specialist, a cloud specialist, and also
63. 1:45a security specialist as well. So,
64. 1:47they're breaking their review process
65. 1:48down into multiple different AI agents
66. 1:50looking at different things.
67. 1:52Perhaps the most interesting parts of
68. 1:53this system though, to me,
69. 1:55are the classification elements. So, as
70. 1:59we have AI agents generating much, much
71. 2:01more code, many of us have pull requests
72. 2:03that are stacking up and we're unable to
73. 2:05keep up with the code changes as humans.
74. 2:07So, what's really clever or interesting
75. 2:09about the approach from OpenAI is they
76. 2:11do a classification on the change. They
77. 2:13will look at whether a change is low
78. 2:14risk or high risk and then decide
79. 2:16whether an agent should auto merge and
80. 2:18auto deploy that change.
81. 2:20And this makes a lot of sense. People
82. 2:21will immediately object to this idea of
83. 2:23agents merging code without a human
84. 2:25review.
85. 2:26But, I think there are some cases where
86. 2:28if you're making a documentation change
87. 2:30or fixing a minor bug,
88. 2:32it's probably unnecessary to get a lot
89. 2:34of human review. And
90. 2:36often this stuff will depend on the type
91. 2:38of business you're in. So, risk is
92. 2:39relative. So, the level of risk or what
93. 2:42you consider to be a low risk or a high
94. 2:43risk change is going to look very
95. 2:45different depending on what you're doing
96. 2:47or which company you're working in. So,
97. 2:49if you're a solo developer making code
98. 2:50changes on GitHub, honestly, the risk of
99. 2:53you making a mistake or merging a bad
100. 2:54change is probably pretty low,
101. 2:56realistically. But, if you're Amazon or
102. 2:58you're building cloud services where one
103. 2:59change can take down an entire region
104. 3:01for thousands of customers, the risk
105. 3:04profile looks completely different. An
106. 3:06example of a high risk change would be
107. 3:08anything touching infrastructure,
108. 3:09anything touching a database. Those
109. 3:11things can cause a lot of problems, so
110. 3:13you always want a human to review. The
111. 3:15other one we're going to take a look at
112. 3:16today is this severity bot. This is an
113. 3:18AI agent that will monitor production,
114. 3:21look for any errors, and then basically
115. 3:23automatically investigate the incidents,
116. 3:25figure out what to do about it, and then
117. 3:27take action or help the on-call
118. 3:28engineers resolve those incidents. This
119. 3:31is something that I'm really, really
120. 3:32interested in myself because I spend a
121. 3:34lot of time working in cloud operations
122. 3:36where engineers are getting hundreds or,
123. 3:38you know, many, many pages every single
124. 3:40day.
125. 3:41So, something like this is really,
126. 3:43really powerful and it's a really great
127. 3:45use of AI agents cuz AI agents can scan
128. 3:47logs and metrics. They can correlate
129. 3:49data across multiple places. They can do
130. 3:52things that humans would struggle to do
131. 3:54during an outage or an incident. So, I
132. 3:56think that's another really interesting
133. 3:57thing we're going to take a look at.
134. 3:59They also have some other interesting
135. 4:00things like they have a performance
136. 4:01factory as well running performance
137. 4:03tests in production. And if there are
138. 4:05any regressions or errors, then they
139. 4:07also send those back into Codex. So, you
140. 4:09can kind of see this whole factory as a
141. 4:11bit of a loop where we're pushing code
142. 4:12through. We're getting it into
143. 4:14production. We're using AI agents at
144. 4:16every single step of the process and
145. 4:18we're feeding back into here. So,
146. 4:20anytime we have bugs or incidents or
147. 4:22production outages, we push tickets back
148. 4:25into the pipeline and then it's kind of
149. 4:26a self-improving loop if you think about
150. 4:28it that way. All right. So, this is the
151. 4:29first demo where we're going to review
152. 4:31open pull request. So, the idea here is
153. 4:32we have an open pull request. We're
154. 4:35going to run a classifier using some
155. 4:37kind of agents to look at the code
156. 4:39change, decide what level of risk it is.
157. 4:42And then depending on that, we can add
158. 4:44labels to the change and then we can
159. 4:46have basically an agent merge that
160. 4:48change into production. So, assuming
161. 4:51that it's low risk and all of our tests
162. 4:53and our checks are passed, we will merge
163. 4:55that to production. But if this is high
164. 4:57risk, maybe it's a database change or an
165. 4:59infrastructure change for example, then
166. 5:01we're going to wait for a human to
167. 5:02review and approve that change. This is
168. 5:04going to be an automated system that
169. 5:06will just run continually. And if you're
170. 5:08like me, you're probably pushing a lot
171. 5:10of code changes every single day. So,
172. 5:11I'm constantly pushing new code changes.
173. 5:13I'm stacking up a large number of pull
174. 5:15requests which are relatively small
175. 5:16anyway.
176. 5:17And it's a real pain to have to review
177. 5:19all of those manually every single time
178. 5:21and remember which changes need review.
179. 5:24And having some something like this can
180. 5:26really improve the throughput or the
181. 5:28cycle time of your development process.
182. 5:30So, this is an example of how you could
183. 5:32set this up. There are many different
184. 5:33ways you could build this out. But I'm
185. 5:35just going to use a simple Codex
186. 5:36automation. So, the way this works is
187. 5:39we're going to review all of the open
188. 5:40pull requests and then apply one risk
189. 5:42label to the change. And then the agent
190. 5:44is basically you're to automatically
191. 5:46merge anything that's open. And this is
192. 5:48really, really useful cuz if you think
193. 5:49about it, often when you're working on
194. 5:50these projects, you're stacking up lots
195. 5:52of small changes that are relatively
196. 5:54easy to review or merge. And this is
197. 5:56just going to help you merge those
198. 5:57things faster. So, we're going to run
199. 5:59this now and see what happens. So, let's
200. 6:01check this out.
201. 6:02We're now going to run this automation
202. 6:05and go through the pull requests and
203. 6:07then see what happens.
204. 6:09These are the open pull requests
205. 6:11currently. So, you can see here we have
206. 6:12some that look relatively trivial. So, I
207. 6:13think this is low risk, but the CI
208. 6:15checks are failing. So, this is an
209. 6:17example of a low risk change that we
210. 6:18could auto merge, but we won't because
211. 6:20the CI build is failing. If it was
212. 6:22green, however, I think we could
213. 6:23automatically merge that. You can see
214. 6:25here that a lot of these changes are
215. 6:26quite old. So, the other thing you want
216. 6:28to think about when it comes to auto
217. 6:30merging changes is don't merge anything
218. 6:31that needs a rebase or like an update
219. 6:34because that may change the the actual
220. 6:36code in general. And so, that's a
221. 6:37slightly more risky in my opinion. So,
222. 6:39I'm generally only just merging things
223. 6:41that are safe to merge.
224. 6:43So, the next demo we're going to do is
225. 6:44take a look at an automated on-call
226. 6:46alert bot. This is going to be like an
227. 6:48SRE agent that monitors your production
228. 6:49systems, looks at alerts and monitoring,
229. 6:52finds any errors, and then takes action
230. 6:55to review.
231. 6:57So, the way I thought to do this was to
232. 6:58have some kind of pub/sub system. So,
233. 7:01basically, I'm running on Google Cloud
234. 7:02for most of my apps. So, we get some
235. 7:04kind of alert, it will then be pushed
236. 7:06into a queue,
237. 7:08and then we have an agent running with
238. 7:10ADK that will basically review this this
239. 7:13issue and then figure out, you know,
240. 7:15it'll look at the logs, it'll look at
241. 7:16metrics, it'll look at revisions, figure
242. 7:19out what's going on, and then store the
243. 7:20records in a place that we can review.
244. 7:23The reason we'd have a system like this
245. 7:24is if you've ever been on call, it's
246. 7:25incredibly stressful. It's very
247. 7:27difficult for humans when they're paged
248. 7:29at 2:00 a.m. in the morning, they're
249. 7:30tired, to remember to find all of the
250. 7:32information. It can take a really,
251. 7:33really long time to search through logs
252. 7:36and correlate the metrics, but machines
253. 7:38are very, very good at doing this. They
254. 7:39can do this incredibly quickly. And so,
255. 7:41I think a system like this is something
256. 7:43that every software engineering team
257. 7:44should have at a certain scale.
258. 7:47So, it's pretty clear that there's a
259. 7:49problem with with our system here. Uh
260. 7:51the system is returning 500 errors. If
261. 7:52you were on call as an engineer, this
262. 7:54would be a pretty stressful
263. 7:56incident trying to figure out what to do
264. 7:57and how to resolve the issue. But,
265. 7:59because we have our incident triage
266. 8:00agent, the agent can now help us figure
267. 8:03out exactly what to do. And so, the
268. 8:05recommended action is to restore traffic
269. 8:06to the previously healthy revision. So,
270. 8:08essentially a rollback is the suggestion
271. 8:10from the agent. And this is exactly the
272. 8:12right thing to do. So, if you were an
273. 8:14on-call engineer, you could then go and
274. 8:16ask the agent how to do the rollback,
275. 8:18for example. But, the agent has
276. 8:20correctly triaged the issue, figured out
277. 8:22what to do, and given guidance to the
278. 8:24on-call engineer. I think this is just
279. 8:25absolutely incredible to have this kind
280. 8:27of system in production. And what's
281. 8:29interesting is it didn't really take
282. 8:30that long to kind of put this together.
283. 8:32Another really interesting idea in this
284. 8:33article that I love is this idea of
285. 8:35using AI agents to help us roll out
286. 8:37changes to production. So, 12 months
287. 8:39ago, people would have said that that
288. 8:40was crazy or reckless to use AI agents
289. 8:43for the deployment process, but I think
290. 8:44it makes a lot of sense right now, and I
291. 8:46certainly use it myself a lot.
292. 8:48Because you can use AI agents to deploy
293. 8:50your changes, use your deterministic
294. 8:52code to roll the change out, but then
295. 8:54you can have the AI agent pay attention
296. 8:55to what's happening in production in a
297. 8:57way that a human would struggle to do.
298. 8:59So, the agent can spin up its own
299. 9:00dashboards, it can look at metrics,
300. 9:02logs, telemetry, and very, very quickly
301. 9:05detect if there's an error in
302. 9:06production.
303. 9:07As soon as the agent detects there's an
304. 9:08error, it can then immediately resolve
305. 9:11the issue by reverting the change. So, I
306. 9:13think using the agents in the deployment
307. 9:15process is actually a really good idea.
308. 9:18If you are interested in this, I'll make
309. 9:19all of the code available so you can
310. 9:20check this out. So, this is the ADK or
311. 9:23Agent Development Kit AI Agent that is
312. 9:25running, monitoring our production
313. 9:27system, looking for errors, and then
314. 9:28taking action in response. So, I really
315. 9:30liked the ideas in this article. I found
316. 9:33the risk classification to be something
317. 9:35I'm going to adopt in my own workflow.
318. 9:37It's relatively easy to get that set up
319. 9:39and working, and the advantages are
320. 9:40quite high. If you're like me, you've
321. 9:42probably got a large number of pull
322. 9:43requests being stacked up every single
323. 9:45day, and it can be a bit of a nightmare
324. 9:47to try and keep up with all of the code
325. 9:48changes you're producing with agents.
326. 9:50So, I think this is a really useful
327. 9:51technique. Again, it depends on what
328. 9:53you're doing. If you're working on
329. 9:55something risky, maybe you want more
330. 9:57human review, but if you're working on
331. 9:58side projects or simpler things, having
332. 10:00agents merge the code at this point in
333. 10:02time seems way more realistic.
334. 10:05Also, I really like this idea of having
335. 10:06agents that monitor production. This is
336. 10:09really, really useful. I've worked in
337. 10:10really high throughput, you know, risky,
338. 10:13operationally heavy environments, and
339. 10:15having some kind of automated on-call
340. 10:17agent to help you diagnose issues would
341. 10:19be incredibly useful. Historically,
342. 10:21these things can be quite hard to
343. 10:23actually build and make work correctly
344. 10:25in production, but I kind of like the
345. 10:27way they've done this. So, the agent
346. 10:28just investigates the outage, gives you
347. 10:30a suggestion about what to do, and you
348. 10:33can also ask the agent questions. So,
349. 10:35this is just going to speed up incident
350. 10:36response. If you've got an outage, it's
351. 10:38going to help you resolve the outage
352. 10:40much faster. So, I think that's a pretty
353. 10:41good use of an AI agent in terms of
354. 10:43operations, as well. Thanks again for
355. 10:45watching, and I'll see you in the next
356. 10:46one. Take care.