My AI Software Factory Ships Hundreds of PRs a Week (Here's How I Manage It) — Transcript

Full transcript

1. 0:00Throughout this year, I've spent an
2. 0:01insane amount of time working on my AI
3. 0:03software factory, the system where I can
4. 0:05send in any issue ticket, even an entire
5. 0:08release, and it plans it all out and
6. 0:10then builds it in an autonomous loop
7. 0:11with all the verification that I need
8. 0:13and human in / near the loop where I
9. 0:16want it. As you can imagine, it takes a
10. 0:19lot to make this kind of autonomous
11. 0:20system reliable. That's why I've been
12. 0:22spending so much time on it, especially
13. 0:24with the hopes that I can give you a
14. 0:26starting point so you can use this kind
15. 0:28of system without having to go through
16. 0:30all the growing pains that I did. And
17. 0:32I'm still working on evolving this
18. 0:34constantly. And so, what I want to do
19. 0:35right now is give you a tour of
20. 0:37everything that I'm working on to help
21. 0:38me manage and evolve my software factory
22. 0:41because I've been putting a lot of time
23. 0:43into building this mission control
24. 0:45dashboard that I think you'll find
25. 0:46really cool and hopefully inspirational
26. 0:47as well. So, this video is going to be a
27. 0:49bit more unstructured than usual. You
28. 0:52can think of this kind of like a dev
29. 0:53log, but also I have a lot more to it
30. 0:55that I want to show you because just
31. 0:57generally what I'm working on now, I've
32. 0:59seen the industry shift towards
33. 1:01recently. All the companies I'm seeing
34. 1:03that are building software factories at
35. 1:04scale, they always end up creating some
36. 1:06kind of dashboard like this to manage
37. 1:09the system cuz the factory is a living
38. 1:11and breathing machine. Even if you make
39. 1:13something really reliable from the
40. 1:14get-go, you have to constantly evolve it
41. 1:17as you change your process and what
42. 1:18you're building. And so, today I'll
43. 1:19quickly go through a lot of the
44. 1:21different components that I have of my
45. 1:22dashboard here. Like this is the full at
46. 1:24a glance view of the software factory
47. 1:27workflow down to all the individual
48. 1:28skills that I run through my agents. We
49. 1:31can look at the key decisions that I've
50. 1:32made over time as I've improved my
51. 1:34factory. We can look at the individual
52. 1:36laps for each one of the factories I
53. 1:37have running here. Like there's two I
54. 1:39have hooked up into this dashboard. We
55. 1:41have the individual decisions that we
56. 1:43make when the factory brings things to
57. 1:45us, the high-level decisions that need
58. 1:47our attention. A lot of why I've built a
59. 1:50lot of this was because I found through
60. 1:52all my testing and working with
61. 1:53companies and implementing these
62. 1:54systems, the only way to truly make this
63. 1:56reliable is if we have some things that
64. 1:59bubble back up to the human to decide.
65. 2:02And so, it's all about how can we have
66. 2:03the minimal set of decisions for the
67. 2:06human that gives the most amount of
68. 2:09control possible. We want things to be
69. 2:11running as autonomous as we can, but it
70. 2:13can't fully be a dark factory. At the
71. 2:16start of this year, when I built my dark
72. 2:18factory, it was the kind of system where
73. 2:20there was no human in the loop at all. I
74. 2:23built this all in public. I've covered
75. 2:24this on my channel before about my
76. 2:26initial goal with the dark factory,
77. 2:28which I've renamed the software factory
78. 2:30because I have added human in the loop,
79. 2:32is you could give a PRD, just a
80. 2:34high-level description of what you want
81. 2:36to build. You barely had to do any
82. 2:37planning with the agent, and then it
83. 2:39would just go through everything
84. 2:41autonomously. So, it's split the whole
85. 2:42PRD into issues, go through each one
86. 2:45with its own verification steps and code
87. 2:47reviews, and then it would just merge
88. 2:49everything straight to production. This
89. 2:51is sort of the dream with generative AI,
90. 2:53where it's reliable enough where you can
91. 2:55go from idea all the way to fully
92. 2:57shipped and validated code. Like, no
93. 2:59human in the loop anywhere, not even for
94. 3:01the merge. And as much as this can work
95. 3:04for some things, like proof of concepts,
96. 3:07it's really not enough. I've had to dial
97. 3:09back the autonomy and add more human in
98. 3:12the loop, which is what I've gotten to
99. 3:13now with my software factory. Like, this
100. 3:15system here that I'm now managing with
101. 3:17the dashboard is what I found to truly
102. 3:19work. Now, to be honest, initially I
103. 3:21really didn't like the idea of having
104. 3:22this kind of control plane over the
105. 3:24software factory cuz I thought it would
106. 3:25make it so that things are less
107. 3:27autonomous and I have more of a
108. 3:28responsibility to manage things. But,
109. 3:31I've built it in a way where it's very
110. 3:33easy to look at things at a high level,
111. 3:35and it really does help me drive
112. 3:36improvements to the factory and managing
113. 3:39things as they are running. And it's
114. 3:41really the only way to manage the
115. 3:43complexity of the factory. I've worked
116. 3:45hard to make things simple, and it's
117. 3:47pretty easy for me to manage things now.
118. 3:49But, there is a lot that goes on under
119. 3:51the hood. I mean, think about it. We
120. 3:52have to manage dependency mapping,
121. 3:54running things in parallel to really
122. 3:55scale it, figuring out how we set up the
123. 3:58app validation for each piece of
124. 3:59software we create, how we manage the
125. 4:01staging environment, all the places we
126. 4:03have human in the loop. And so, having a
127. 4:05system like this to really help us keep
128. 4:07track of things as they're running,
129. 4:09bubbling up the decisions in a nice view
130. 4:11instead of just always living in the
131. 4:12GitHub issues like I was before. I mean,
132. 4:14this is making a big difference for me.
133. 4:17I've built my own second brain, and it
134. 4:18helps me with a lot of my work. But, it
135. 4:20only works for me. So, for this part of
136. 4:22the video, I partnered with Victor, the
137. 4:24AI employee our whole team shares. When
138. 4:26someone on my team needs to know what's
139. 4:28been decided or what's on their plate,
140. 4:30they can't ask my second brain. They
141. 4:31have to ask me and wait for my answer,
142. 4:34at least until Victor. So, now Victor
143. 4:36lives in our Slack. And for example,
144. 4:38before every team call, we can have him
145. 4:40draft the agenda, pull what's stuck in
146. 4:41Jira, take a look at what's merged in
147. 4:43our Arkon repository, look at our open
148. 4:46questions in our threads. And for any
149. 4:48decisions that we make in the call, we
150. 4:49can just drop it right there for Victor.
151. 4:51And then after the call, anyone on the
152. 4:53team can ask Victor to turn our
153. 4:55decisions into Jira tasks, or at least
154. 4:57that's how my team works. And so, here I
155. 4:58did it exactly as a teammate would. And
156. 5:01so, he drafted the agenda earlier, and
157. 5:03then I gave him the decisions, told him
158. 5:05to make Jira tasks. And so, he proposed
159. 5:07them, I approved, and now I have them as
160. 5:10new tasks in Jira. My teammates used to
161. 5:12wait 4 and 1/2 hours on average for me
162. 5:14to get everything in Jira after a
163. 5:16meeting. But, now no one has to wait for
164. 5:18me, and it gets done in a couple of
165. 5:20minutes. Victor is an AI employee that
166. 5:22lives in Slack or Teams and is connected
167. 5:24to over 9,800
168. 5:26tools. And he does the actual work
169. 5:28instead of just answering questions
170. 5:30about it. And because he's shared,
171. 5:32anyone on my team can hand real work to
172. 5:34him, not just me. If you want to hire
173. 5:36Victor for yourself, go to victor.com
174. 5:38for $100 of his work for free running in
175. 5:41minutes. Use my code colmadine100, and
176. 5:44my link is in the description.
177. 5:46And so, as you probably guessed, I am
178. 5:48also using the software factory to build
179. 5:51this dashboard. So, the mission control,
180. 5:53this option in the dashboard, is the
181. 5:55application itself that you're looking
182. 5:56at right here. And it's also really nice
183. 5:58because as I make my factory better and
184. 6:00better, it's going to make this
185. 6:02dashboard better and better, which in
186. 6:03turn makes it easier for me to manage
187. 6:05the software factory. So, it's this nice
188. 6:07flywheel where it's almost going to
189. 6:09become sort of self-evolving over time,
190. 6:11which I'm really looking forward to. I
191. 6:13mean, I just got started building this.
192. 6:14Before I had this dashboard, I was
193. 6:16managing everything directly in GitHub,
194. 6:18which kind of worked cuz I would have PR
195. 6:21comments for the owner decisions, and I
196. 6:22could look directly at the merges to see
197. 6:25how my laps are doing and things like
198. 6:27that. But, this is definitely a lot
199. 6:28better. And so, yeah, let me show you
200. 6:30around a little bit. Now, the number one
201. 6:32thing that I wanted for the home page of
202. 6:33my system, what you're looking at here,
203. 6:35is a high-level visualization of my
204. 6:37entire factory, all of the decision
205. 6:39points, all of the individual skills.
206. 6:42Before I had this, it was hard for me to
207. 6:44picture in my head the shape of the
208. 6:45factory. So, in turn, it was also
209. 6:47difficult for me to understand how to
210. 6:49evolve things as I wanted to make my
211. 6:51factory better. So, this has been a
212. 6:53game-changer for me. So, I also want to
213. 6:55cover the underlying infrastructure for
214. 6:57the factory, how everything's stored in
215. 6:59a Postgres database, and I'm using Arcan
216. 7:01to drive it all. I'll get to that in a
217. 7:03little bit. But, first, I want to blitz
218. 7:05through a tour of all the different
219. 7:07sections that I have of the dashboard.
220. 7:09So, the first thing that I wanted after
221. 7:10the main visualization is a tab to list
222. 7:13the key decisions that I made for my
223. 7:15software factory as I've evolved it over
224. 7:16time cuz I talked to my coding agent
225. 7:18when there's issues that come up in the
226. 7:20factory, and I think about like, "How
227. 7:21can I make the factory better?" And I
228. 7:23want to document all that so I don't go
229. 7:25back and try the same things that I've
230. 7:26already tried. And I also want to see
231. 7:28what improvements are currently being
232. 7:30tested. So, there's all these different
233. 7:31ideas I have for different changes I
234. 7:33want to make to my skills or different
235. 7:35steps I want to add, and I can manage
236. 7:36the testing of all of that here and go
237. 7:38and look through everything I've changed
238. 7:40already. Like, the very first key
239. 7:41decision that I made is using Arcan for
240. 7:44everything. We'll talk about that in a
241. 7:45little bit. And then, for all these
242. 7:47decisions that I make, obviously, they
243. 7:49produce different versions of my factory
244. 7:51in source control. So, I have all the
245. 7:52different commits, and I can take a look
246. 7:54at the diff here between any two
247. 7:56versions that I pick. And so, I can see
248. 7:57if between these two versions, here's
249. 7:59what's different with the workflows, the
250. 8:01steps, the underlying prompts in the
251. 8:02skills that I'm using for these steps.
252. 8:04That's been very useful as well. I also
253. 8:06have the overlying health of the
254. 8:08dashboard. So, whenever there's a
255. 8:09failure, what caused it? Twice it was
256. 8:11because I hit my rate limits with Claude
257. 8:13code. And out of the hundreds and
258. 8:15hundreds of laps that I've been running
259. 8:16over the past couple of weeks, 21 of the
260. 8:18failures were because of issues within
261. 8:20the workflows that I had to adjust. I
262. 8:22also have a couple of tabs here to
263. 8:23manage the ongoing runs of each
264. 8:25individual software factory. So, these
265. 8:27two tabs are custom depending on the
266. 8:29factory that I select. And so, I gave it
267. 8:32a PRD, it split it into tickets, and
268. 8:34that's what we're looking at here. I'm
269. 8:35working on doing more grouping. There's
270. 8:37not too much grouping right now, but you
271. 8:38can see a couple of instances here. For
272. 8:41this release, I've worked on most of the
273. 8:43issues already. So, there's a couple
274. 8:45that need my attention. That gets to the
275. 8:47owner tab we'll talk about in a bit. And
276. 8:49then these three blue ones are the
277. 8:50ongoing laps. And so, this one's in the
278. 8:52middle reviewing the pull request. These
279. 8:54two are currently in implementation. And
280. 8:56then I can also look at the laps here to
281. 8:58get more details on, you know, the step
282. 9:00it's on and the key decisions that have
283. 9:01been made like the complexity of the
284. 9:03task. My factory is dynamic. It does
285. 9:05different depending on how complex the
286. 9:08issue is. Cuz we don't want to spend too
287. 9:10much time on the really simple stuff,
288. 9:12and we don't want the factory to just be
289. 9:14a basic build and validate for the
290. 9:15really complex stuff. So, we'll talk
291. 9:17about the owner tab in a little bit with
292. 9:19the key decisions that bubble up to us.
293. 9:21The scorecard is really cool. It shows
294. 9:23for this current version of the software
295. 9:25factory, how many laps have we run, how
296. 9:27many merges have we had in this specific
297. 9:29code base, how long has this been going
298. 9:31for. These are the key metrics that I
299. 9:33look at to see if I need to make
300. 9:35improvements to this factory version and
301. 9:37what improvements I should make. And
302. 9:39then for the testing here, these are all
303. 9:41of the scenarios that it devises for
304. 9:43regression testing and testing each
305. 9:45feature as it's building out the code
306. 9:47base. And so, I can also take a look at
307. 9:48which ones it's deciding to run, cuz
308. 9:50obviously for every PR we're not going
309. 9:52to run every single one of these, and
310. 9:54also the success and failures for all of
311. 9:56them. And then I have the record of all
312. 9:57of the versions and what changed for
313. 9:59each one of them, at least what the core
314. 10:00changes are. And then I have the rewind,
315. 10:03which this is something newer I'm
316. 10:04working on as well. It's using the
317. 10:05branching in Neon, which is what I'm
318. 10:07using for my back end and my Postgres
319. 10:09solution. And so, this allows me to
320. 10:12actually go back to different versions
321. 10:14of the software factory. If I want to
322. 10:16roll back or just test more on that
323. 10:18version to compare to what I have right
324. 10:19now. So, let me go back to the live
325. 10:21branch. And with that, that's the
326. 10:23high-level of my dashboard. I know that
327. 10:25some of these things might be specific
328. 10:27to me, but I hope you can take
329. 10:28inspiration from a lot of these ideas
330. 10:30for how I'm building in this
331. 10:31observability and this easier way for me
332. 10:33to manage the whole system. And so, with
333. 10:36that, let's talk a bit more about the
334. 10:37underlying infrastructure for the
335. 10:40dashboard and the factory itself. You'll
336. 10:42also notice here a lot of different
337. 10:43mentions of Archon. Archon is my
338. 10:46open-source harness builder. It allows
339. 10:48you to take any AI coding process and
340. 10:50turn it into a reusable workflow, and it
341. 10:52handles a lot of the more difficult
342. 10:53things with agentic engineering under
343. 10:55the hood like work trees. So, we can
344. 10:56very easily manage things in parallel.
345. 10:58So, this is the backbone of my entire
346. 11:01software factory. A lot of people ask me
347. 11:02how Archon plays into it. Basically,
348. 11:05there's not a single coding agent
349. 11:06executed in my factory that is not done
350. 11:09so through an Archon workflow. All of
351. 11:12this is just a collection of Archon
352. 11:14workflows that together drive the entire
353. 11:16factory. All the planning, all the
354. 11:18implementation, all the validation. And
355. 11:20so, going into the Archon UI, which is
356. 11:22separate, we can see all of the
357. 11:24individual runs all the way down to the
358. 11:26level of the logs for each run. So, we
359. 11:28have full observability here. And my
360. 11:30dashboard also links to this whenever we
361. 11:32want to click into any individual step
362. 11:35within each lap of the factory. Now,
363. 11:37usually for this level of detail, you're
364. 11:39going to have your agent dig into it,
365. 11:40but I'm just showing that everything
366. 11:42really is running Archon under the hood.
367. 11:44My co-author of Archon, Rasmus, has put
368. 11:47an insane amount of effort building all
369. 11:50of the workflows that run my factory.
370. 11:51It's here in the SDLC pack that ships
371. 11:53default to Archon, and you can click
372. 11:55into any of these, and I know you'll be
373. 11:57impressed with every single one of them.
374. 11:58There's so much care that's gone into
375. 12:00each one of these workflows, and
376. 12:02everything running in the factory is
377. 12:03using one of these for each step. And
378. 12:06the beauty of Archon is everything that
379. 12:08we can observe down to the individual
380. 12:10levels of the agent's tool calls and
381. 12:12turns and decisions it's made is all
382. 12:14stored in a Postgres database. So, I
383. 12:16have my Archon hooked up here to Neon.
384. 12:18And so, everything that happens in my
385. 12:20software factory is stored here. So, I
386. 12:23have full audit trails, and I can also
387. 12:25digest this later to figure out things
388. 12:27that went wrong, and make improvements
389. 12:28to my software factory that way. Cuz if
390. 12:30I'm not storing every agent turn and
391. 12:32decision and tool call, I don't really
392. 12:34have a way to know how things went wrong
393. 12:36or why they did when I want to go back
394. 12:39retroactively and figure out ways to
395. 12:41evolve my factory. So, everything that
396. 12:43goes on in Archon, everything that I'm
397. 12:45seeing here in my dashboard is stored in
398. 12:48Neon. For example, one of the things
399. 12:49that used to be difficult for me to
400. 12:51manage is all of the owner decisions.
401. 12:53So, questions for me, blockers that need
402. 12:55my attention. It's important to have
403. 12:57this cuz it's not a dark factory. We
404. 12:58need some human in the loop when the
405. 13:00agent deems it necessary. But before, it
406. 13:02was kind of difficult for me to manage
407. 13:04things. Here's an example of what it
408. 13:06looked like before I had the dashboard.
409. 13:08I would have my agent mention me in
410. 13:10either an issue or pull request. That
411. 13:12was my key that it's my time to go in
412. 13:14and resolve the issue, answer the
413. 13:16question, either by talking to my
414. 13:18manager agent, or posting a comment here
415. 13:21in the issue or PR. And that works, but
416. 13:23then I had to watch the repository
417. 13:25closely, or look out for those email
418. 13:27notifications when I mentioned, or the
419. 13:29Slack alerts I had set up. It was okay,
420. 13:31but I want to manage everything in one
421. 13:33place. Not just the owner decisions, but
422. 13:35everything happening in my repository,
423. 13:37both for things that need my action, and
424. 13:39just for my observability. And so, what
425. 13:41I've done is I've taken advantage of
426. 13:42Neon's functions, which is a part of
427. 13:44their new backend with authentication,
428. 13:46object storage, functions, AI gateway. I
429. 13:49mean, it's incredible. There's a reason
430. 13:50I use Neon for everything now. And so,
431. 13:52specifically for functions here, I have
432. 13:54a GitHub webhook that triggers a
433. 13:56function whenever there is a new GitHub
434. 13:58issue or pull request. And because I
435. 14:00manage my entire factory with issues and
436. 14:02PRs, that means that I now have a single
437. 14:05entry point to process everything that's
438. 14:07happening in my factory, and store that
439. 14:09in the Neon Postgres database. And
440. 14:11because my Archon runs are also stored
441. 14:13in the Postgres database, I have a
442. 14:15central place for literally everything,
443. 14:17for all observability, and when I want
444. 14:19to dive into things with my agent to
445. 14:20make improvements to the system. And if
446. 14:22we go to the monitoring for the Neon
447. 14:24function, we can see every single minute
448. 14:26there is a lot going on in my factory as
449. 14:28all of these labs are running in
450. 14:29parallel. And the way that my dashboard
451. 14:31is able to receive this information so
452. 14:33quickly as everything is changing is by
453. 14:35taking advantage of Neon's new real
454. 14:37time, which is in public beta right now.
455. 14:39It makes any SQL query live data. So, I
456. 14:42have these different SQL queries hooked
457. 14:44up into my dashboard, so it's pulling
458. 14:46all this information as things are
459. 14:48happening with the underlying Arkon
460. 14:50workflows. And so, obviously I don't
461. 14:52really want to get into the weeds with
462. 14:53the code here, but just showing you an
463. 14:54example of what it looks like to run one
464. 14:56of these queries with Neon real time.
465. 14:59So, this is one of the SQL queries that
466. 15:01my dashboard runs, and we can see the
467. 15:03data come in in real time. Like there we
468. 15:06go, we had a couple of new records that
469. 15:07just came in. So, as things are
470. 15:08happening with new pull requests, new
471. 15:11issues, new comments, whatever it is,
472. 15:13the dashboard is able to get that right
473. 15:15away and display it to me. So, I don't
474. 15:16just have a record of things, I can also
475. 15:19watch things as they're going live,
476. 15:20which is especially helpful as I'm
477. 15:22working on improving my software factory
478. 15:24itself. So, that's the dashboard and the
479. 15:27factory at a high level to give you some
480. 15:29ideas and inspiration. And I just want
481. 15:31to keep you up to date with what I'm
482. 15:32working on here because the software
483. 15:34factory is the big thing that I'm
484. 15:36pouring so much time into right now, and
485. 15:38I'm giving it all to you. I'm
486. 15:39open-sourcing this entire thing, so you
487. 15:41can use the factory to build any
488. 15:43software that you want while I work
489. 15:45under the hood on making this more and
490. 15:47more reliable using the dashboard to
491. 15:49help me do so. Now, if you want to build
492. 15:51your own software factory, you don't
493. 15:52want to lean on my repository or Neon or
494. 15:54Postgres or whatever, that's okay. Even
495. 15:56if you just want to take my resources as
496. 15:58a starting point, I would encourage you
497. 16:00to do that. But obviously, if you want
498. 16:02the most power out of the box, there's a
499. 16:04reason why I'm building this as an
500. 16:06open-source project. I would encourage
501. 16:08you to use Neon. You need some kind of
502. 16:10back end to store and process
503. 16:12everything. It's not enough for it to
504. 16:13just all live in GitHub. I mean, there's
505. 16:15a reason why I've been partnered up with
506. 16:16Neon for so long, and I'll have a link
507. 16:18to them in the description. And so with
508. 16:20that, if you found this interesting,
509. 16:22you're looking forward to more content
510. 16:23on my software factory and agentic
511. 16:25engineering, I would really appreciate a
512. 16:26like and a subscribe. And with that, I
513. 16:28will see you in the next video.