YouTube2Text

How One Founder Manages 20 AI Coding Agents at Once - Ryan Carson, Untangle — Transcript

by AI Engineer · 2,902 words · 397 segments · language en · Watch on YouTube

Full transcript

  1. 0:01[music]
  2. 0:12So, my name is Ryan Carson. I'm the
  3. 0:14founder and CEO of Untangle. So, we're
  4. 0:18building an AI parallegal specifically
  5. 0:20for family law, uh, focusing on divorce
  6. 0:23in particular. Um, so, quick audience
  7. 0:25poll. Who has been divorced? Raise your
  8. 0:27hand. Raise it. Be don't be shy. Raise
  9. 0:29your hand. Who's been divorced? Okay,
  10. 0:30keep your hands up. Who knows somebody
  11. 0:32that's been divorced? Raise your hand.
  12. 0:34Everybody. Yeah. So, for you watching
  13. 0:36the video, everyone raised their hand by
  14. 0:37the end. Divorce touches a lot of
  15. 0:39people. Um, two of my sisters got
  16. 0:41divorced. My parents got divorced. Um,
  17. 0:43and it's one of the most painful things
  18. 0:45in your whole life. And I wanted to work
  19. 0:47on that problem and just make it a
  20. 0:49little less painful. Um, started off as
  21. 0:51a consumer app. Uh, quickly pivoted to
  22. 0:53B2B and we sell to family law firms now.
  23. 0:56And thankfully product market fit is
  24. 0:58there and we're growing really fast. So
  25. 1:00that's really fun. Um so I've built and
  26. 1:04sold four companies. Um my last startup
  27. 1:06we scaled to about 110 people and I
  28. 1:09learned a lot about managing people and
  29. 1:11it turns out managing people is like
  30. 1:13managing agents. And so I'm going to
  31. 1:14share with you how we manage agents at
  32. 1:17Untangle and how everything I've learned
  33. 1:19uh managing people kind of fits there.
  34. 1:22So I'm hoping to teach you some of those
  35. 1:23things. Um, I'm also going to show you
  36. 1:26how we hire. I I think hiring is very
  37. 1:28different in the age of agents. Um, I've
  38. 1:30hired a lot of people over my career.
  39. 1:32Um, I've made a lot of mistakes and I'm
  40. 1:34going to share how we're hiring now at
  41. 1:35Untangle and I think how it's really
  42. 1:37working. Um, and then I'm going to show
  43. 1:38you a live demo.
  44. 1:41So, let's get started. This is me on X
  45. 1:45and this is Untangle. If you want to
  46. 1:46know more about us,
  47. 1:49if my little clicker works, you can do
  48. 1:51it.
  49. 1:55Okay, so what does the org chart look
  50. 1:58like now in the age of agents? How do
  51. 2:01you run your agents? Um, so the way we
  52. 2:04do it is pretty straightforward. So we
  53. 2:05have humans at the top. Um, and
  54. 2:07obviously we set priorities, we talk to
  55. 2:09our customers, we're doing all the human
  56. 2:10stuff, but what we use are manager
  57. 2:13agents and these are literally just
  58. 2:14sessions. So we use Devon. Love Devon.
  59. 2:17We do all our engine the cloud. Um, I've
  60. 2:20done [clears throat] everything, you
  61. 2:21know, from coding by hand to, you know,
  62. 2:24using cursor to here and we're in Devon.
  63. 2:26But we have a manager session. I'm going
  64. 2:28to show you this. It's literally just a
  65. 2:29session. I I call it engineer manager.
  66. 2:32Um, and it never writes code. And I
  67. 2:34mostly talk to this agent. Um, and it
  68. 2:37manages a bunch of children. So, I'll
  69. 2:38talk to you how about how we do that. We
  70. 2:40also I mostly use audio to manage this
  71. 2:43agent. Um, I don't know about you, but
  72. 2:46I'm not a fast reader. I'm definitely
  73. 2:48not an agent and I prefer audio. I can
  74. 2:50ingest audio a lot better for some
  75. 2:53reason than reading. So, I talk mostly
  76. 2:55to my manager agent over audio. I'll
  77. 2:58show you that. Um, I just find it I I
  78. 3:01can absorb it. I can understand it. I
  79. 3:03can see what's happening. It's much more
  80. 3:04natural. So, I'm going to show you that
  81. 3:06as well. Now, the manager agent uh
  82. 3:09primarily works from a ticket board. We
  83. 3:11use linear. Doesn't really matter what
  84. 3:13you use, but this is kind of the source
  85. 3:14of truth about what needs to be built.
  86. 3:16every ticket should be one PR um which
  87. 3:19is how we we work and that the uh
  88. 3:21manager agent knows which tickets to
  89. 3:23pull based off dependencies. It's like
  90. 3:25pretty basic engineering manager stuff
  91. 3:27here, but this is where it gets
  92. 3:29interesting. So each manager agent uh
  93. 3:32concurrently manages up to about 20
  94. 3:35builder agents. Um this number varies
  95. 3:38like it can go anywhere from five to
  96. 3:40probably 25. Um, this is somewhat
  97. 3:43bottlenecked by your ability to talk to
  98. 3:45the manager agent and understand what's
  99. 3:47happening. Um, uh, I think if you push
  100. 3:51too hard on this, you you quickly spin
  101. 3:53out of control. Um, I find we can
  102. 3:56concurrently manage about 25 builders.
  103. 3:58Um, now the key here is that the manager
  104. 4:01is talking to this these builders. You
  105. 4:02never go into these sessions. You never
  106. 4:04talk to them. You never manage them. You
  107. 4:06never do anything. Um, it's all done
  108. 4:08through the manager agent. And usually
  109. 4:10what it's doing is
  110. 4:12answering technical questions that don't
  111. 4:14I don't need to answer. Um or unblocking
  112. 4:17small things. Um I also have an
  113. 4:19automation that forces the manager agent
  114. 4:22to read all the children sessions every
  115. 4:24it's like every 15 minutes. This says
  116. 4:2610, but I've decided 15 is about the
  117. 4:28right call. You have to you have to
  118. 4:30build an automation. You can't trust
  119. 4:32your Androme your manager agent manager
  120. 4:35agent to do this right. It's not
  121. 4:38deterministic. So this is basically a
  122. 4:40script again in Devon. It has
  123. 4:42automations. You just tell it, hey,
  124. 4:44every 15 minutes go read every single
  125. 4:46child, push things along, keep things
  126. 4:49moving, right? Um, and then any
  127. 4:51questions go back to the manager agent.
  128. 4:53So, who's tracking so far? Do you
  129. 4:55understand what I'm saying? Okay, good.
  130. 4:57And probably a lot of you are doing
  131. 4:59this. I'm not claiming this is rocket
  132. 5:00science, but this is what we're doing
  133. 5:02right now in prod and it's working,
  134. 5:04right? Um, each of these builder agents
  135. 5:07uh has a a a review agent that it calls
  136. 5:10upon. Um, Devon has built-in reviews.
  137. 5:12Probably every code harness you have has
  138. 5:14built-in reviewers. Um, we typically
  139. 5:17stick to a three uh review max where uh
  140. 5:21we allow the builders to loop about
  141. 5:23three times with reviews. If they still
  142. 5:26have any important bugs that have been
  143. 5:27fixed, after three, then they bubble
  144. 5:29that back up to the manager agent. And
  145. 5:31if the manager agent needs to, it
  146. 5:32bubbles it back up to me, right? Um,
  147. 5:35that seems to work pretty well.
  148. 5:38Okay, so what's happening under the code
  149. 5:40under under the hood here? Like all the
  150. 5:41normal stuff, right? You've got code
  151. 5:42plus tests. You've got a QA agent. This
  152. 5:45is actually an additional uh process
  153. 5:48that we run. Um, it actually spins up a
  154. 5:50child of the builder to run QA. Um,
  155. 5:54y'all, this is cheap. like you should be
  156. 5:57spinning up as many agents as you can as
  157. 5:59long as you're not holding up your merge
  158. 6:01queue, right? So I this works well for
  159. 6:04us now. Um a a wonderful engineer named
  160. 6:07Dale recently joined. He helped install
  161. 6:09this QA process. I think it's working
  162. 6:11pretty well so far. The thing that
  163. 6:12you're all going to encounter is just
  164. 6:14merge Q slowdown. Um there's this fine
  165. 6:16balance of just, you know, pecking your
  166. 6:19PRs to death um versus just getting them
  167. 6:22merged, right? Um, the other thing I
  168. 6:25love, love, love, love is videos. So,
  169. 6:28Devon has this built in. Um, if not, you
  170. 6:31should build this. Um, it's hard. Um,
  171. 6:34every builder records a video of the
  172. 6:36feature being used and passing tests.
  173. 6:39So, if it's a UI facing feature, you've
  174. 6:42got to look at it, right? Um, this is
  175. 6:45how we know that what we've built looks
  176. 6:47good, how it works, right? So, in Devon,
  177. 6:50it's a built-in feature. It's called
  178. 6:51test. And basically it it builds test
  179. 6:53cases and then it records a video and it
  180. 6:55gives you green or or red on those on
  181. 6:57those tests. And if you get a video
  182. 6:59that's all green, you can watch it and
  183. 7:01know what we think is happening is
  184. 7:03happening and visually it looks good.
  185. 7:05Right? So this is the human in the loop
  186. 7:07where a human has to say ship it. So the
  187. 7:11manager session will will let me know
  188. 7:13there's a video ready for you. Watch it.
  189. 7:15Right? So I watched the video and this
  190. 7:17is the hard part. you've got to zoom in
  191. 7:19and understand like what is this PR
  192. 7:20doing? Um what am I watching? Um so I
  193. 7:24usually read a summary of like what is
  194. 7:26this thing? What is it supposed to do? I
  195. 7:28watch it and I ga engage. And I would
  196. 7:30encourage you this is the part where you
  197. 7:31want to cut corners. You got to watch
  198. 7:33the video. Like there is no this is the
  199. 7:36part of the bottleneck you have to
  200. 7:38engage in as a product owner. Um got to
  201. 7:40watch the video, got to pay attention.
  202. 7:42Usually it has to be on your laptop. Um
  203. 7:45I I I this literally before this talk I
  204. 7:47was on my phone you know talking to my
  205. 7:49manager my manager agent and it said
  206. 7:51there's a video for you. I watch it and
  207. 7:53I just couldn't see it well enough and I
  208. 7:55was so [snorts] tempted to be like it's
  209. 7:56okay ship [laughter] it like no get back
  210. 8:00to my machine watch the watch the video.
  211. 8:02Um this is working like you know we we
  212. 8:06ship a lot of PRs uh at high quality
  213. 8:09this way. We've iterated, you know,
  214. 8:11hundreds of times on this process and I
  215. 8:13I feel pretty good about this. Now, this
  216. 8:14will change tomorrow [laughter] and
  217. 8:17probably will be useless a week from
  218. 8:18now. Uh, but it works well today. Um,
  219. 8:21uh, the thing I will say is like if
  220. 8:23you're coding locally, stop. Like, you
  221. 8:25really need to be coding with a a cloud
  222. 8:27agent. I don't care if it's Devon or
  223. 8:29whatever it is, but if you're running
  224. 8:31local dev, you are screwing up. You're
  225. 8:34holding yourself back. You're holding
  226. 8:35your team back. You're holding
  227. 8:36everything back because you can't do
  228. 8:38this. Um so that's where we're going and
  229. 8:41I feel strongly about that.
  230. 8:43So how does this work at scale? Well,
  231. 8:45you know, say you have 10 folks um uh in
  232. 8:49the who are working on things. Typically
  233. 8:51they'll have an engineering manager uh
  234. 8:54session each. Sometimes they'll have two
  235. 8:56like right now I have two running. Um
  236. 8:59and what's beautiful is that these
  237. 9:01things talk to each other. And so uh
  238. 9:04again in Devon this is built in. It's
  239. 9:06probably built into whatever harness you
  240. 9:08use. the sessions all need to be able to
  241. 9:10talk to each other and they all need to
  242. 9:11know who each other is, right? And
  243. 9:14they're really good at this. Like this
  244. 9:16is such a delight. So what you'll see if
  245. 9:19if you if you zoom into any sessions,
  246. 9:21they're messaging each other. It's
  247. 9:23clear, it's straightforward, it's fast,
  248. 9:26it's good. Like you should be allowing
  249. 9:29your sessions to talk to each other. Um
  250. 9:31and again, if you're not, you are
  251. 9:33holding yourself back and you're
  252. 9:34probably shipping worse software. Um so
  253. 9:36they all talk a lot. Um, and it's not
  254. 9:38chitchat or stupid stuff like this is
  255. 9:41important stuff that they need to know
  256. 9:42to unblock each other. Um, which is very
  257. 9:45useful. However, you still need what we
  258. 9:48call a merge Q manager. Um,
  259. 9:51so this is how we coordinate this just
  260. 9:55wave of PRs that come in. You know, you
  261. 9:59have things like database migration
  262. 10:01numbers. These need to be coordinated.
  263. 10:03Like if all the agents are trying to
  264. 10:04grab these things, you constantly
  265. 10:06collide, right? So the merge manager
  266. 10:08just helps all that go away. It also is
  267. 10:11a little more intelligent about how the
  268. 10:13merge Q is happening. So this is really
  269. 10:15nice. Um it's working well for us. And
  270. 10:18again, these are just sessions. This is
  271. 10:20not like something special. These are
  272. 10:22just sessions in Devon and I and I name
  273. 10:25them so I understand what's happening.
  274. 10:27Um and it's pretty straightforward. You
  275. 10:30know, the talk from the gentleman from
  276. 10:31OpenAI, you know, said basically you got
  277. 10:33to unhobble. Just unhobble the thing,
  278. 10:35right? So set up the roles and then
  279. 10:39unhobble the model, right? Um, and I
  280. 10:41think just having a merge manager say,
  281. 10:43"Hey, make all this better." Uh, really
  282. 10:45works well. All right. Who's tracking so
  283. 10:47far? Good. Who thinks this is
  284. 10:52Raise your hand. Oh, one guy. Two. Wait.
  285. 10:54One. Come on. Be be more brave. We have
  286. 10:58to talk afterwards. Where Where are you?
  287. 11:00Um, okay. And then we use GitHub merge
  288. 11:02cues. It's fine. It's not rocket
  289. 11:04science. It works pretty well. So,
  290. 11:07all right. Next up, how we hire. Um, so
  291. 11:10who's hiring engineers or employees
  292. 11:12right now? Raise your hand. Yeah, decent
  293. 11:15number of you. Um, I think everything
  294. 11:17has changed. Every everything you used
  295. 11:19to do about hiring employees is over and
  296. 11:22none of it makes sense. Um, it's all
  297. 11:24outdated and I would encourage you to do
  298. 11:26something like this. So, we don't do
  299. 11:30number one any sort of pre-screens. We
  300. 11:32don't do any video chats. We don't do
  301. 11:34anything with people um it for most of
  302. 11:37this process because it will bias you.
  303. 11:40You'll like their hobbies. You'll like
  304. 11:42the way they talk to you. You'll think
  305. 11:44they're interesting. And it's all noise.
  306. 11:46None of it is signal, right? All that
  307. 11:48matters is can they operate an agent,
  308. 11:51right? Can they do that? So, the way we
  309. 11:54do that is we ask candidates to submit a
  310. 11:5745ish minute video, whole screen
  311. 12:00recording. Um, and we ask them to ship a
  312. 12:03feature PR typically on a app that they
  313. 12:06already maintain and we just we just ask
  314. 12:09them to record that. No instruction, no
  315. 12:12guidance, no nothing. And we just want
  316. 12:14to see like how good of an agent
  317. 12:16operator is this person, right? So, um,
  318. 12:20that is free. We don't pay for that.
  319. 12:21That's basically how you get in the
  320. 12:23door. Um, and what we then do is we feed
  321. 12:26these videos into a ranking agent and
  322. 12:28basically say watch the video, watch the
  323. 12:30transcript, stack rank these people. Um,
  324. 12:33now this what's interesting is I did
  325. 12:35this and I didn't trust it the first
  326. 12:37time. I didn't like what it said. Um,
  327. 12:41and what's ironic is it actually was
  328. 12:43correct. Like the person that was number
  329. 12:45one, I I should have hired them. Um, uh,
  330. 12:48but I didn't trust it because I was I
  331. 12:50was biased. So you stack rank them. Now,
  332. 12:53this is just a lot of video to watch.
  333. 12:54You could watch all these videos if you
  334. 12:56want. If you have time, do it. But I
  335. 12:58think you'll find actually a very
  336. 13:00intelligent model is pretty good at
  337. 13:01understanding like how how good is this
  338. 13:03person. Um, what's interesting though is
  339. 13:06uh candidates will mumble um or uh
  340. 13:10sometimes or they won't think out loud,
  341. 13:11which is kind of fascinating. So, you
  342. 13:13can't really tell what they're thinking
  343. 13:14or doing. Um, which is interesting. So,
  344. 13:16that's uh something to sort of note. Um,
  345. 13:19and some talk a lot. [laughter] So, you
  346. 13:23never know. Uh, then we basically pick
  347. 13:25the top five and we invite them to a
  348. 13:27paid project. So, we pay them a fixed
  349. 13:28fee for 16 hours of their time. We
  350. 13:31actually give them a seat on Devon and
  351. 13:33we ask them to uh to get a real PR ready
  352. 13:37for merge. Um, we have them sign an NDA.
  353. 13:40Um, and we record all the video. This is
  354. 13:4416 hours of video. Um, we basically use
  355. 13:46Loom and then uh we feed all the videos
  356. 13:49into uh our our agent basically Devon
  357. 13:52and we say hey who you know which one
  358. 13:54should we pick? I actually watched all
  359. 13:5716 hours of these videos for the last
  360. 13:59round. Uh it was brutal but it did give
  361. 14:01me a good read on the people. Um and
  362. 14:04then you basically pick one. You pick
  363. 14:06your finalists and then what's important
  364. 14:09is that you don't do calls before this.
  365. 14:12Um you will get biased. you'll get
  366. 14:14confused. You'll try to pick somebody
  367. 14:16you shouldn't. And I would just call the
  368. 14:19finalists and say, "Hey, you know, we
  369. 14:22think we'd like to hire you. Let's talk
  370. 14:23a little bit." Um, and that gives you
  371. 14:26the final sort of human read, but you're
  372. 14:27going to watch videos of these people
  373. 14:28for hours. Like, you're not really going
  374. 14:30to need that call. Um, but I think
  375. 14:32there's a little bit of of of human
  376. 14:33connection there that's important.
  377. 14:36I think this is how everyone's going to
  378. 14:37hire. Like, this works. Um, you know,
  379. 14:40I've hired hundreds and hundreds and
  380. 14:42hundreds of people in my career.
  381. 14:44This is just way easier and way better.
  382. 14:46Um, and I think it's where we're all
  383. 14:48moving to. So,
  384. 14:52I'm actually going to finish 3 minutes
  385. 14:53early. Um, if you want to know more
  386. 14:56about Untangle, just go to Untangle. Um,
  387. 14:58I'm on X like just all the time. I
  388. 15:00literally been on X for 20 years. Um,
  389. 15:03and so, uh, that's where I live and
  390. 15:05breathe. So, if you want to know more
  391. 15:06about uh leading companies, managing
  392. 15:09agents, um that's where I am. If you're
  393. 15:12interested in family law and you're
  394. 15:14passionate about making divorce less
  395. 15:16painful, um get in touch. Um and that is
  396. 15:19all I have to say. Thanks everybody.
  397. 15:21Appreciate it. [applause]

About this transcript

This page contains the full transcript of How One Founder Manages 20 AI Coding Agents at Once - Ryan Carson, Untangle by AI Engineer, generated from the public captions YouTube serves with the video. The transcript has 2,902 words across 397 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.