YouTube2Text

225,000 Lines of Tests, Written and Merged by Agents - Kush Agrawal, Duolingo — Transcript

by AI Engineer · 3,148 words · 440 segments · language en · Watch on YouTube

Full transcript

  1. 0:01Hey everyone, thanks for tuning in.
  2. 0:03Today I'm going to walk you through how
  3. 0:04we use AI agents at Dualingo to write
  4. 0:06and ship more than 200,000 thousand
  5. 0:07lines of unit tests on iOS codebase.
  6. 0:09I'll share what we built and what we
  7. 0:11learned along the way. Uh I'm Kush. I'm
  8. 0:13a staff engineer on the platform team
  9. 0:14here at Dualingo. My job is to build
  10. 0:15developer tooling and AI assisted
  11. 0:16workflows for the 100 plus engineers who
  12. 0:18contribute to our client apps. Before
  13. 0:19Duelingo, I spent close to a decade at
  14. 0:21Stripe and Uber working across a few
  15. 0:22teams there. Let's start with some
  16. 0:24background on the scale of mobile at
  17. 0:26Dualingo. Uh we ship our apps to the app
  18. 0:29store every week. Um we're also a very
  19. 0:31experiment-driven company. So at any
  20. 0:32given point of time, we have more than
  21. 0:34600 AB tests running live and our code
  22. 0:36bases are pretty mature. Both the iOS
  23. 0:38and Android apps have been around for
  24. 0:39more than 10 years at this point.
  25. 0:42Here's also the number of executable
  26. 0:43lines of app code in our iOS repository.
  27. 0:45So as you can see back in 2019, we were
  28. 0:47at about 200,000 lines and today we're
  29. 0:49hit close to 2 million. And as AI
  30. 0:51assisted coding picks up, you can see
  31. 0:52that curve starting to bend upward.
  32. 0:55And it's not just the amount of code
  33. 0:56that's changing, it's also who's writing
  34. 0:57it. So last year about 200 people
  35. 1:00contributed to the iOS repository. And
  36. 1:01this year we've already crossed 250. And
  37. 1:03a lot of these folks aren't even iOS
  38. 1:04engineers. We have backend engineers,
  39. 1:06Android engineers, even PMs and
  40. 1:07designers all shipping code to the iOS
  41. 1:09app. And that's mostly been unlocked by
  42. 1:11AI coding tools like cursor and cloud
  43. 1:13code. Um, another thing to note about
  44. 1:15our codebase is that it's a codebase in
  45. 1:16transition. So a few years ago, we
  46. 1:18standardized on the MVVM architecture
  47. 1:19pattern. Today about half the code in
  48. 1:21our repository follows that pattern and
  49. 1:23the other half is the legacy code that
  50. 1:24we're in the process of migrating. Since
  51. 1:26the MVVM half is the easiest to test,
  52. 1:28that's where we focused on this project.
  53. 1:31Here's where we started back in March of
  54. 1:32this year. We were at overall unit test
  55. 1:35coverage of 11%. The MVVM layers were a
  56. 1:37bit lower, repositories were at 11%,
  57. 1:39view models at 9, data sources at 5, and
  58. 1:41combined we are about 9% for the MVM
  59. 1:43slice of the app.
  60. 1:46So why does this matter now more than it
  61. 1:48used to? As agents and humans write more
  62. 1:50code and write it faster, writing code
  63. 1:52isn't the hard part anymore. Verifying
  64. 1:53it is. So when someone puts up a PR, how
  65. 1:56do you know that it's actually safe to
  66. 1:57ship? The bottleneck is really shifting
  67. 1:58from writing to verifying. And that's
  68. 2:00where unit tests come in. They make up
  69. 2:02the base of the testing pyramid. They're
  70. 2:04pretty cheap to write and pass to run.
  71. 2:05And they also act as guardrails for
  72. 2:06agents. So when an agent's change breaks
  73. 2:08a test in CI, it gets immediate feedback
  74. 2:10on what to fix. They also act as living
  75. 2:12documentation on what the code is
  76. 2:13supposed to do. And here's where we
  77. 2:15ended up 8 months later. Overall unit
  78. 2:16test coverage went from 11% to 24%.
  79. 2:19Repositories went from 11 to 72. View
  80. 2:21models from 9 to 58. Data sources from 5
  81. 2:23to 41. And the combined MVM slice of the
  82. 2:25app went from 9% to 59%. So what did we
  83. 2:28actually end up building? On paper, it's
  84. 2:30a pretty simple loop. First, you
  85. 2:32identify the files that need tests. Then
  86. 2:33you write those tests with an agent. You
  87. 2:35verify that the tests are good quality
  88. 2:36and they pass CI. Then you resolve the
  89. 2:38PR, which means getting attributed by an
  90. 2:39engineer merged. And finally, it on the
  91. 2:41feedback so that the next batch of PRs
  92. 2:43is better than the last one. But in
  93. 2:45production, it's a little messier. This
  94. 2:47is what our system ended up looking
  95. 2:48like. I won't go through every box right
  96. 2:49now, but at a high level, it's a few
  97. 2:50temporal workflows that coordinate
  98. 2:52through shared state stored in S3. And
  99. 2:54every box you see here maps back to one
  100. 2:56of the five steps I just talked about.
  101. 2:58Here are the building blocks that we
  102. 2:59ended up using. Temporal handles
  103. 3:01workflow orchestration and execution.
  104. 3:03Cloud code acts as the coding agent. S3
  105. 3:05holds a shared state that keeps our
  106. 3:06workflows in sync. And we use GitHub for
  107. 3:08actions, labels, and PRs. Data dogs acts
  108. 3:11as observability layer where we keep an
  109. 3:12eye on the health of our pipelines.
  110. 3:15If you haven't used empodoral before,
  111. 3:16it's basically a durable workflow
  112. 3:18execution engine. You write your
  113. 3:19workflows as normal Python code and if a
  114. 3:21worker crashes, the workflow just picks
  115. 3:22up right where it left off. This fits
  116. 3:24agents really well and agent session can
  117. 3:26run for seconds to hours and it can call
  118. 3:28out to tools like GitHub, CI, S3, and
  119. 3:30Claude. We also lean on a few built-in
  120. 3:32features of Temporal. There are
  121. 3:34schedules which are basically chronike
  122. 3:35schedules for our jobs. There are
  123. 3:36composible workflows so that we can fan
  124. 3:38out work in parallel. Then there's first
  125. 3:40class support for retries and timeouts
  126. 3:41and item potency so that we never kick
  127. 3:43off duplicate runs of the same file.
  128. 3:45To split up this work, we split this
  129. 3:47project into four milestones. The first
  130. 3:49was local validation. Basically, is this
  131. 3:51project even viable? The second was a
  132. 3:52scheduled backfield workflow to write
  133. 3:54tests for existing code. The third was a
  134. 3:56PR life cycle workflow to actually get
  135. 3:58those PRs merged. And the fourth was a
  136. 3:59label trigger so engineers can get tests
  137. 4:01for the new code that they're writing.
  138. 4:03Now, we'll go through each of these in
  139. 4:04order. Before building anything real, we
  140. 4:06wanted to do the least amount of work
  141. 4:07possible that would tell us whether an
  142. 4:09agent could actually do this job well
  143. 4:10and do it at scale. We started with a
  144. 4:12very simple local cloud code setup where
  145. 4:14we had one cloud code command that acted
  146. 4:15as an orchestrator. It ran an Xcode
  147. 4:18build script to generate a code coverage
  148. 4:19report locally. Then it fed that report
  149. 4:21through a file selection script that we
  150. 4:22wrote and then it handed each selected
  151. 4:24file off of sub two sub aents. One wrote
  152. 4:27the tests and the other one opened the
  153. 4:28PR. We ran this in a loop and generated
  154. 4:30a bunch of PRs. We basically ran this in
  155. 4:32two batches. The first patch had 17 PRs
  156. 4:34and we merged eight of them. The second
  157. 4:36had 40 PRs and we merged 19. So that's
  158. 4:38roughly a 50% merge rate. But the goal
  159. 4:39here wasn't to merge a lot of PRs. It
  160. 4:41was to learn how the agents fail and to
  161. 4:42fix those paper cuts before
  162. 4:43productionizing anything. Here are some
  163. 4:45of the failure modes that we saw. Since
  164. 4:47we're a large legacy code base with both
  165. 4:48XC test and swift testing, the agents
  166. 4:50would sometimes get confused and mix the
  167. 4:51two in one file. We fixed that by giving
  168. 4:53the agents clear rules on when to use
  169. 4:54each. We also had a giant user client
  170. 4:56singleton class. Some code passed around
  171. 4:58the concrete version of this class which
  172. 4:59is really hard to mock whereas others
  173. 5:00passed around a protocolized version. So
  174. 5:02we migrated the entire codebase over to
  175. 5:03the protocolized version and added lint
  176. 5:05rule to enforce it. This made the whole
  177. 5:07host of files easier to test for agents
  178. 5:09and for humans. We also ran into six
  179. 5:12concurrency related issues and agents
  180. 5:13writing timing based tests which are a
  181. 5:15big source of flakiness in CI. So we
  182. 5:16improved the state for both of these
  183. 5:18with better rules and examples in our
  184. 5:19testing skills.
  185. 5:21Now this brings us to milestone 2. Since
  186. 5:23we were so far behind on code coverage,
  187. 5:24the biggest leverage that we saw was
  188. 5:26backfilling tests for code that was
  189. 5:27already merged in. That's where the
  190. 5:29scheduled backfill workflow comes in.
  191. 5:30This is a system design diagram of the
  192. 5:32workflow as it runs in production and it
  193. 5:34essentially does two jobs. The left half
  194. 5:35is to identify which is picking the
  195. 5:37files that need tests and the right half
  196. 5:39is right which is having the agents
  197. 5:41actually write those tests. Let's start
  198. 5:43with identify. Every time a PR merges to
  199. 5:45the main branch in the iOS repo, we kick
  200. 5:46off a GitHub access job that generates
  201. 5:48an Xcode coverage report and uploads it
  202. 5:50to S3 that becomes a source of truth for
  203. 5:52the latest code coverage. The back fill
  204. 5:54workflow then fetches that report and
  205. 5:55runs it through a selector script which
  206. 5:56picks the next set of files to test.
  207. 5:59Let's get a deeper a little deeper into
  208. 6:00the selector script. Its job is
  209. 6:02essentially to score every Swift file
  210. 6:03based on how confident we are that an
  211. 6:05agent can write a good test for it. The
  212. 6:07strongest signal is file type. So view
  213. 6:09models, repositories, and data sources
  214. 6:10get a 2.5x boost while legacy files get
  215. 6:12pushed down. We skipped UI code
  216. 6:14completely for now since snapshot tests
  217. 6:15are better fit there. And larger files
  218. 6:17also get a penalty. So we prioritize
  219. 6:19code that we're confident that we can
  220. 6:20test well.
  221. 6:22The next step is to actually write the
  222. 6:24tests. We kick off one child workflow
  223. 6:26per file, all running in parallel. In
  224. 6:28each one of these runs, the agent clones
  225. 6:29the iOS repository, reads the testing
  226. 6:31skill that we keep in the repo, which
  227. 6:33spells out how to write a good test.
  228. 6:34Then it writes the test plus any mocks
  229. 6:36and fakes it needs and opens a PR with a
  230. 6:38generated test label so we can identify
  231. 6:40it easily later.
  232. 6:42These agents run on Dualingo's cloud
  233. 6:44coding agent which is a platform that
  234. 6:45also runs on Neporal. The simplest way
  235. 6:46to think about it is as an abstraction
  236. 6:48on top of cloud code and codex running
  237. 6:49in the cloud. We can invoke it from
  238. 6:51Slack, temporal or GitHub. It talks to a
  239. 6:52bunch of our internal tools for MCP and
  240. 6:54it logs a bunch of events so we can see
  241. 6:56what each agent run is doing and how
  242. 6:57much it costs. We started running these
  243. 7:00on Linux machines and then later move
  244. 7:02them to MacRunners which will become
  245. 7:03important in a minute. Here's what a
  246. 7:05generated test PR looks like. Every PR
  247. 7:07description follows the same template
  248. 7:09and it says explicitly that the PR is AI
  249. 7:10generated. The reviewer has four
  250. 7:12options. They can approve it and the PR
  251. 7:14automarges. They can push changes to it
  252. 7:16directly. They can reassign it to
  253. 7:17someone better suited or they can close
  254. 7:18it out with feedback on why it isn't
  255. 7:20mergeable.
  256. 7:21Now once we had the backfl running in
  257. 7:23production we had a slightly new
  258. 7:24problem. We suddenly had tens of tens to
  259. 7:26hundreds of open PRs that nobody could
  260. 7:27manage by hand. So how do we get all of
  261. 7:29these to a mergeable state and in front
  262. 7:30of the right reviewer
  263. 7:32that's where the PR life cycle workflow
  264. 7:34comes in. You can think of it as a state
  265. 7:36machine. It runs every couple of hours
  266. 7:37looks through every open generated test
  267. 7:39PR and takes exactly one action on each.
  268. 7:42It covers the next two steps of our
  269. 7:43loop. Verify which is does this PR pass
  270. 7:46CI and is it high quality and resolve
  271. 7:47which is getting a human to review it
  272. 7:49and monitor it. The logic for this
  273. 7:51workflow is a simple decision tree. If
  274. 7:52there are any merge conflicts or the PR
  275. 7:54has been open for about 3 weeks, we
  276. 7:55close it out and mark it as still. If CI
  277. 7:57is still running or a reviewer is
  278. 7:58already on it, we skip it. If CI is
  279. 8:00green and there's no reviewers, we find
  280. 8:01one assign them. If CI is red, we try to
  281. 8:03fix it. And if a reviewer rejects the
  282. 8:05PR, the file goes into 30-day cool down.
  283. 8:07So, we're not spamming engineers with
  284. 8:08the same PR again and again.
  285. 8:10When CI fails, the first thing we check
  286. 8:12is which job failed. If it's pre-commit,
  287. 8:14a GitHub action applies the fix. It's
  288. 8:16cheap, it's deterministic, and there's
  289. 8:17no agent needed. If it's a real test
  290. 8:18failure, we kick off another agent that
  291. 8:20reads the CI logs and pushes a fix to
  292. 8:21the PR. We cap that whole loop at five
  293. 8:23attemps. Past that we close the PR
  294. 8:25rather than trying to fix it forever.
  295. 8:27Once CI is green, we need to find the
  296. 8:29right reviewer. What we do is we run git
  297. 8:31blame on the source file being tested.
  298. 8:32Find the top contributor and assign it
  299. 8:34to them and turn on auto merge. If they
  300. 8:35approve it, the PR gets merged. If they
  301. 8:37rejected, the file goes into the 30-day
  302. 8:38cool down, which I previously mentioned.
  303. 8:40This brings us to milestone four. Once
  304. 8:42all of this was back filling tests for
  305. 8:44existing code, we wanted to make it just
  306. 8:45as easy to get tests for the new code
  307. 8:47engineers were writing. Now any engineer
  308. 8:49can add or write test label to their own
  309. 8:50PR, it hooks into the same temporal
  310. 8:52infrastructure we just walked through.
  311. 8:54The only real difference is selection.
  312. 8:55Instead of the selection script, we take
  313. 8:57the file straight from the engineer's
  314. 8:58PR. The agent writes the test and you
  315. 9:00get a draft SPR link back to your
  316. 9:01original one. Now let's talk about some
  317. 9:03of the challenges that we ran into. Our
  318. 9:04agents originally ran on Linux machines
  319. 9:06which don't have X code or Swift
  320. 9:07installed. So when an agent would open a
  321. 9:09PR after about 20 or 30 minutes CI would
  322. 9:11fail frequently on lint. This wastes the
  323. 9:13agents time and CI time as well which is
  324. 9:15an increasingly scarce resource. For a
  325. 9:17while 30 13.6% of our PRs were failing
  326. 9:19due to lint and then when we dug into it
  327. 9:2162% of those failures came from just to
  328. 9:22lint rules. We fixed this in layers. The
  329. 9:25first was better examples in the prompt
  330. 9:27along with pitfalls to avoid. The second
  331. 9:28was a GitHub action that auto fixes lint
  332. 9:30errors in CI. The third was an audit of
  333. 9:32the rules that couldn't be autofixed. We
  334. 9:34essentially removed the ones that
  335. 9:35weren't high signal and this made life
  336. 9:36easier for both agents and human
  337. 9:38engineers. And finally, the real fix
  338. 9:39came when we moved the agents to run on
  339. 9:41Mac runners. So now the agents could
  340. 9:42build and lend the code locally before
  341. 9:44ever opening a PR. The second challenge
  342. 9:47was PR quality. When we assign a PR to a
  343. 9:49human, we wanted to be something they'd
  344. 9:50actually feel comfortable proving. But
  345. 9:52agents sometimes take the shortest paths
  346. 9:53to green CI. We saw agents editing
  347. 9:55production code to make a failing test
  348. 9:56pass. We also saw them modifying,
  349. 9:58deleting, or silencing existing tests.
  350. 9:59And then we saw them adding long weight
  351. 10:01to tests which can make CI flaky.
  352. 10:03Again, there was no single fix here, so
  353. 10:05we built three layers. The first is the
  354. 10:07deterministic rule. If a PR touches any
  355. 10:09file that isn't a test, a mock, a fake,
  356. 10:11or a stub, we close it out. The second
  357. 10:13builds on our AI code review. At
  358. 10:14Dualingo, we use Claude and Codex to
  359. 10:16review every PR. And those B bots often
  360. 10:18leave comments. So, we built an agent
  361. 10:20that resolves any P 0 comments before a
  362. 10:22human ever looks at the PR. And the
  363. 10:24third is an LLM as a judge. It's a
  364. 10:25reviewer agent built just for these test
  365. 10:27PRs. It reads the PR diff and rejects
  366. 10:29production edits, changed or deleted
  367. 10:30tests, disabled tests, and long weights.
  368. 10:33And as you learn more, we keep adding
  369. 10:34additional checks to it.
  370. 10:36So far, the LM as a judge has reviewed
  371. 10:38about,200 PRs and rejected 73 of them.
  372. 10:41Every rejection comes with a comment
  373. 10:42like this one, saying exactly why the PR
  374. 10:44was rejected. We then read that feedback
  375. 10:46and fold it back into our prompts and
  376. 10:47skills, so the same mistakes get less
  377. 10:48likely each time. So, let's talk about
  378. 10:51what all of this added up to. Since we
  379. 10:53started, the pipeline has merged more
  380. 10:54than 700 test PRs, adding about 225,000
  381. 10:57lines of test code. Every one of those
  382. 10:59PRs was reviewed and approved by a
  383. 11:00human. MVVM test coverage went from 9%
  384. 11:03to 59% and overall coverage went from
  385. 11:0411% to 24%. Most importantly, we now
  386. 11:07have a code base so engineers and agents
  387. 11:08can ship faster and with more
  388. 11:10confidence. Here's a few more
  389. 11:12interesting stats. Since March, the
  390. 11:13pipeline has written 34% of all new test
  391. 11:15files in the repo, 44% of all new test
  392. 11:17cases, and 10% of every swift line added
  393. 11:19to the codebase. And interestingly, that
  394. 11:20share has actually dripped in the last
  395. 11:22few weeks because engineers are now
  396. 11:23writing a lot more tests themselves,
  397. 11:24which is honestly the end goal.
  398. 11:27Um, and here's the coverage curve since
  399. 11:29April when the backfield started running
  400. 11:30on a schedule. You can see it bent
  401. 11:32slightly upward around June and July,
  402. 11:33which is when we really ramped up the
  403. 11:35pipeline.
  404. 11:36So, what have we learned while doing
  405. 11:37this project? The first thing is that
  406. 11:39this pattern generalizes the loop we
  407. 11:41talked about identify, write, verify,
  408. 11:43resolve, iterate. It isn't specific to
  409. 11:44testing. Unit tests were just the lowest
  410. 11:46hanging fruit and any job that fits the
  411. 11:48shape can run on the same infrastructure
  412. 11:49and that's exactly what we're starting
  413. 11:50to do at the company. We're applying the
  414. 11:52same loop to modernization to
  415. 11:53modernizing screens to MVM and Swift UI,
  416. 11:55migrating code uh on Android to Corines
  417. 11:58and generating snapshot tests.
  418. 12:00So before I wrap up, I want to leave you
  419. 12:02with a thought experiment. There's a new
  420. 12:04class of fast general purpose
  421. 12:05classifiers like Jev. They taken some
  422. 12:07state and output an action in
  423. 12:08milliseconds. So what if we replace a
  424. 12:10handwritten decision tree in the PR life
  425. 12:11cycle workflow with one of these? PR
  426. 12:13state goes in, the classifier picks the
  427. 12:15next action and we execute it all in
  428. 12:16many seconds.
  429. 12:19I also want to thank a few folks who
  430. 12:20worked on this with me, Jesse, Gordalup,
  431. 12:22Hunter and Jiho. They built a lot of the
  432. 12:23platform pieces this project stands on
  433. 12:25and helped review a lot of this code.
  434. 12:27That's it for me. Thank you so much for
  435. 12:28making it this far. If you want to go
  436. 12:29deeper, we've written about this
  437. 12:30pipeline on the dualing engineering
  438. 12:32blog. And I'd also love to hear from you
  439. 12:33if you're building something similar.
  440. 12:34You can find me on LinkedIn or next

About this transcript

This page contains the full transcript of 225,000 Lines of Tests, Written and Merged by Agents - Kush Agrawal, Duolingo by AI Engineer, generated from the public captions YouTube serves with the video. The transcript has 3,148 words across 440 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.