225,000 Lines of Tests, Written and Merged by Agents - Kush Agrawal, Duolingo — Transcript
Full transcript
- 0:01Hey everyone, thanks for tuning in.
- 0:03Today I'm going to walk you through how
- 0:04we use AI agents at Dualingo to write
- 0:06and ship more than 200,000 thousand
- 0:07lines of unit tests on iOS codebase.
- 0:09I'll share what we built and what we
- 0:11learned along the way. Uh I'm Kush. I'm
- 0:13a staff engineer on the platform team
- 0:14here at Dualingo. My job is to build
- 0:15developer tooling and AI assisted
- 0:16workflows for the 100 plus engineers who
- 0:18contribute to our client apps. Before
- 0:19Duelingo, I spent close to a decade at
- 0:21Stripe and Uber working across a few
- 0:22teams there. Let's start with some
- 0:24background on the scale of mobile at
- 0:26Dualingo. Uh we ship our apps to the app
- 0:29store every week. Um we're also a very
- 0:31experiment-driven company. So at any
- 0:32given point of time, we have more than
- 0:34600 AB tests running live and our code
- 0:36bases are pretty mature. Both the iOS
- 0:38and Android apps have been around for
- 0:39more than 10 years at this point.
- 0:42Here's also the number of executable
- 0:43lines of app code in our iOS repository.
- 0:45So as you can see back in 2019, we were
- 0:47at about 200,000 lines and today we're
- 0:49hit close to 2 million. And as AI
- 0:51assisted coding picks up, you can see
- 0:52that curve starting to bend upward.
- 0:55And it's not just the amount of code
- 0:56that's changing, it's also who's writing
- 0:57it. So last year about 200 people
- 1:00contributed to the iOS repository. And
- 1:01this year we've already crossed 250. And
- 1:03a lot of these folks aren't even iOS
- 1:04engineers. We have backend engineers,
- 1:06Android engineers, even PMs and
- 1:07designers all shipping code to the iOS
- 1:09app. And that's mostly been unlocked by
- 1:11AI coding tools like cursor and cloud
- 1:13code. Um, another thing to note about
- 1:15our codebase is that it's a codebase in
- 1:16transition. So a few years ago, we
- 1:18standardized on the MVVM architecture
- 1:19pattern. Today about half the code in
- 1:21our repository follows that pattern and
- 1:23the other half is the legacy code that
- 1:24we're in the process of migrating. Since
- 1:26the MVVM half is the easiest to test,
- 1:28that's where we focused on this project.
- 1:31Here's where we started back in March of
- 1:32this year. We were at overall unit test
- 1:35coverage of 11%. The MVVM layers were a
- 1:37bit lower, repositories were at 11%,
- 1:39view models at 9, data sources at 5, and
- 1:41combined we are about 9% for the MVM
- 1:43slice of the app.
- 1:46So why does this matter now more than it
- 1:48used to? As agents and humans write more
- 1:50code and write it faster, writing code
- 1:52isn't the hard part anymore. Verifying
- 1:53it is. So when someone puts up a PR, how
- 1:56do you know that it's actually safe to
- 1:57ship? The bottleneck is really shifting
- 1:58from writing to verifying. And that's
- 2:00where unit tests come in. They make up
- 2:02the base of the testing pyramid. They're
- 2:04pretty cheap to write and pass to run.
- 2:05And they also act as guardrails for
- 2:06agents. So when an agent's change breaks
- 2:08a test in CI, it gets immediate feedback
- 2:10on what to fix. They also act as living
- 2:12documentation on what the code is
- 2:13supposed to do. And here's where we
- 2:15ended up 8 months later. Overall unit
- 2:16test coverage went from 11% to 24%.
- 2:19Repositories went from 11 to 72. View
- 2:21models from 9 to 58. Data sources from 5
- 2:23to 41. And the combined MVM slice of the
- 2:25app went from 9% to 59%. So what did we
- 2:28actually end up building? On paper, it's
- 2:30a pretty simple loop. First, you
- 2:32identify the files that need tests. Then
- 2:33you write those tests with an agent. You
- 2:35verify that the tests are good quality
- 2:36and they pass CI. Then you resolve the
- 2:38PR, which means getting attributed by an
- 2:39engineer merged. And finally, it on the
- 2:41feedback so that the next batch of PRs
- 2:43is better than the last one. But in
- 2:45production, it's a little messier. This
- 2:47is what our system ended up looking
- 2:48like. I won't go through every box right
- 2:49now, but at a high level, it's a few
- 2:50temporal workflows that coordinate
- 2:52through shared state stored in S3. And
- 2:54every box you see here maps back to one
- 2:56of the five steps I just talked about.
- 2:58Here are the building blocks that we
- 2:59ended up using. Temporal handles
- 3:01workflow orchestration and execution.
- 3:03Cloud code acts as the coding agent. S3
- 3:05holds a shared state that keeps our
- 3:06workflows in sync. And we use GitHub for
- 3:08actions, labels, and PRs. Data dogs acts
- 3:11as observability layer where we keep an
- 3:12eye on the health of our pipelines.
- 3:15If you haven't used empodoral before,
- 3:16it's basically a durable workflow
- 3:18execution engine. You write your
- 3:19workflows as normal Python code and if a
- 3:21worker crashes, the workflow just picks
- 3:22up right where it left off. This fits
- 3:24agents really well and agent session can
- 3:26run for seconds to hours and it can call
- 3:28out to tools like GitHub, CI, S3, and
- 3:30Claude. We also lean on a few built-in
- 3:32features of Temporal. There are
- 3:34schedules which are basically chronike
- 3:35schedules for our jobs. There are
- 3:36composible workflows so that we can fan
- 3:38out work in parallel. Then there's first
- 3:40class support for retries and timeouts
- 3:41and item potency so that we never kick
- 3:43off duplicate runs of the same file.
- 3:45To split up this work, we split this
- 3:47project into four milestones. The first
- 3:49was local validation. Basically, is this
- 3:51project even viable? The second was a
- 3:52scheduled backfield workflow to write
- 3:54tests for existing code. The third was a
- 3:56PR life cycle workflow to actually get
- 3:58those PRs merged. And the fourth was a
- 3:59label trigger so engineers can get tests
- 4:01for the new code that they're writing.
- 4:03Now, we'll go through each of these in
- 4:04order. Before building anything real, we
- 4:06wanted to do the least amount of work
- 4:07possible that would tell us whether an
- 4:09agent could actually do this job well
- 4:10and do it at scale. We started with a
- 4:12very simple local cloud code setup where
- 4:14we had one cloud code command that acted
- 4:15as an orchestrator. It ran an Xcode
- 4:18build script to generate a code coverage
- 4:19report locally. Then it fed that report
- 4:21through a file selection script that we
- 4:22wrote and then it handed each selected
- 4:24file off of sub two sub aents. One wrote
- 4:27the tests and the other one opened the
- 4:28PR. We ran this in a loop and generated
- 4:30a bunch of PRs. We basically ran this in
- 4:32two batches. The first patch had 17 PRs
- 4:34and we merged eight of them. The second
- 4:36had 40 PRs and we merged 19. So that's
- 4:38roughly a 50% merge rate. But the goal
- 4:39here wasn't to merge a lot of PRs. It
- 4:41was to learn how the agents fail and to
- 4:42fix those paper cuts before
- 4:43productionizing anything. Here are some
- 4:45of the failure modes that we saw. Since
- 4:47we're a large legacy code base with both
- 4:48XC test and swift testing, the agents
- 4:50would sometimes get confused and mix the
- 4:51two in one file. We fixed that by giving
- 4:53the agents clear rules on when to use
- 4:54each. We also had a giant user client
- 4:56singleton class. Some code passed around
- 4:58the concrete version of this class which
- 4:59is really hard to mock whereas others
- 5:00passed around a protocolized version. So
- 5:02we migrated the entire codebase over to
- 5:03the protocolized version and added lint
- 5:05rule to enforce it. This made the whole
- 5:07host of files easier to test for agents
- 5:09and for humans. We also ran into six
- 5:12concurrency related issues and agents
- 5:13writing timing based tests which are a
- 5:15big source of flakiness in CI. So we
- 5:16improved the state for both of these
- 5:18with better rules and examples in our
- 5:19testing skills.
- 5:21Now this brings us to milestone 2. Since
- 5:23we were so far behind on code coverage,
- 5:24the biggest leverage that we saw was
- 5:26backfilling tests for code that was
- 5:27already merged in. That's where the
- 5:29scheduled backfill workflow comes in.
- 5:30This is a system design diagram of the
- 5:32workflow as it runs in production and it
- 5:34essentially does two jobs. The left half
- 5:35is to identify which is picking the
- 5:37files that need tests and the right half
- 5:39is right which is having the agents
- 5:41actually write those tests. Let's start
- 5:43with identify. Every time a PR merges to
- 5:45the main branch in the iOS repo, we kick
- 5:46off a GitHub access job that generates
- 5:48an Xcode coverage report and uploads it
- 5:50to S3 that becomes a source of truth for
- 5:52the latest code coverage. The back fill
- 5:54workflow then fetches that report and
- 5:55runs it through a selector script which
- 5:56picks the next set of files to test.
- 5:59Let's get a deeper a little deeper into
- 6:00the selector script. Its job is
- 6:02essentially to score every Swift file
- 6:03based on how confident we are that an
- 6:05agent can write a good test for it. The
- 6:07strongest signal is file type. So view
- 6:09models, repositories, and data sources
- 6:10get a 2.5x boost while legacy files get
- 6:12pushed down. We skipped UI code
- 6:14completely for now since snapshot tests
- 6:15are better fit there. And larger files
- 6:17also get a penalty. So we prioritize
- 6:19code that we're confident that we can
- 6:20test well.
- 6:22The next step is to actually write the
- 6:24tests. We kick off one child workflow
- 6:26per file, all running in parallel. In
- 6:28each one of these runs, the agent clones
- 6:29the iOS repository, reads the testing
- 6:31skill that we keep in the repo, which
- 6:33spells out how to write a good test.
- 6:34Then it writes the test plus any mocks
- 6:36and fakes it needs and opens a PR with a
- 6:38generated test label so we can identify
- 6:40it easily later.
- 6:42These agents run on Dualingo's cloud
- 6:44coding agent which is a platform that
- 6:45also runs on Neporal. The simplest way
- 6:46to think about it is as an abstraction
- 6:48on top of cloud code and codex running
- 6:49in the cloud. We can invoke it from
- 6:51Slack, temporal or GitHub. It talks to a
- 6:52bunch of our internal tools for MCP and
- 6:54it logs a bunch of events so we can see
- 6:56what each agent run is doing and how
- 6:57much it costs. We started running these
- 7:00on Linux machines and then later move
- 7:02them to MacRunners which will become
- 7:03important in a minute. Here's what a
- 7:05generated test PR looks like. Every PR
- 7:07description follows the same template
- 7:09and it says explicitly that the PR is AI
- 7:10generated. The reviewer has four
- 7:12options. They can approve it and the PR
- 7:14automarges. They can push changes to it
- 7:16directly. They can reassign it to
- 7:17someone better suited or they can close
- 7:18it out with feedback on why it isn't
- 7:20mergeable.
- 7:21Now once we had the backfl running in
- 7:23production we had a slightly new
- 7:24problem. We suddenly had tens of tens to
- 7:26hundreds of open PRs that nobody could
- 7:27manage by hand. So how do we get all of
- 7:29these to a mergeable state and in front
- 7:30of the right reviewer
- 7:32that's where the PR life cycle workflow
- 7:34comes in. You can think of it as a state
- 7:36machine. It runs every couple of hours
- 7:37looks through every open generated test
- 7:39PR and takes exactly one action on each.
- 7:42It covers the next two steps of our
- 7:43loop. Verify which is does this PR pass
- 7:46CI and is it high quality and resolve
- 7:47which is getting a human to review it
- 7:49and monitor it. The logic for this
- 7:51workflow is a simple decision tree. If
- 7:52there are any merge conflicts or the PR
- 7:54has been open for about 3 weeks, we
- 7:55close it out and mark it as still. If CI
- 7:57is still running or a reviewer is
- 7:58already on it, we skip it. If CI is
- 8:00green and there's no reviewers, we find
- 8:01one assign them. If CI is red, we try to
- 8:03fix it. And if a reviewer rejects the
- 8:05PR, the file goes into 30-day cool down.
- 8:07So, we're not spamming engineers with
- 8:08the same PR again and again.
- 8:10When CI fails, the first thing we check
- 8:12is which job failed. If it's pre-commit,
- 8:14a GitHub action applies the fix. It's
- 8:16cheap, it's deterministic, and there's
- 8:17no agent needed. If it's a real test
- 8:18failure, we kick off another agent that
- 8:20reads the CI logs and pushes a fix to
- 8:21the PR. We cap that whole loop at five
- 8:23attemps. Past that we close the PR
- 8:25rather than trying to fix it forever.
- 8:27Once CI is green, we need to find the
- 8:29right reviewer. What we do is we run git
- 8:31blame on the source file being tested.
- 8:32Find the top contributor and assign it
- 8:34to them and turn on auto merge. If they
- 8:35approve it, the PR gets merged. If they
- 8:37rejected, the file goes into the 30-day
- 8:38cool down, which I previously mentioned.
- 8:40This brings us to milestone four. Once
- 8:42all of this was back filling tests for
- 8:44existing code, we wanted to make it just
- 8:45as easy to get tests for the new code
- 8:47engineers were writing. Now any engineer
- 8:49can add or write test label to their own
- 8:50PR, it hooks into the same temporal
- 8:52infrastructure we just walked through.
- 8:54The only real difference is selection.
- 8:55Instead of the selection script, we take
- 8:57the file straight from the engineer's
- 8:58PR. The agent writes the test and you
- 9:00get a draft SPR link back to your
- 9:01original one. Now let's talk about some
- 9:03of the challenges that we ran into. Our
- 9:04agents originally ran on Linux machines
- 9:06which don't have X code or Swift
- 9:07installed. So when an agent would open a
- 9:09PR after about 20 or 30 minutes CI would
- 9:11fail frequently on lint. This wastes the
- 9:13agents time and CI time as well which is
- 9:15an increasingly scarce resource. For a
- 9:17while 30 13.6% of our PRs were failing
- 9:19due to lint and then when we dug into it
- 9:2162% of those failures came from just to
- 9:22lint rules. We fixed this in layers. The
- 9:25first was better examples in the prompt
- 9:27along with pitfalls to avoid. The second
- 9:28was a GitHub action that auto fixes lint
- 9:30errors in CI. The third was an audit of
- 9:32the rules that couldn't be autofixed. We
- 9:34essentially removed the ones that
- 9:35weren't high signal and this made life
- 9:36easier for both agents and human
- 9:38engineers. And finally, the real fix
- 9:39came when we moved the agents to run on
- 9:41Mac runners. So now the agents could
- 9:42build and lend the code locally before
- 9:44ever opening a PR. The second challenge
- 9:47was PR quality. When we assign a PR to a
- 9:49human, we wanted to be something they'd
- 9:50actually feel comfortable proving. But
- 9:52agents sometimes take the shortest paths
- 9:53to green CI. We saw agents editing
- 9:55production code to make a failing test
- 9:56pass. We also saw them modifying,
- 9:58deleting, or silencing existing tests.
- 9:59And then we saw them adding long weight
- 10:01to tests which can make CI flaky.
- 10:03Again, there was no single fix here, so
- 10:05we built three layers. The first is the
- 10:07deterministic rule. If a PR touches any
- 10:09file that isn't a test, a mock, a fake,
- 10:11or a stub, we close it out. The second
- 10:13builds on our AI code review. At
- 10:14Dualingo, we use Claude and Codex to
- 10:16review every PR. And those B bots often
- 10:18leave comments. So, we built an agent
- 10:20that resolves any P 0 comments before a
- 10:22human ever looks at the PR. And the
- 10:24third is an LLM as a judge. It's a
- 10:25reviewer agent built just for these test
- 10:27PRs. It reads the PR diff and rejects
- 10:29production edits, changed or deleted
- 10:30tests, disabled tests, and long weights.
- 10:33And as you learn more, we keep adding
- 10:34additional checks to it.
- 10:36So far, the LM as a judge has reviewed
- 10:38about,200 PRs and rejected 73 of them.
- 10:41Every rejection comes with a comment
- 10:42like this one, saying exactly why the PR
- 10:44was rejected. We then read that feedback
- 10:46and fold it back into our prompts and
- 10:47skills, so the same mistakes get less
- 10:48likely each time. So, let's talk about
- 10:51what all of this added up to. Since we
- 10:53started, the pipeline has merged more
- 10:54than 700 test PRs, adding about 225,000
- 10:57lines of test code. Every one of those
- 10:59PRs was reviewed and approved by a
- 11:00human. MVVM test coverage went from 9%
- 11:03to 59% and overall coverage went from
- 11:0411% to 24%. Most importantly, we now
- 11:07have a code base so engineers and agents
- 11:08can ship faster and with more
- 11:10confidence. Here's a few more
- 11:12interesting stats. Since March, the
- 11:13pipeline has written 34% of all new test
- 11:15files in the repo, 44% of all new test
- 11:17cases, and 10% of every swift line added
- 11:19to the codebase. And interestingly, that
- 11:20share has actually dripped in the last
- 11:22few weeks because engineers are now
- 11:23writing a lot more tests themselves,
- 11:24which is honestly the end goal.
- 11:27Um, and here's the coverage curve since
- 11:29April when the backfield started running
- 11:30on a schedule. You can see it bent
- 11:32slightly upward around June and July,
- 11:33which is when we really ramped up the
- 11:35pipeline.
- 11:36So, what have we learned while doing
- 11:37this project? The first thing is that
- 11:39this pattern generalizes the loop we
- 11:41talked about identify, write, verify,
- 11:43resolve, iterate. It isn't specific to
- 11:44testing. Unit tests were just the lowest
- 11:46hanging fruit and any job that fits the
- 11:48shape can run on the same infrastructure
- 11:49and that's exactly what we're starting
- 11:50to do at the company. We're applying the
- 11:52same loop to modernization to
- 11:53modernizing screens to MVM and Swift UI,
- 11:55migrating code uh on Android to Corines
- 11:58and generating snapshot tests.
- 12:00So before I wrap up, I want to leave you
- 12:02with a thought experiment. There's a new
- 12:04class of fast general purpose
- 12:05classifiers like Jev. They taken some
- 12:07state and output an action in
- 12:08milliseconds. So what if we replace a
- 12:10handwritten decision tree in the PR life
- 12:11cycle workflow with one of these? PR
- 12:13state goes in, the classifier picks the
- 12:15next action and we execute it all in
- 12:16many seconds.
- 12:19I also want to thank a few folks who
- 12:20worked on this with me, Jesse, Gordalup,
- 12:22Hunter and Jiho. They built a lot of the
- 12:23platform pieces this project stands on
- 12:25and helped review a lot of this code.
- 12:27That's it for me. Thank you so much for
- 12:28making it this far. If you want to go
- 12:29deeper, we've written about this
- 12:30pipeline on the dualing engineering
- 12:32blog. And I'd also love to hear from you
- 12:33if you're building something similar.
- 12:34You can find me on LinkedIn or next
About this transcript
This page contains the full transcript of 225,000 Lines of Tests, Written and Merged by Agents - Kush Agrawal, Duolingo by AI Engineer, generated from the public captions YouTube serves with the video. The transcript has 3,148 words across 440 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.