YT YouTube2Text

YouTube2Text → Transcripts → How Claude Fixes It's Own Code

How Claude Fixes It's Own Code — Transcript

by Theo - t3․gg · 9,307 words · 1,296 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00We've spent a ton of time recently
0:01talking about how to build as
0:02effectively with agents as possible. But
0:04I don't think we've talked enough about
0:05the communication around it. How do we
0:07work together as a team? How do we
0:09review code? How do we choose what
0:10merges and what doesn't? How do we
0:12manage all of this? Especially at the
0:14scale we can ship at now with tools like
0:16Opus 5.5. As you can imagine, anthropic
0:19teams like the Cloud Code team are
0:21experiencing this at a scale that is
0:23hard to comprehend. They can't even use
0:25Git and GitHub anymore because they slow
0:27them down too much. So, how does a team
0:29possibly review all of the change
0:31requests from other people? How do you
0:33manage a poll request load like this?
0:36We've been experiencing this ourselves
0:38with T3 Code as we've been getting as
0:40many as 800 PRs filed a day. Figuring
0:42out how to get through all of that with
0:44a small threeperson team is nearly
0:46impossible. This is just a different
0:48world we're in. And the idea of how code
0:51is reviewed and landed is and should be
0:54done in a fundamentally different way.
0:56And that's what we are here to talk
0:58about today. How do you handle this
1:01level of load, this burden of constant
1:04requests for changes being made on all
1:07of our repos now that Opus has made it
1:09as easy as sending a prompt to get a
1:10pull request? How do you decide? How do
1:13you manage? How do you deal with all of
1:14this? I have a feeling this is going to
1:16be a fun one. I've actually not read
1:18this yet because I wanted to read it
1:19with y'all together. And I'm hoping we
1:21can learn some things too here because
1:23this isn't easy and figuring this out is
1:25a a really important piece of getting
1:28this new era of engineering right. I
1:30also think I should be able to share
1:31some special insights at the end on how
1:33we were able to ramp up so much with T3
1:35code. I'm very excited to show off more
1:37of our process cuz it's been getting
1:38really fun. Julius is cooking like I've
1:40never seen. But I do have one other cool
1:42thing I want to show you first. Today's
1:44sponsor. When you think about it, it's
1:45kind of crazy how much more powerful
1:47agents have gotten recently. And it's
1:48not just cuz the models have gotten
1:50smarter. The majority of the wins have
1:51come from the fact that the tools have
1:53gotten so much better. And more
1:54importantly, the models have gotten
1:55better at using those tools. Now that
1:57agents can do so much more on our
1:59behalf, it's important that we give them
2:00what they need to do those things. And
2:02when we see tools like Grockbot, Muse,
2:04and more all taking over our day-to-day
2:07lives, they need to be able to do things
2:08on our behalf. A lot of those things are
2:10on a browser. 85% of the APIs on the web
2:14are not accessible to just be hit over
2:16curl. You need to actually use the
2:17website in order to do things. You could
2:19build that all yourself or you could use
2:21today's sponsor, Browserbase. These guys
2:23have built the best portal for the web
2:25for your agents. The simplest way to
2:27think about it is that they turn
2:28everything on the web into a
2:30programmable API. Your agents need
2:32search, they have you covered. Do you
2:33need to fetch content from a page in a
2:35format your agents can use? They have
2:36you covered. Whether it's HTML, JSON, or
2:39markdown, they can get it from the web.
2:40And of course, if your agents want to do
2:42things on the web, like click buttons,
2:44navigate, fill out forms, clear
2:46captions, and more, that's where the
2:48browser as a service part comes in.
2:50There is so much work blocked by the
2:52browser. And if you give your agents the
2:54ability to do it, you'll be amazed at
2:55what they can do. If you don't have any
2:57ideas yet, just go ask Claude. Say,
2:59"Hey, look at what I do every day. See
3:01how much of this can be automated." I
3:03bet you'll be surprised. And if you are,
3:04start today at soy.link/browserbase.
3:07Now that's out of the way, we can dive
3:08in. Let's start with how the cloud code
3:11team does code review and then see how
3:13it compares contrasts to the way we do
3:14it and what my team can learn from it
3:16because I'm actually very excited to
3:18possibly improve our process here.
3:20Starts with the obvious question. How do
3:22you do code review at the volume of code
3:24that agents are producing on the cloud
3:26code team? We once had high coverage in
3:28deep reviews. We started optimizing for
3:30the minimum amount of human review time
3:32that doesn't raise reverts or incidents.
3:34Sounds about right. Although this
3:36sentence is horribly formatted. Let's
3:38see how this goes. I've said before that
3:39Claude writes around 90% of Claude
3:41Code's own code. It's got to be more
3:44than that at this point. I don't think I
3:46or anyone on my team has manually edited
3:49a line of code in T3 code other than
3:51like pros in markdown files in at least
3:543 months, probably longer. Just Claude
3:57is probably over 90% of our code and the
4:00rest is codecs. Yeah, I I am confused by
4:03this. I I would have thought it'd be
4:04more by now. Previously, the numbers
4:06were insane. Like we barely touch the
4:08code by hand. It's 85% of our code is
4:10claws. Like that's insane. A year later
4:12it's 90. That feels low. Anyways, how do
4:14they review code at the volume that
4:15they're producing it? Does anyone read
4:17the code anymore? And if something
4:18breaks, who catches it? This post is
4:20meant to cover the code review flow on
4:22the Cloud Code team and how it has
4:23evolved over time. From June to
4:25September, merged PRs to Cloud Code's
4:27codebase have more than doubled while
4:29the share that had to be reverted fell
4:31by more than 40%. This feels as much
4:34like model improvements as it does
4:36process improvements. Like our process
4:39improvements have given us some bigger
4:41wins than this, but model improvements
4:43have given us even bigger than that.
4:44Like I would say our revert rate has
4:46dropped much more than 40% since Opus 55
4:50dropped. Personally, we also had another
4:52big drop when I banned all Gemini models
4:54from landing code. Addy and crew point
4:56out that they didn't remove. They did
4:59move most of the reading of the code to
5:01machines. optimized for human time and
5:04attention and invested heavily in making
5:06it fast and easy to undo mistakes. Is
5:09this going to be a safety nets not
5:10guardrails? Please tell me it's a safety
5:12net guard rails. That's going to make me
5:13so happy. I have a good feeling about
5:15this one. So, we start by rethinking the
5:17purpose of code review. When agents
5:19write code, the number of changes go up
5:22fast. In Anthropics Monoro, changes
5:24landing per day went from 2500 a day in
5:27May to more than 30,000 in September.
5:30Cloud code lives in its own repo
5:32separate from the monor repo, but it's
5:34seeing a lot of pressure, too. When
5:36you're shipping this fast, manual line
5:37by line code review is incredibly
5:39difficult, if not impossible. It's also
5:40interesting to know that cloud code
5:42lives in its own repo. That means they
5:43were considering open sourcing it is
5:45what that means. So, code review has two
5:46main jobs. Find and fix issues before
5:48they ship and more importantly, in my
5:50opinion, building a shared context and
5:52codebase understanding across the team.
5:54This is the most underrated aspect of
5:56code review and I'm still working on
5:58figuring out how to maintain this as the
6:01pace we ship increases. Like how do I
6:03make sure my team understands the crazy
6:05changes I am making to our wire
6:07protocol? How do I make sure that
6:09Julius's changes to our potential
6:11integration of computer use are
6:12understood by the rest of the
6:13contributors? How do I make sure
6:15everyone gets Orchestrator V2 before we
6:17merge it? We had random contributors who
6:20were pushing Orchestrator V2's like
6:21branch really hard before we had landed
6:24it that understood the overhaul of our
6:26orchestration layer better than I do
6:29today before it even merged. That is a
6:31thing that's going to be hard to balance
6:32over time and I'm excited to see where
6:34it goes. So, let's see what their
6:36recommendations are here. Point of this
6:37article is mainly to focus on finding
6:39and fixing issues, but we'll come back
6:40to the second part near the end. Okay, I
6:43can wait. They personally believe that
6:45quote did a human read the code end
6:47quote is a very imprecise proxy for
6:50quote what could go wrong with this
6:51change in which check mdash automated or
6:54human mdash is the right one to catch
6:56it. This is an important detail. I I
6:58feel like the conversation around code
7:01review has eroded horribly because it
7:03was based around this idea that agents
7:06make mistakes so much more aggressively
7:09and actively than humans do. So, if you
7:11don't have a human checking to make sure
7:13the AI didn't make some dumb obvious
7:15mistake that the code is safe, otherwise
7:18you're going to just ship slop. I'm
7:19going to be so real with you. Pretty
7:21much every single regression we have
7:23shipped in either T3 chat or T3 code,
7:25whether or not I reviewed the code, I
7:27would not have caught it in code review.
7:29And I promise you, not a single one of
7:30you would have either. The types of bugs
7:33that get through now are not the ones
7:35where just staring at the code for a
7:37long time, you're going to be able to
7:39deduct what could go wrong. You need
7:42lots of other ways to analyze and verify
7:45your code. I also have the hut take that
7:47most agents are now better at code
7:48review than most humans. And if they're
7:49not better in your codebase, that's cuz
7:51your codebase sucks, not cuz the agents
7:52suck. But that's a conversation for
7:54another time. Asking that question also
7:56changed when reviews happen. Before,
7:58review was a gate at the end of the
8:00road. You write the code, open the PR,
8:02and then you wait for your colleagues.
8:04Now, reviews are running constantly,
8:06more like a spell check while you write.
8:08Most changes have been reviewed and
8:09fixed by Claude at least once before a
8:12human is asked to look. That shouldn't
8:14be most, that should be all. I was
8:16saying this for a bit. If you're putting
8:18up PRs that your agent hasn't even
8:20reviewed yet, you are wasting the time
8:22of your team, and it upsets me.
8:25Apparently Claude also raised their
8:27remaining 10% but they have to fix up
8:28their attribution to be super confident
8:30before they claim that. Good luck.
8:31That's hilarious. Here's a fun example
8:33from chat in my chat of the like bad
8:36code makes it through review. There was
8:37a bug in LVM that broke safe Rust for 10
8:41years. Some of the most well-reed code
8:44by some of the smartest engineers in the
8:45world left an aggressive bug in simply
8:49because the best humans still can't
8:50catch everything. Humans and agents are
8:52both lossy. The main difference here
8:55between the smartest AI and the smartest
8:58human is that the smartest AI isn't
9:00going to get burnt out if you tell it to
9:01check over and over and over and over
9:03again. The humans that are smart enough
9:04to find these things are also busy
9:06enough they're not going to bother. Oh
9:08boy, we're about to get into my favorite
9:09part of this conversation. The only
9:11truly scarce resource right now other
9:13than GPUs, human time and attention.
9:15We're going to get into all my favorite
9:16themes I get to talk about in my other
9:18videos here, aren't we? This is going to
9:19be fun. We want to preserve human time
9:22and attention for issues that only
9:24humans can solve and make sure automated
9:26checks are catching the rest. At our
9:28current rate of change, there is still
9:30an incredible amount of work for humans
9:32to do and our team encounters new and
9:34novel challenges almost daily. I agree
9:37here fully. I would add one other thing.
9:40Human energy is different from time and
9:43attention because certain things demand
9:45more energy and less time than others
9:47do. and figuring out how to minimize the
9:50energy cost to keep humans from doing
9:52things that feel painful and slow and
9:55miserable so that the humans are
9:56spending time doing engaging and
9:58important things is a valuable
10:00distinction. I'll go more into detail on
10:01that in a bit. Human time covers both
10:03sides of a review's cost. The time
10:05people spend reviewing and the time
10:07people spend fixing what bad code
10:09breaks. There's another aspect of time
10:11here which is the time between all of
10:13these parts which is human files PR.
10:16Time is waited for. Another human comes
10:18in and reviews it. Time is waited for.
10:20The original human sees the review,
10:23reads it, goes and makes changes, pushes
10:25out the PR. Time is waited for before
10:27another review happens. That is a lot of
10:30time and more of it's being spent
10:31waiting and the in between of coms than
10:33is being spent actually doing the work.
10:36And this is one of the biggest
10:37bottleneck that prevents things from
10:38shipping. So even if you can make it so
10:40less time is being wasted doing the code
10:43review. If time is still being wasted
10:45fixing the code, it doesn't really
10:48matter that much. So the goal is that
10:50most pull requests in the cloud code
10:51repos for the CLI, the desktop web apps,
10:53and the mobile apps should be able to
10:55merge with at most one human in the loop
10:57who can be the author with no increase
11:00in reverts or in incidents. To measure
11:02human time, we track human approvals per
11:05PR, human comments, and how long a PR
11:07waits on a person. Revert and incident
11:10rates are the guardrail. If they rise,
11:12we haven't saved human time. We've only
11:14moved it. So, let's look at the life of
11:16a change. Typical change from an
11:17engineer's laptop or a cloud session in
11:19Slack goes through multiple layers of
11:21code review. So, after the change is
11:23initially made, the work is verified.
11:24This could be Claude running tests,
11:26building and linting, maybe has a second
11:28agent come in and review the diff in a
11:30fresh context. I would add where
11:32possible the work should be verified
11:35beyond just like a quick test check,
11:38build check, lint check. If the change
11:40is complex enough, it is almost always
11:42going to be worth it to spin something
11:45up and really test it, especially for
11:48like UI work, whether you're spinning it
11:49up in a simulated browser or you're
11:51spinning it up with computer use in an
11:53environment where it can open up the app
11:54and actually check things. One of the
11:56things I've been loving with T3 OS is
11:57that I made it easy for each thread to
11:59spin up its own like window in like the
12:02equivalent of in Mac OS how you have the
12:04different workspaces here. I made it so
12:06every thread can get one of those. It
12:07has made it very easy for a thread to
12:09spin up an Electron dev server for T3
12:11Code, record a video of the change,
12:13showing it working, and then send it to
12:15me or put it in the PR. I also added the
12:17ability for during dev to have fake
12:20model selections that have pre-baked
12:22events that they will send through the
12:24pipes so that an agent can run a full
12:27end toend test without ever having to
12:29actually burn tokens beyond the tokens
12:31it's burning doing the like
12:32visualization of the check. So it can
12:35test our code without burning tokens.
12:37And all of these types of things have
12:38actually helped a lot with our confident
12:40shipping and the frequency that I can
12:42land scary changes because my agents now
12:45can actually test and validate the code
12:47the same way a user would. So by the
12:49time my agent is telling me the code is
12:51ready to be looked at, it's already
12:52verified it in most of the ways I would
12:54have verified it. But that's my process,
12:56not Claude team's process. Let's hear
12:57more about theirs. On every poll
12:59request, they go through the required CI
13:01which includes lint types, tests across
13:03platforms, building a packaging, red
13:05blocks, the merge expected. Then they
13:07have the cloud code review step where
13:08cloud code itself comes in and runs
13:11through this like a human would play
13:13with it. It spins up multiple agents in
13:15parallel hunting for bugs, verifying
13:17changes, using verifiers to vote on each
13:19finding. Survivors are ranked by
13:21severity. Then there's security review,
13:23which is a separate pass where they find
13:25things that could be potentially
13:27insecure. And then there's a risk based
13:29approval. They have a classifier for how
13:31risky is this change and how much human
13:34intervention should be used here. No
13:36person, one person or two people. This
13:38is interesting. I've been thinking about
13:39doing something similar in T3 code where
13:42in like the template or the like skill
13:44we have for filing PRs, I'm considering
13:46having it put a blurb at the end of
13:48every PR saying on a scale of 1 to 10,
13:51how impactful is this change? How many
13:53users are impacted by this change? And
13:55how risky is this change? to make it
13:57easier to digest the PRs when they go
13:59up. Then we have a merge queue. Uh we
14:02don't for T3 code yet. We probably
14:04should. We've broken main enough times.
14:05Not broken in the sense that like the
14:07build is broken, but broken in the sense
14:08that like one or two random CI things
14:10fail. We probably need a merge queue at
14:11this point, but we don't have one, and I
14:13would like to avoid it as long as I can.
14:14And then after the merge, the changes
14:16will be behind a flag that's used
14:18internally initially. Then they'll do a
14:20gradual release. Then they watch this
14:22release to make sure everything goes
14:23well. fix forward if it's broken or roll
14:26back if it's too broken to just fix.
14:27That all makes sense to me. I like this
14:29framing of layers where you want the
14:31good change to make it through and you
14:33want all of these blockers to block
14:35things that aren't going down the right
14:37path. So maybe a change that is bad
14:40leaks through the verification system
14:41but then fails the CI layer. Maybe it
14:44gets through CI and then fails the AI
14:46review. Maybe it gets through that and
14:47then fails the AI security review. These
14:49layers increase the likelihood that bad
14:51code is filtered out before it makes it
14:52to the bottom. But there's another issue
14:54here that is worth noting. Each of these
14:56layers isn't just trimming out bad. It's
14:59sometimes appending additional
15:02unnecessary stuff, too. I cannot tell
15:04you how many times I've seen a PR that
15:06should have been like 50 lines of code
15:07get bloated to be 800 to 2,000 because
15:10the agent in the center addressed every
15:13single piece of feedback from every
15:14single layer on the way. And the result
15:16was just bad. And the most important
15:18thing is at the bottom here. If
15:19something bad gets through to the end,
15:21you need to be able to roll it back
15:22fast. This is like the most important
15:24thing. Failures should affect as few
15:26people as possible and be as cheap as
15:27possible to undo. This has always been
15:29the case, but it is worse now than ever.
15:32Everybody's systems needs to be set up
15:34so roll backs are viable. So step one is
15:36the work is verified before anyone else
15:38looks. Most reviews are happening before
15:40the PRs even exist. We give Claude
15:42something it can run like a test build,
15:44screenshot to compare against the design
15:46and ask it to show the output instead of
15:48saying it's done. Then a sub agent
15:49reviews the diff with fresh context.
15:51Both habits come from their best
15:52practices guide. Before opening a PR,
15:55many of us have also run a few internal
15:57skills. One hunts for bugs, one
15:59simplifies the diff, one checks the
16:00behavior end to end. UI changes also get
16:03checked against our design guidelines.
16:05Yep, this is aligned with what I was
16:07talking about before. Boris who was the
16:08original creator of cloud code has given
16:10claude ways to verify its work and says
16:12that this is the most important thing
16:13for getting good results. This is I
16:15absolutely agree. I think a lot of the
16:17people who are struggling to get good
16:18results out of working with agents are
16:20struggling because they haven't given
16:22the agent any way to check if what it
16:24did worked or not. And to be fair, a lot
16:26of their co-workers, especially if they
16:28hired a new edge, they'd probably run
16:29into a lot of those same problems
16:30because if you make code that is hard to
16:32check, then you made a codebase that
16:34sucks to work in. Boris estimates that
16:36this makes the final result two to three
16:38times better. And he writes the
16:39following. Verification looks different
16:41for each domain. It might be as simple
16:42as running a bash command or running a
16:44test suite or testing the app in a
16:45browser or a phone simulator. It could
16:47also be significantly more complex. Like
16:49I've had to build some crazy VM for
16:50testing T3os. Tests are not easy always.
16:53Building these systems to verify things
16:55are not easy always. But you got to
16:56figure out a way to do it. There's
16:58always some way. And if you notice the
17:00automated reviewers are consistently
17:01flagging the same things, then it's time
17:03to fix at the source. You can fix this
17:05in a skill with a lint rule or with
17:06updates to instructional files as such
17:08as a claude MD change to tell claude
17:11what the project is and like what
17:12conventions to follow. A review MD which
17:14is written by the code owners that tells
17:16the reviewer what matters in their part
17:18of the code. We check separately whether
17:19the reviewer actually follows it. The
17:21good ones are short. They say what
17:23failure looks like in that repo. For
17:25example, in the docs it could say it
17:27lies. For config it says it misroutes.
17:30You can cap the nits list paths to skip
17:32yada yada yada. find good place to put
17:34this info. I will say this is not the
17:36right starting point. I'm gonna cite
17:38some of Lauren's stuff here because I
17:41think her guidance here is a little bit
17:42better. When you have to correct your
17:44agent, you should look for the place
17:46closest to the top to fix what caused
17:49the agent to do things wrong. The top
17:51spot is the code base. If it's making
17:53mistakes in the code, see if you can fix
17:54the code to make it less likely to make
17:56mistakes. Don't do this by putting
17:58comments everywhere. Do this by making
17:59the architecture better and easier to
18:01work in. Then static analysis is a great
18:03next step. Things like llinters,
18:04checkers, rules, type checks, all those
18:07types of things that can give agents or
18:09even humans feedback that the thing
18:11isn't implemented correctly. That gives
18:13the agent the feedback it needs to do it
18:15correctly if it couldn't figure out what
18:17correct is from the code itself. Then if
18:19both of these fail to catch the problems
18:22and you've tried and been unable to make
18:24it so these specific problems cannot be
18:26fixed at those two layers, then you
18:29bring in pros. That's when you start
18:31touching claude MDs, review MDs, agent
18:33MDs, you start building skills, you
18:35start doing these things. You do that
18:37after the rest failed. But you should
18:39always be trying your damnedest to make
18:41it so the code base and so the lints,
18:44checks, etc. guide humans and agents to
18:47success. The silliest way to frame this
18:48is imagine a really, really smart human
18:51that just joined your team but doesn't
18:52know where anything is. You should make
18:54it as easy as possible for that person
18:56who has worked in thousands of code
18:58bases in the past but has never worked
19:00in Europe specifically. Should make it
19:01as easy as possible for them to just
19:03blindly dive in and work and minimize
19:05the chance they make dumb mistakes. And
19:06just imagine this person doesn't care
19:08about TDM and will do tedious work all
19:10day every day without reason. If you
19:11were already setting up your code bases
19:13for success for your teammates, this
19:15transition should be very easy for you.
19:17The next section that Addi has here is
19:19AI review on every PR. Each poll request
19:22gets a claw review before a person
19:23looks. It reads the diff against the
19:25whole repo and the PR's stated intent
19:28and it asks, "Does this change solve the
19:30problem, fit the system, or does it
19:32break something else?" On a bug fix, it
19:34looks for the root cause. The public
19:36version code review for cloud code is
19:38now generally available on the team and
19:39enterprise plans and/code review runs a
19:41lighter version locally. When we
19:42launched the research preview, the share
19:44of internal PRs getting substantive
19:46review comments went from 16% to 54%.
19:50Most PRs over a thousand lines get
19:52findings, and most small ones come back
19:54clean. Admins set each repo to quick,
19:57standard, or deep. Deep is worth it when
19:59one missed bug costs more than the extra
20:01review time does. I'm not going to pitch
20:04the cloud review stuff cuz I think their
20:06review is way, way, way too expensive
20:08for most real world cases, especially
20:10when you're shipping as fast as they're
20:11recommending here. There's lots of other
20:13review tools that are cheaper or can be
20:15used subsidized locally. One of the
20:17things I love is my council skill where
20:19I have my code get reviewed by three
20:21different models from three different
20:22families. That gets subsidized. I'm
20:24using my subscriptions for it and gets
20:26as good if not better stuff than these
20:28code review processes do. And it runs on
20:31a real computer so it can spin up the
20:32code and check. So I don't think you
20:35should go use their product for it. It's
20:36not bad but it's expensive for what it
20:39is and you could just use better prompts
20:41and other agents. And I think the
20:43multimodel side here is an aspect
20:44they'll never talk about that I do think
20:46is really good. Another important piece,
20:47and I'm somehow surprised people haven't
20:50caught on to this part yet, but I know a
20:52lot of you haven't, and it's really
20:53important. We shouldn't just have agents
20:56and review bots leaving a bunch of PR
20:58comments that we go through by hand and
21:00choose which ones to act on and which
21:02ones to not act on. That's the job of
21:03your agent. Comment to my sponsors cuz I
21:06know y'all watch my videos, especially
21:07ones like this, and I know a handful of
21:09you guys are doing AI code review
21:10products. Get the goddamn prompt
21:12suggestions out of the code review. That
21:15is 2024 core. We're in 2026. I'm not
21:18copy pasting from the code review
21:21back to my agent. My agent's babysitting
21:23the PR. My agent is deciding if your
21:26feedback is good or not. All of the code
21:28reviewers do this. Even like Bugbot,
21:30which you would think from cursor would
21:32understand this. Nope. All them by
21:33default leave the one-click AI fix
21:36prompt suggestion that you copy paste
21:38over. No, that. And and not the
21:39autofix stuff either. Get rid of all
21:41that. I want my agents to decide if your
21:43feedback is good. And no human should
21:45have to look until that is done. I
21:47should not be seeing the comments that
21:49my agents leave on the PRs. So, please
21:52make sure that the findings are verified
21:53by other agents before you force humans
21:55to review them. Several agents can look
21:57for bugs at the same time and then a
21:58separate group votes on each finding and
22:00only posts them if they're enough
22:03agreements that this finding is real.
22:05Findings that are only about style,
22:06diagnostics, or cleanup get dropped. We
22:08AB tested these filters on our own PRs
22:10before we kept them. What's left shows
22:12up as inline comments sorted by
22:14severity. Each is tagged as important,
22:16nit, or pre-existing. So, a bug that PR
22:18just didn't introduce doesn't look like
22:20the author's fault. Every finding that
22:21gets through this filter does have to be
22:23answered. You either fix it or resolve
22:25the thread with a reason. Often, an
22:26agent picks the fix up overnight. So,
22:29the author comes back to a PR that's
22:31already been through another round. Is
22:32Anthropic moving slower than us? I'll
22:35have a PR go through like eight plus of
22:37these rounds in an hour or two and then
22:40bother me when it's ready to go. Next
22:42section is security reviews that can
22:44block. Security gets its own automated
22:46review too. A PR can't merge while any
22:48finding from the bug review or the
22:50security review is still open. You
22:51either fix it or you close the thread
22:53with a short reason which is how you
22:55clear the false positives. If a later
22:57commit fixes the issue, the findings
22:59clear on their own. You can set up the
23:01same thing with open source security
23:02review, GitHub actions, or the /security
23:04review command. Setup is covered in
23:06their guide. Cool. Awesome. Now, we have
23:08automated checks that block. This is the
23:10usual CI type checks, list, linting,
23:13testing, all those types of things. You
23:15guys hopefully know what those are now.
23:17And we have the human approval layers
23:19where low risk means no human is
23:21required. You can just merge this when
23:23it's green. Medium, which is an ordinary
23:25change. You should have one person look.
23:27high is this is severe so at least two
23:29people should look e I don't love the
23:32012 way of framing this I would rather
23:35we and our agents can decide on a more
23:38dynamic scale I am going to play with
23:40these ideas though so we will see going
23:42forward but the point of the tiering is
23:44that it doesn't pick an author it picks
23:47the risk level a lot of the bots I've
23:49seen are trying to choose which human to
23:51tag in and be like this person has to
23:53review this thing this person has to
23:54review this thing Ideally your team
23:56trust trust each other enough to know
23:57who can review code. And if one person
24:00comes in and reviews it, they're like, I
24:01should have this other person look
24:02instead, they can. But the bot shouldn't
24:04be the one tagging specific people. The
24:07bot should be describing the severity of
24:09risk and then the humans can do the
24:10rest. Friends from cursor are saying
24:12that risk hearing is a losing battle.
24:14They're still doing it, but they have
24:16not had as good of a time with it so
24:18far. They call it that there are some
24:19paths that always need a person.
24:20Anything owners have listed as sensitive
24:22in any CI workflow files. Yeah, checks
24:24out. I have a couple of those in T3
24:25code. Like anything touching scroll
24:27areas, that's mine. That's not merging
24:29until I look. Next section is that only
24:31trusted authors are getting the lighter
24:33paths. Only authors of the repo's
24:35trusted list get the light paths, mostly
24:37the team that owns it. Everyone else
24:38needs a second person to approve. An
24:40agent's approval never counts as a
24:42person's either. Automated review can
24:44clear a low-risk change, but it can't
24:46fill a slot that calls for a human. This
24:48is one of the things that's been pissing
24:49me off on GitHub a lot. I want my bots
24:52to be tagged differently than me. I've
24:54done this in the sense that I added
24:56skills where they put a big call out
24:57saying I am opus 55 via cloud code and
25:00T3 code leaving this comment on Theo's
25:01behalf. But I want to see when Theo's
25:03bot approves separate from when Theo
25:05approves. I want two different approval
25:07checks for that. They also say that
25:08whoever is prompting for these changes
25:10is the person on the hook. You can't
25:12blame the agent if you prompted it.
25:14That's your ownership. That's your
25:15problem. And approvals only cover
25:17specific commits. If new commits land
25:19that the system can't vouch for, like an
25:21agent push or merge someone resolved by
25:22hand, the approval has to be restarted.
25:24They have a merge cue with flags and
25:26gradual releases. They start with the
25:28latest channel first, then stable gets
25:30the changes after. Fast detection, fast
25:32roll back. Some mistakes will still get
25:34through, so the last layer is about
25:35catching them fast. They have a weekly
25:37on call rotation. Incident review asks
25:40which layers should have caught the
25:42issue. Noticing a bad merge still
25:43depends on the author and on call as
25:45well as the revert and incident
25:47dashboards. We don't yet have circuit
25:48breakers that tighten approval when
25:50those numbers move. Engineers here want
25:52them before more code moves to zero
25:54human merges. That makes sense. So,
25:56they're not doing much zero human. We're
25:57trying to do a bit of this now for easy
25:59enough bug fixes. That's how Julius is
26:01able to start landing like 400 PRs a
26:03day. One colleague said that slowing
26:05down lowers your error rate, but mostly
26:07just cuz you're doing less of
26:08everything. Teams decided that they'd
26:10rather get faster at noticing and fixing
26:12problems rather than slow down to
26:13prevent them. And I agree there. If your
26:16solution to not shipping breaking
26:18changes is to just go slower, then
26:20you're not actually solving the problem.
26:22Banger. So now a more important section.
26:24How do we actually improve the quality
26:26of the reviews and the reviewer? The
26:28code review process is constantly
26:29improving in a loop. Any incidents
26:31become eval. The team collects past PRs
26:33that cause incidents with their fixed
26:35forwards and reverts and had engineers
26:37who know the code confirm each bug and
26:39where it lived. The reviewer runs
26:40against that set daily. Not how I would
26:43do that, but I like where they're going
26:44here. So, there's an incident, an expert
26:47checks and confirms, they add it to the
26:48set, and then they have this daily eval
26:50run where they check to see if these
26:53things would have happened again. Cool.
26:55We measure noise by watching what people
26:56do with the comments. It's easy to catch
26:59more bugs if you don't mind flagging a
27:00lot of non-bugs, so we need to check on
27:03that. Absolutely agree. Noise in PRs as
27:06hell. They sort every comment that the
27:08reviewer leaves by how the person on the
27:10PR responded, if they fixed it, or if
27:12they said that was wrong, or they said
27:13the behavior was intentional, or if they
27:14decided it didn't matter. We also run
27:16the reviewer on PRs that they know are
27:18clean, and they count how often they
27:19report something that would have led
27:21someone to make a harmful change. Our
27:22priority, written down, is catching real
27:25bugs first and cutting reviewer rounds
27:27second. That makes sense. Noise costs
27:30more when the author is an agent. A
27:31person can wave off a knit in seconds.
27:33an agent will fix it, which means
27:34another commit, another round of review,
27:36more unnecessary code, and so much other
27:38slop. But so the reviews do have to
27:40converge. An agent author and an agent
27:42reviewer can keep each other busy
27:44indefinitely. I had what I thought was
27:45going to be a simple change. I'm working
27:47on the like Grockbot style experience in
27:49T3 Code where there's like one main
27:51thread that controls all the other
27:52threads. I had a working prototype that
27:54wasn't quite how I wanted it to work. So
27:56I threw it out into council review and
27:59said, "Here's what I want. keep
28:00iterating until the whole council
28:03passes. And then I forgot about it
28:04because it was folded in my working tab.
28:06I just noticed like yesterday it's been
28:08going for four days. The PR that it's
28:10working on has gotten to like 80,000
28:12lines of code and I had to hop in and be
28:14like, "No, you're just running in
28:15circles right now." And I'm
28:17realizing I need to add short circuits
28:18to some of my skills cuz it's just it's
28:20not making any progress at this
28:21point. So yeah, you need the reviews to
28:24converge. You need the things to to end
28:26eventually. One CLI PR went through more
28:29than 50 rounds of review and fixes
28:31overnight before the author stopped it.
28:32I didn't even read and say that when I
28:33was describing mine. That's hilarious.
28:35Yeah. Anyways, when we ran get blame on
28:39what the reviewers kept flagging, about
28:4160% of it was code that had been there
28:43since the reviews first pass. The
28:45reviewer was rarely finding new bugs. It
28:47was finding a few more that the old
28:49rounds just didn't catch. We made it
28:51worse ourselves. We' gotten used to
28:52running several rounds by default since
28:54each pass was catching something new.
28:56This can also be worthwhile on risky
28:57changes, but not for every PR. The team
28:59now tracks rounds to pass alongside
29:02recall and changed re-review. I did also
29:05manage to fix this in a silly way.
29:08Instead of just tracking all of the nits
29:10and going until the agent said, "No,
29:13this is fine." I moved over to a scoring
29:15system where the review agents leave a
29:18score 1 to 10, and I pick a bar for what
29:21score is high enough. Because a PR might
29:24have 18 comments but still be a 9 out of
29:2610 cuz all the comments are just silly
29:27small knits. But you might have a PR
29:29that only has one comment that's a 6 out
29:31of 10 and the one with one comment is
29:33higher priority to be fixed than the one
29:35with 20 comments in the state. So
29:37finding a way to to set a bar that isn't
29:39by how many comments are left but is
29:41instead by how much confidence can you
29:43build in the system. That is the path
29:45that works. So they started with a round
29:47cap. I also did this with soloop. I
29:48haven't done this with council yet and I
29:50need to. After three rounds, the
29:51reviewer will ask the author to step
29:53back and reconsider the approach.
29:55There's a diff only re-review after the
29:57first few passes. There's a layer to
29:59flag the class of bug so the author can
30:01fix every instance, not just one line.
30:03And then no repeats of findings that
30:05have already been posted. These are good
30:06changes. Review agents also are running
30:08nightly over existing code, helping
30:10catch old bugs before they show up
30:12somewhere else. After making the
30:13convergence changes, they saw a 13%
30:17increase in PRs that were passing after
30:19one review and fix round. They had a
30:21decrease for PRs that need five or more
30:23from 9% to 1.5. And the mean reviews to
30:26pass went from 2.08 to 1.32. These are
30:30good changes. So, where do humans stay
30:31in the loop with all this? The thing
30:33that we're all curious about, where do
30:34we go? Actually, one more good piece of
30:37feedback from Jake over at our friends
30:40at Graphite aka Cursor aka SpaceXi aka
30:43Grockbot. He thinks that scoring is not
30:44the solution. Agent should be able to be
30:46trusted enough to make a boolean
30:48decision with the right context just
30:50like human engineers can who you pay to
30:52do that. Quote, can I ignore this
30:54comment or not? This is a good solution.
30:57His goal here is to keep it simple. I
30:59largely agree. I do need to experiment
31:01with other options here more. I kind of
31:02just if I'm not happy with one, I play
31:04with it until I have an improved
31:06solution and then stick with that until
31:07it pisses me off. I need to be doing
31:09more of the like different tries and
31:11ways to do this. So, uh yeah, shout out
31:13to all my friends at all of the labs who
31:15come in and give feedback as I record
31:16these. It helps a ton with getting this
31:18all right. So, where do the humans
31:19actually come in here? They decide
31:20whether the thing is worth building or
31:22not. Direction, taste, and scope. One
31:24team that declared its review bottleneck
31:26solved still kept human reviews for
31:28changes from outside the team. So
31:30someone asks quote should we even do
31:32this? Next we have reviewing risky code
31:34like billing autoupdater and CI
31:35workflows. Reviewing architecture
31:37because this sets patterns that others
31:39copy. Reading the evidence test output
31:42commands that run and what they actually
31:44return. Screenshots review findings
31:46things like that. If you can't say what
31:48the checks would miss, ask for a real
31:50review rather than a quick approval. And
31:52of course owning the outcome. Whoever
31:54put up the PR is the person who owns it.
31:56It doesn't matter where the code itself
31:58came from. So, how do we share all of
32:00this context across teams? Because
32:02again, this is the thing I love code
32:03review for, not just catching bugs. I
32:05love code review for making sure
32:06everybody knows what's going on in the
32:08codebase. And to this day, like when I
32:10ask a teammate to review code, it's
32:11usually for this reason. I want them to
32:13understand it because if they don't
32:15understand it, it means I have to fix
32:16things. And if they do understand it, it
32:17means I'm not solely responsible going
32:19forward. Code review second job's always
32:21been to spread codebase understanding.
32:23Reading a teammate's diff is how you
32:24learn what changed and how the pieces
32:26fit. If machines do most of the reading,
32:28how does anyone learn the code? We don't
32:30have a formal process for this. Nobody
32:32on the team keeps a complete mental
32:34model of cloud code in their head. I
32:36know you, some of y'all are going to
32:37think this is insane. Like, someone has
32:38to know how it all works. I hope
32:40unemployment is treating you well if you
32:41feel that way because any sizable
32:43codebase does not have one person who
32:45knows everything. The code bases that
32:47are, and I've got in arguments with
32:49this, and I I've went back and forth on
32:51it. There are lots of really important
32:53code bases that are well understood by
32:55the one or two people that maintain them
32:56that are very small. Those aren't
32:58full-time jobs anymore. If your codebase
33:00is small enough that one person can
33:02understand it, then that codebase
33:03doesn't have enough throughput, not in
33:05terms of traffic. Like there can be a
33:06small codebase that does billions of
33:08requests a day, but it's not changing
33:10enough that it is that important in the
33:13sense that like you need to build a
33:15super religious process around it so
33:17that you can make changes every day.
33:18Like if you're only making one change a
33:20week, spending five of the six days
33:22you're working making sure the change is
33:24good. That's fine. That makes a lot of
33:26sense. It's just not the types of work
33:28that we're talking about here at all.
33:30So, as I'm saying, most full-time jobs
33:32are working in code bases that no human
33:35can understand all of and no agent can
33:36understand all of either. Nobody needs
33:38to understand it all, though. When
33:39someone wants to understand a part of
33:41the codebase, they can ask Claude to
33:43write an explainer for exactly that
33:44slice. It takes minutes and it reflects
33:46the code in its current state. I think
33:48people severely underrate how powerful
33:50this is. When I'm reviewing heavy
33:52changes, I'll often just paste the PR
33:54into T3 code with Claude and ask it to
33:57give me a breakdown of the changes. What
33:59is changing? Why is it changing? And
34:00where is the risk? Some people on the
34:02team go a step further and have Claude
34:04build an artifact to visualize the part
34:06of the code to teach them how it works
34:08in whatever way that person likes to
34:10learn. This is awesome. I highly
34:11recommend doing it. If the goal is to
34:12spend human attention well, then this
34:15hugely beats out reading a trivial diff
34:17line by line. Oneline copy changes teach
34:19you nothing. Explainers for the module
34:21you're about to change teach you a lot.
34:23Our culture of defaulting to working on
34:25open Slack channels is helping here too
34:27since you see what others are building
34:28while they build it. I do love that.
34:30It's one of the things I really hope we
34:31can figure out in T3 code is how to like
34:34work together as a team and see who's
34:36doing what and when. There's a lot of
34:37reasons it's hard. I want to figure that
34:39out though. And then of course the you
34:41land it, you own it. If you land a small
34:43or medium change or and something still
34:44goes wrong, it's the team norm that
34:46you're going to be the one to fix it. A
34:48few things keep this from going badly.
34:50Changes are small so they're easy to
34:51review, debug, and revert. Everything's
34:53reversible. Blast radius sets the bar.
34:55Expensive code gets more people in more
34:56checks. Regressions surface fast through
34:58telemetry and internal use. And it isn't
35:00blame. The ownership doesn't mean you're
35:02at fault. It means you have the best
35:04understanding. So you are ready to come
35:06in and fix the thing you have the most
35:08context on. Ownership is also shared.
35:10Much of the team works in open channels
35:12where anyone can see what the others are
35:13building, pick up threads, or point a
35:15claw at a gap. And feedback comes from
35:17people who don't own the code as well.
35:19That openness is a big part of how
35:20quality stays high without gating every
35:22single line. Here's another big piece. I
35:24love they put this in here. Changing
35:26direction. Agents make it cheap to start
35:28things. So, some work shouldn't land. I
35:30would guess that even if my PR is a T3
35:33code, about half of them merge, the
35:35other half get closed for various
35:37different reasons. Usually someone in
35:38the channel asks why, points at an
35:40earlier decision, or suggests a simpler
35:42route, and the author closes or reshapes
35:44the PR. Since the code is fast to write,
35:46closing the PR doesn't feel like a loss.
35:49The trick is to raise a different
35:51direction in the thread early, not as a
35:53blocking comment or a finished PR. So,
35:55how can we adopt the processes that
35:57Anthropic is sharing here? God, I hate
35:59how I don't know who wrote this. Like, I
36:01know Addy wrote a lot of it, but some of
36:02the sentence formatting here is awful.
36:04This is one of the worst sentences I've
36:05ever read. Most of the pieces to do this
36:08with your team exist today. There is not
36:11there's no pause or sectioning of this
36:13sentence at all. This needs a comma or
36:16two or to be split into multi multiple
36:18sentences so badly. Sorry, I I'm picky
36:21about pros. I I like writing. Crazy how
36:23large language models have gotten good
36:25at pretty much everything other than
36:26generating text. I hate LLM writing so
36:29much and I am I'm smelling things I
36:31don't like here. But we can still learn
36:33from how to adopt the process. I like
36:34that we start from before the PR because
36:36there is before, during, and after the
36:38PR, like the sections you should think
36:39about this stuff through. Before the PR,
36:41you should give your agent checks it can
36:43run and ways it can verify its work. You
36:45should run the code review and security
36:46review skills locally. I just have this
36:48as part of my like main skill for
36:50landing changes and the agent decides
36:52what to pull in and not pull in. You add
36:54a review MD and more repeat review
36:57comments can go into that. You should
36:59fix the code first. On every PR, you
37:02should require CI, add testing, risk,
37:04how to revert to PR templates. I like
37:06this part. Like your PR template and the
37:08skills or whatever you use to open PRs
37:10should include how to test, what the
37:11risk is, and how to undo this change.
37:14Add an AI reviewer. You can use one of
37:16theirs that they suggest or any of the
37:17awesome sponsors of the channel on the
37:19repo. Tune the review MD for a few weeks
37:22and then widen what this checks going
37:24forward. Totally fine. add automated
37:26security reviews and require every
37:28review thread resolve before merge. I
37:30love that they just drop this in here as
37:32though we can do this and our claude
37:34doesn't block us from asking questions
37:37about security stuff. There are
37:39solutions to this problem, but they're
37:40not going to be very frequent on the
37:42anthropic side. Sadly, match review
37:44efforts to risk. So, a quick pass on
37:45routine checks, a deep one where a
37:47missed bug is expensive, quick,
37:49standard, or deep encoder review. And
37:51then capping the review round so that
37:52you don't go off forever. like I've
37:54burned so many tokens with accidental
37:56forever reviews. Then you have the
37:57approval process. You write down risk
37:59tiers, yada yada. Set the risk based on
38:02the changes, not than the author. Keep
38:04lighter approval paths for authors you
38:05trust. Use code owners and branch
38:07protection. So O, billing, CI, and
38:09release flows, other important things.
38:10Get specific people. Never let a bot's
38:12approval replace the humans. Yes. Then
38:15we have the merge and after section. Use
38:17a merge Q, feature flags, internal use
38:19first, etc. Staged releases and onestep
38:22roll backs are huge as well. Turn
38:24incidents and reverts into eval sets.
38:26Rescore the reviewer when it prompts and
38:29use the prompts, tools, and model
38:30changes to get a new score to figure out
38:32if the model's actually making progress
38:34here. Anything you cannot measure, you
38:36cannot fix at this point because AI is
38:38so good at fixing things when you give
38:40it the tools to climb against. And if
38:42you can make an eval of the types of
38:44changes that have been getting into your
38:45code that shouldn't have and turn that
38:47into a test you can run, it gets very
38:49easy to fix these things. I talk about
38:50this a lot in my hill climb video where
38:52I talk about running claude forever, I
38:53go in depth on how to set these types of
38:55environments up. They also suggest that
38:56measuring human time is useful with
38:58reverts and incident rates being
38:59measured as well. So, where is all of
39:01this going now? When they announced the
39:02code review product for Claude Code,
39:04they said that approving a PR is still a
39:06human call. The product posts findings,
39:08not approvals. Our own repos are now
39:10going further, letting owners decide
39:12which low-risk changes merge on
39:14automated checks alone, and that line
39:16keeps moving. When a review misses
39:17something, we ask what the model didn't
39:19have in front of it, and we put the fix
39:21in a cloud MD, a skill, or a new check.
39:23Nope, this is not right. This is the I
39:25do not like this guidance. You should
39:27try to fix the code first. And if you
39:29can't fix the code in a way that
39:30prevents the problem, then you add a
39:32check. If you can't fix it there, then
39:34you might maybe add a skill or a cloud
39:37MD. We somewhat recently overhauled all
39:39of our repo guidance. And I bet if we go
39:43to the agent, oh, we did make a change
39:46to agent MD eight hours ago. Let's see
39:47how frequently this file changes. More
39:49than it used to, but still not that
39:52much.
39:54One month of changes still fits on one
39:56page, which is much less than other
40:00places in this codebase. The chat view
40:02component, for example, changes like 30
40:05to 50 times a day. Our agent MD has
40:08changed onceish a dayish and even less
40:12earlier on. Like it changed once every
40:14fiveish days before. So like we're not
40:17constantly adjusting all of this. Did
40:19Julius move us off of TS Rust? Oh,
40:23Julius, you should have just told me to
40:24keep going. I would have just spun
40:26back up my agent. Apparently we turned
40:28off of TS Rust because Julius wanted new
40:30effect features. But yeah, like actual
40:32guidance change there though. We didn't
40:34adjust it because we saw some small
40:36thing that annoyed us. So like even
40:38there, we didn't change this because we
40:40saw our agents doing the wrong thing. We
40:42changed it because the repo itself
40:44changed and one of the pieces of
40:45guidance was no longer true. That is
40:47what leads us to changes, not problems
40:50in the codebase that we're trying to
40:51like give you like language to avoid.
40:55Imagine that you're like blind and
40:57you're navigating a maze in a dark room
41:00and you keep running into walls and
41:01tripping. We could give you better
41:03instructions before you walk into the
41:04maze. Or we could just cover up the
41:07holes and make the guidance in the maze
41:10better so it doesn't matter how well you
41:11can see when you walk in. Like I don't
41:13want to just keep giving the model more
41:16instructions before it goes in the door.
41:18I want to make it so the room it's
41:19navigating in is as easy for it to
41:21navigate as possible. And I don't think
41:23more and more of these review MD files
41:25are the solution at all. They actually
41:27have a call out at the end here asking
41:29to hear more from us about which checks
41:31we're automating, which ones still need
41:32people, and what we think they have
41:34wrong. I'm even more excited about this
41:36video going out now cuz I know they'll
41:38listen. They might actually try some of
41:39these things, especially now that it's
41:40like Addie and Emma who are two old
41:42friends. I also like Emma's absolutely
41:44unbelievable. I'm so happy that she's at
41:45Anthropic now. Not as familiar with
41:46Michael and Amy, but I've heard good
41:48things. This is an awesome article. It's
41:50really nice to have Anthropic being this
41:52much more transparent with their process
41:54now. blogs like this just did not exist
41:56from them a year ago. They were not
41:58sharing this much about how they worked
41:59internally. And it's such a nice change
42:01to get this type of info. Hopefully, you
42:03enjoyed this as much as I did. I love
42:05peering behind the curtains on how
42:06people work with these tools. And maybe
42:08you got to learn some things about how
42:09you can use them, too. Let me know how
42:10you guys are doing reviews in the
42:11comments. And until next time, peace
42:14nerds.

About this transcript

This page contains the full transcript of How Claude Fixes It's Own Code by Theo - t3․gg, generated from the public captions YouTube serves with the video. The transcript has 9,307 words across 1,296 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.