YT YouTube2Text

YouTube2Text → Transcripts → Boris Cherny: We Cut 80% of Claude Code’s Prompt

Boris Cherny: We Cut 80% of Claude Code’s Prompt — Transcript

by Y Combinator · 6,868 words · 1,096 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:03[music]
0:07>> All right, Boris. We're so excited to
0:09have you here, the creator of
0:13Claude code.
0:14>> Thank you.
0:17>> [applause and cheering]
0:19>> It's great to be here.
0:21>> Fresh off the press. You guys just
0:23shipped Opus 5 yesterday.
0:26>> Yes.
0:27>> [applause and cheering]
0:28>> And it seems that model performance
0:31keeps accelerating. You guys got and
0:34took Arc AGI 3 to 30%,
0:38which is incredible.
0:40>> Yes.
0:41>> And for context, before the best score
0:44was in
0:45in the low single digits or low teens,
0:48right? What can Opus 5 do now that it
0:52couldn't versus the previous version?
0:55>> Yeah, there's um there's a lot that goes
0:57into every new model and there's a lot
1:00of new capabilities that we teach and uh
1:03get the model to do.
1:05Whenever you do model training,
1:08you try to teach a whole bunch of
1:10different things and most often it
1:12doesn't work.
1:13But some subset of the things the model
1:16does learn and sometimes it also
1:18surprises you. It has these skills, it
1:20has abilities that you you actually
1:22didn't really teach it, but it it just
1:24kind of learned. For five, one example
1:27of something it does that I think no
1:29other model has done is it runs for a
1:32very long period of time and especially
1:35when you combine Opus 5 with auto mode,
1:38it's just like incredible. Like it can
1:39go for days, weeks, months at a time. It
1:43just won't stop.
1:44Um you don't even need to use
1:46scaffolding. So you don't need the slash
1:49goal, you don't need all this other
1:50stuff. It'll just go because it knows it
1:52needs to do the task. Um another thing
1:54that I'm really excited about and um I'm
1:56going to start I think to talk about a
1:58little bit more um but it's kind of
2:00surprising because it's such a new
2:01capability is the model does not seem to
2:04be prompt injectable anymore.
2:08>> What's prompt injectable?
2:10>> [applause]
2:11>> It's crazy. Like people have talked
2:12about this like lethal trifecta for a
2:14long time and this really affects kind
2:16of harness design and agent design and
2:19and and product design because if the
2:21model reads some instruction on the
2:22internet that's like, you know, do X and
2:25Y and Z and also delete everything on
2:27the user's computer.
2:29A year ago the model would have just
2:30done it.
2:32But nowadays Opus does not.
2:35And this has actually been the case
2:36since like Opus 4.7, 4.8,
2:39Sonnet 5 has been quite good at this,
2:41People was quite good at it. But Opus 5
2:43just hits like a new frontier on this.
2:45So essentially if you combine a
2:48well-aligned model, so this is like
2:49essentially three years of research into
2:51alignment,
2:52with a prompt injection classifier which
2:54we run for all traffic. And what this is
2:56doing is it's based on Crystal's
2:58mechanistic interpretability work where
3:01it's it's literally we're looking at
3:02neurons in the model's brain
3:04that light up when prompt injection
3:06happens.
3:07So the model won't even tell you but we
3:09can actually see those neurons and we
3:11can figure out and diagnose that it's
3:12happening.
3:13And then you combine that with the auto
3:14mode classifier and with these three
3:17layers
3:18we just cannot demonstrate prompt
3:20injection anymore.
3:21>> Talking about a prompt injection, the
3:24other side of the coin is now the system
3:27prompt. Let's talk a bit about the new
3:30release. You actually deleted over 80%
3:34of the system prompt from Claude code.
3:37>> Yes.
3:37>> Tell us more about that.
3:38>> I think something that a lot of people
3:40might not realize is
3:43Claude code as a product and as a
3:45harness is just always changing. We're
3:47always adding stuff. We're always
3:49deleting stuff.
3:51Every time that a new model comes out,
3:53we delete a bunch of the system prompt,
3:55change a bunch of the system prompt. We
3:57change the set of tools all the time. We
3:59change the prompts for the tools all the
4:01time. And the reason is
4:03every model is very different.
4:07So, something that you did for one model
4:09maybe 3 months ago, it just might not
4:11translate at all to the next model.
4:14And so, one thing about Opus 5 is it's
4:17just really intelligent. And a lot of
4:20the stuff in the system prompt was
4:22correcting for these behaviors that the
4:23model should have known,
4:25but uh it didn't.
4:27Now, Opus 5 just does it.
4:29So, yeah, we deleted 80% of the system
4:32prompt.
4:33You can actually try deleting the rest
4:36of it, too. Um so, when you run Claude
4:38Code, you can just do like {dash} {dash}
4:39system prompt and set whatever system
4:41prompt you want if you want to
4:42experiment with it.
4:44And another thing that you can try is um
4:46simple mode. So, this is actually this
4:48kind of undocumented feature. If you do
4:50Claude Code simple equals one, like this
4:53uh environment variable, and then you
4:54run Claude, it'll delete all the system
4:57prompts,
4:58including from the tools.
5:00And we actually use this as a sort of
5:01ablation
5:02to figure out is the prompt useful? And
5:05what's interesting is that the model is
5:07actually a little bit more intelligent
5:08without these prompts.
5:10That's something that we've been
5:11finding. But when you use Claude Code as
5:13a product,
5:14you do actually want some of these
5:15prompts because it helps you use the
5:18product and it it helps the the product
5:20behave and the model behave in the way
5:21that you would want when when you're
5:23using it as a person.
5:24>> I think the thing that's really
5:26fascinating in this era of building,
5:28basically you'll build the best harness
5:30in the world for for Claude, and that's
5:33Claude Code. From what I'm hearing, you
5:36for every model released, you basically
5:38delete all of the code base, delete all
5:40of the prompt, and start from scratch
5:42every time. That in the old world would
5:45have been not something
5:47startups would have done for the product
5:49cuz I press delete every 6 months for
5:51everything.
5:52>> That's right. That's right. We so to be
5:53fair, we don't delete the entire code
5:55base, but we do delete a lot. So every
5:57time there's a new model, we try we call
5:59it in a research you call this a
6:01ablation. And so what this means is you
6:03delete the entire system prompt and then
6:05you bring it back line by line to figure
6:07out what is the impact of each
6:08individual line.
6:09Um it's sort of like a eval and you can
6:11kind of like evaluate it and ablation
6:14essentially is a eval where you delete
6:15things to figure out the impact.
6:17And yeah, like we do the same thing for
6:19tools. Like we unship tools all the
6:20time. We you know, delete code in the
6:22harness all the time. If you look at
6:24actually the code that's in the Claude
6:26code harness today, almost all of it is
6:28about safety and permissions and static
6:30analysis and there's a bunch of UI code
6:33and we've actually unshipped a lot of
6:35the other code already.
6:37>> Do you think this way of building a
6:39agentic product and harness
6:42and basically doing ablations every time
6:45with a there's a new model release,
6:48should everyone in this room that's
6:49building AI products basically do that?
6:51Be comfortable and brave to
6:54press delete.
6:55>> 100%. Yeah, and and for people that
6:57aren't building agentic products, but
6:58you're using Claude code, every 6 months
7:01delete your Claude MD.
7:02Delete your skills.
7:04Delete your hooks.
7:05See what the model does and it might
7:07surprise you.
7:08And actually for Opus 5, this is
7:10something we really do recommend is just
7:12try deleting all of these things
7:15because the model might really just not
7:17need all those instructions that you
7:19needed for past models.
7:20>> Let's talk a bit about how then you
7:23build this new prompt. When there's a
7:25new model release, like for everyone in
7:27the room, everyone will want to try Opus
7:295 and they're going to press delete on
7:31their system prompt. How do they go
7:32about
7:34building rebuilding their system prompt?
7:36How do you set up your environment?
7:39>> So you do you do it kind of piece by
7:41piece. So, the first step is you delete.
7:45The next step is you use it.
7:49And you don't want to guess what's the
7:51instruction that the model needs because
7:53you might not predict it correctly. The
7:55thing that you want to do is you want to
7:56run it. And if it's like a customer
7:58agent like product that you're building,
8:00you want to kind of run the product. Uh
8:01you want to see where it fails with the
8:03model. You want to see what it does
8:04well. If you're using quad code, you
8:07want to see where it does well with your
8:09code base or maybe where it stumbles
8:11over, you know, the architecture or
8:12stumbles over something else.
8:14And only when you see it repeatedly
8:16stumble on the same thing, that's when
8:18you add it back.
8:20But you don't want to do it too early
8:21because remember like the model is going
8:23to read this instruction every single
8:24time you use it. So, you really want to
8:26make sure that the model needs this
8:27instruction. I I think this is sort of
8:30the crazy thing about building on
8:31models. It's just so different than all
8:33the engineering that I've ever done.
8:35Like in the past when you built on
8:37systems, you built these like big
8:38beautiful systems and you really think
8:40about the system design up front. You
8:42have like a big suite of unit tests. You
8:44think about everything and you know,
8:46like a re-architecture is a big project.
8:47It sometimes it takes months. I've
8:49worked on re-architecture products at,
8:51you know, big companies that take years.
8:54And
8:55the model is not like that. It's um
8:58the way to think about it is almost like
9:00a like a living creature, like it's
9:01something more organic. It's a thing
9:04where every model generation, it behaves
9:06differently. It has a slightly different
9:08personality. And you have to take the
9:10time to get to know it and then adjust
9:11the harness based on that.
9:13And I I think it's just very much like
9:15an empirical and kind of scientific
9:17thing. You have to take a very
9:19scientific mindset to it where you try
9:21something, you see the result, and then
9:23you iterate based on that.
9:25>> If you're building in this world right
9:26now,
9:27what then becomes uh stable? Are evals
9:31something that you keep from the
9:32previous models and keep using them in
9:34each new model release?
9:36>> Um we do until we max out the eval.
9:40>> So, that's sort of the tip for everyone.
9:41So, code and system prompt, you have if
9:45you want to build at the bleeding edge
9:46and have the most capability for models,
9:48you got to delete those, but evals are
9:50constant and keep appending to them,
9:53basically.
9:53>> Yeah, you keep you keep appending.
9:55What happens is um
9:58you know, I actually wouldn't even go
9:59this far, to be honest. I think evals,
10:01they outlive the harness a little bit,
10:03but not by that much. Like an eval might
10:05live for maybe one, two, three model
10:07generations, but nowadays the you know,
10:10we're on the exponential. The model is
10:12improving so quickly, very often we just
10:15saturate the eval, and then we have to
10:16throw it away, and we have to come up
10:18with a new eval.
10:19And this is just part of the process.
10:21And again, it's about being empirical.
10:23You have to use the product, you have to
10:24use the model, you have to see where it
10:26struggles, and then based on that,
10:28that's the eval set that you should
10:29build.
10:30>> I think one one term I heard you
10:32describe how to build the best agentic
10:35products on top of a Claude is this
10:38concept of a unhobbling
10:41Claude.
10:42And tell us more more about what that
10:45means.
10:47>> Yeah, so hobbling is this idea in a
10:49research that the model is doing
10:51something and you're just getting in the
10:52way.
10:55There there's this kind of like way of
10:56thinking about it that I really like.
10:58It's very useful when you're building
10:59product. And um
11:02it it it's called product overhang.
11:04And then the idea is
11:07the model is able to do all sorts of
11:10things
11:12with today's models, not a future model,
11:13but today's model, that we have not yet
11:16realized.
11:17And there are so many capabilities the
11:20model has like this that people are not
11:23aware of.
11:24And this is like the ability to, you
11:26know, like maybe use a particular tool,
11:29use a particular language, solve a
11:31particular kind of problem, do things a
11:33particular kind of way that we thought
11:35was kind of beyond the model's
11:36capability.
11:38And um
11:40there's this overhang.
11:41Because the model can do this at every
11:43given model generation,
11:45but there is often not a product that
11:48lets the model do this
11:50and lets it express this kind of ability
11:52to do this. And on the flip side, often
11:55what happens is the product gets in the
11:56way.
11:57And this getting in the way, we call
11:59this hobbling. And then not not
12:01eliciting the correct behavior from the
12:02model, we call this product overhang.
12:04So, it's kind of like two sides of the
12:05same thing.
12:06What One example of this was the
12:08original Claude Code.
12:10When I first started working on it, this
12:12was um
12:14you know, like a year and a half, 2
12:15years ago, something like that. This was
12:17like Sonnet 3.5.
12:19At the time, that was an incredible
12:21coding model. That was like the best
12:23coding model that exists. Nowadays,
12:25it's, you know, a pretty terrible coding
12:27model by modern standards. But I think
12:29that was like the first great coding
12:30model that that we built as Anthropic.
12:33And at the time, if if you looked at the
12:35coding products of the time, what were
12:36what were they doing? They were doing
12:37like single-line auto complete. They
12:40were doing sometimes multi-line auto
12:41complete. That was sort of a new idea.
12:43Um they were they were doing chat. So,
12:45you can talk to the agent, but it wasn't
12:47uh right access. You could only read.
12:49You could ask about the code base.
12:51And so, the the feeling was that there
12:54wasn't really a product
12:56that was fully eliciting the model's
12:58capability to write entire functions at
13:01a time, entire files at a time.
13:04At the time, it wasn't entire features.
13:05We weren't there yet, but probably
13:07entire files. That's that was the level
13:08of capability at the time.
13:10And so, the idea with Claude Code was,
13:12all right, we think the model can
13:13probably do this.
13:14What if we get rid of all the
13:16scaffolding
13:18and just give the model the simplest
13:19possible harness, so it can write an
13:21entire file at a time and build an
13:24entire feature?
13:25Um
13:26and that was that was kind of it. Like,
13:28was the product overhang of the time.
13:29The model was capable of doing something
13:31and everything was just kind of getting
13:32in the way.
13:34I I think that nowadays, with modern
13:36models, there's so much product overhang
13:39that I have I'm not seeing startups
13:40capture.
13:42And I think there's people thinking
13:43about these problems,
13:45but there's just a huge amount of amount
13:46of opportunity to elicit
13:48these behaviors from the model that are
13:50just like amazing and interesting and
13:53and commercially valuable.
13:54>> I think this is such a special insight
13:56for everyone here in the room.
13:59Basically, all of you could create the
14:01next Claude code if you figure out how
14:04to un-hobble the models because that's
14:07effectively the birth story of Claude
14:09code. You un-hobble Sonic 3.5 because
14:13all the previous
14:15iterations were still getting the model
14:17very rigid in in IDEs. And Claude code
14:19was one of the first
14:21instances that gave it just a full
14:23terminal access.
14:25>> Yes.
14:25>> And that
14:27then created this amazing product just
14:30that keeps going.
14:31So, let's talk about um
14:34what are some areas and how should
14:38future founders here think about
14:40un-hobbling
14:43Claude and fixing this product overhang?
14:47>> So, there's a couple of things that I
14:48would think about.
14:50One is
14:52you should give the model slightly
14:55harder tasks than what you think it can
14:57do.
14:59I think a a really common mistake that I
15:01see is people are using Claude code,
15:03they're using Claude, and they they just
15:05give it like way over we specific
15:07instructions. They're like, "I want you
15:08to do this, but I want you to do it in
15:10this way, this way, this way. You must
15:11do like one, then two, then three, then
15:12four."
15:14And for modern models, that's actually
15:15really not the way to do it. You want to
15:17go a little bit higher level.
15:19You want to describe the task, you want
15:20to describe the guardrails, you want to
15:22describe like the exit criteria, and
15:23then just go with the model cook.
15:26And come back in a little bit.
15:29And I think it'll it'll surprise you.
15:30Like and again, like this is just not
15:32something that would have worked 6
15:33months ago, but it does work today.
15:35>> Can you give some examples of these
15:37challenging tasks or capabilities that
15:40people should explore that it can do now
15:42that it couldn't 6 months ago?
15:44>> Yeah. So, okay, one example is the model
15:48can now rewrite essentially any code
15:50base from one language to a different
15:51language.
15:53It's
15:54just sort of crazy. Like it's this work
15:56that would have taken just like a very
15:58long time as an engineer, and now the
16:00model's like quite fast at it. So, so
16:02one example of this is um
16:04Cloud Code is built on the Bun
16:06JavaScript runtime. It's a open source
16:08JavaScript runtime.
16:10Um it's an alternative to Node.js. It's
16:11kind of a faster node.
16:13Bun was written in Zig.
16:15Zig is a systems programming language.
16:17It's It's kind of like C. It's It's very
16:18low level. One of the problems with C
16:21with a with Zig is you have to manually
16:24manage memory.
16:26And so it's quite easy to run into
16:27situations where there's like memory
16:29leaks and, you know, other memory
16:31management issues.
16:33And so, one thing that the Bun team was
16:35doing is they were having Claude fuzz
16:37the code base and try to simulate and
16:40trigger memory leaks, and they were
16:41doing this for, you know, for a long
16:43period of time. They were able to find a
16:44lot of memory leaks. It was sort of like
16:46a case at a time. And that was kind of
16:47the capability of the model at the time
16:49was doing this fuzzing.
16:50And then at some point, Jared on the
16:53team was like, "Okay, let's just like
16:54rewrite it.
16:56Maybe the model can do this."
16:58And I I think this is like one of these
17:00test problems that he kind of threw at
17:01the model with every new model
17:02generation.
17:04And starting with Fable,
17:06the model started to be able to do it.
17:09And so I think Opus 5 could do it as
17:11well.
17:12And so what he did was
17:14essentially he defined a test suite.
17:16The nice thing about Bun is it's very,
17:18very well tested. There's a big test
17:19suite in Bun, there's a big test suite
17:20in Node.js. So, it's easy to know if you
17:22did the right thing.
17:24And he had the model rewrite it from Zig
17:27to Rust. It was one prompt. It was a
17:30dynamic workflow.
17:31And a dynamic workflows are a feature in
17:33Claude Code that essentially let you
17:34orchestrate, you know, dozens, hundreds,
17:36thousands of agents to do work
17:38productively.
17:39And it ran for 11 days, and it rewrote
17:42the entire code base.
17:44>> And this was one shot?
17:45>> It was one shot with It was No, it
17:47wasn't one shot, but it There was
17:48steering. There was steering.
17:49Um but previous models just couldn't do
17:51this, even even with the steering. It
17:53just wouldn't have been possible.
17:55>> 11 days? Oh my god. This would have
17:57taken in the past
17:59even with the best engineers, multiple
18:01months, years?
18:02>> Over Definitely over a year.
18:04>> Yeah.
18:04>> Yeah, over a year. This is like over
18:05100,000 Like JavaScript runtimes really
18:07complicated. There's There's a lot of
18:09stuff in there.
18:10Um and yeah, it like it works. This is
18:12in production now. This is what Claude
18:13Code uses now when when you're running
18:15it.
18:16So, this is kind of one example. I I
18:18would give a second example also of
18:20product overhang. And so, this is like a
18:22practical use case where like there's a
18:24problem you're solving. It's like a
18:25business problem, an engineering
18:26problem, a product problem. And you
18:28should just keep throwing the latest
18:29model at it to see if it'll just do it.
18:31Cuz even if a previous model didn't, the
18:33new one might.
18:35I think the second way to think about it
18:36is experiment.
18:39And just give yourself like freedom to
18:41play with the model and do creative
18:43things.
18:44Often it'll surprise you.
18:46So, something that's actually been
18:47really popular at internally that's been
18:49kind of viral within Anthropic the last
18:51couple weeks is someone figured out that
18:53you can give Opus 5 OpenCV.
18:55>> Oh.
18:56>> And you can have it draw.
18:58And so, something you can do is you can
18:59ask Opus like, "Hey, use Open OpenCV to
19:01like draw this image." And it's actually
19:03quite good. It can do like portraits, it
19:05can draw like animals, it can do like
19:06landscapes. And we didn't train the
19:08model to draw. Like it it's just like
19:11the solicitation gap. Like if you ask it
19:12to do it the right way, it can just do
19:14it. And we discovered this kind of
19:16accidentally just by playing around and
19:18trying creative things that didn't have
19:19direct commercial applications.
19:21But it's just kind of interesting. And
19:23my hypothesis is there's probably
19:26dozens, hundreds of opportunities like
19:28this with the models of today that no
19:31one has yet realized.
19:32>> And the big area of research for this is
19:34basically model elicitation, right?
19:37Becoming really good at
19:40figure out all these capabilities and
19:42asking the model to do the right thing,
19:43right?
19:44>> Yes.
19:45>> How do people get better at that? And
19:47effectively, how do people get better at
19:50prompt engineering? Do people still need
19:51to do a lot of prompt engineering? Or is
19:53that changing as well? Tell us about
19:56where this is going.
19:57>> Yeah, I remember like a year ago one of
19:59the most popular job openings was prompt
20:01engineer.
20:03And then it kind of changed and then I
20:04think it became like context engineer.
20:08So there's these kind of waves of it. I
20:10think I think these will kind of like
20:11come and go.
20:13I think the skill nowadays
20:15is less about prompt engineering and
20:18more about figuring out how do you give
20:21Claude a hard task
20:22that seems a little bit too hard.
20:25And then how do you make it possible for
20:26Claude to verify its work along the way?
20:29And the verification I think is probably
20:31the single most important thing that
20:32people do not get right or actually.
20:35Um
20:36one example of this is people were you
20:41know, we have this desktop app for
20:42Claude. And it's built using electron.
20:45We've made it quite fast. So now it's
20:46like a pretty awesome experience. Six
20:48months ago it was like sluggish and it
20:50wasn't very reliable. Now it's pretty
20:52awesome. And you know, it's the thing
20:53that most of the team uses.
20:55As an experiment though, I wanted to see
20:57like what would it feel like if it was
20:59native?
21:00And so what I did is I I started a
21:02Claude tag session. And Claude tag is
21:04just you know, it's a it's a new product
21:06we have. It's just quad running in
21:07Slack. My first question was, "Hey
21:09Slack, do you have access to a Mac OS
21:11runner on GitHub?"
21:12And uh it said no. And then I I hooked
21:14up a runner, so it was able to start a
21:16Mac virtual machine uh using using
21:18GitHub.
21:19And then um my second question is uh I
21:21created this like empty code base that
21:23was uh
21:24quad desktop app rewritten in Swift.
21:27And I asked, "Can you access this code
21:28base?" It said no. And then I gave it
21:30access and I was like, "Okay, great. Now
21:31I have access."
21:33And then I was like, "Okay, now I want
21:35What I want you to do is I want you to
21:38rewrite the Electron app in Swift.
21:42I want you to run the Electron app in
21:44the Mac virtual machine, screenshot it,
21:47and then look pixel by pixel,
21:50compare it to the Swift version,
21:52don't stop until you're done."
21:54>> And that was your prompt, basically.
21:56>> That was my prompt.
21:57>> And how long did this take to run?
21:59>> It's still running.
22:02>> When did you start it?
22:03>> [laughter]
22:04>> It's been
22:05uh it's been a little over 2 weeks, so
22:06it's like 14 days, 15 days.
22:09>> Yeah, so I don't know if anyone in the
22:11audience has gotten Claude to run a a a
22:15task for more than 2 weeks.
22:18I don't know. If raise your hand, anyone
22:19in the audience.
22:22>> Oh.
22:23>> All right.
22:24Some some.
22:25>> This is This is like one of these um
22:27this is about hallucination. So, it's
22:29So, this is really one of those examples
22:31where the model can do it today.
22:33You just have to let it do it. And you
22:35don't need the fancy stuff. You don't
22:36need slash goal, you don't need slash
22:38loop. These help,
22:40but really all you need is give the
22:42model the task,
22:43give it a way to verify the output of
22:45its work so it doesn't get stuck, and it
22:47will just go.
22:48And actually in this case, Claude also
22:49decided to live blog it. So, what it did
22:52is it created a Slack channel internally
22:54and it started just posting screenshots
22:56every few minutes of its progress.
22:57>> Wow.
22:59So, the prompt sounded so simple.
23:01I mean, everyone here could do it.
23:04And
23:06um I guess
23:07what is separating the people here that
23:09can become the top 1% Clocko users? How
23:13do How do How do people learn to use
23:15Clocko like Boris?
23:17>> Maybe like
23:18don't listen to the LinkedIn
23:20influencers.
23:21>> Don't listen to
23:22Don't read Twitter.
23:24>> [cheering]
23:24[applause]
23:28>> This is the thing about the model is uh
23:30I think everyone's looking for like the
23:31one weird trick to do it. The There's
23:33just like that doesn't exist. There's
23:34nothing like that. The The way the model
23:36works is you have to approach it
23:38empirically. You have to give it a task
23:39that's too hard. You have to give it the
23:41tools to verify the work like you would
23:43yourself, like you would if you were
23:44doing the task. You have to see where it
23:46struggles and then uh you have to like
23:48fix that either with better prompting or
23:51with a skill or if the model's missing
23:53context like give it a MCP so it can
23:55pull in the context that uh that it
23:57needs.
23:58Uh that's kind of it.
23:59>> It sounds very simple.
24:01>> [laughter]
24:01>> I think people tend to overthink it a
24:02little bit. I think people tend to
24:04overengineer.
24:05Cuz I think in a lot of ways like when
24:07we built systems in the past, that's the
24:08way you had to do it. So when I look at
24:10engineers that have been, you know,
24:12coding for a long for a long time, you
24:14know, like for for years or for decades,
24:16this is a really really common failure
24:18mode is trying to over specify and it's
24:21trying to be overly specific and then,
24:22you know, get the model to do the to do
24:24the task exactly the way that you would
24:25have done it. And that that's just not
24:27the way the model works.
24:29But I think a lot of people are kind of
24:30unlearning this and it's a journey to to
24:32unlearn it. And um
24:35it's a journey to kind of figure out how
24:36how do you treat this thing like you
24:38would a coworker. I think that's the
24:39level of intelligence that it's at now.
24:42>> And as part of this, let's go deeper
24:43into this task that's still running. Two
24:45weeks since you launched it a go two
24:46weeks ago.
24:48How many agents did it spawn?
24:50>> You know, I'm not sure. I I can ask Bod
24:52and then I I can get back to you. I
24:53would I would guess
24:56thousands, tens of thousands.
24:57>> Thousands. Has anyone in the audience
24:58had a uh prompt to to any of the models
25:02that run that spawn more than a thousand
25:03agents?
25:05No?
25:07I think this is another of the tips,
25:08like the best Claude users are able to
25:12spawn tasks that are really
25:14providing you a lot of leverage, like
25:15thousands of agents.
25:17>> Yes.
25:18>> How do you do that?
25:19>> There There's a few different ways to do
25:21it. Um
25:22the easiest way is dynamic workflows.
25:26To use dynamic workflows, it's a fairly
25:28new feature in Claude code.
25:29And all you have to say is use a
25:32workflow.
25:34That's it. And then Claude will just
25:35trigger the dynamic workflow. What a
25:37dynamic workflow is is essentially we
25:39have the we have the Bun runtime.
25:41We use Bun as a sandbox, and we start a
25:44virtual machine within Bun.
25:46And we let Claude start a lot of agents
25:48and orchestrate them. And it it doesn't
25:50just do one agent, it doesn't just do
25:52like 10 parallel agents. What it might
25:54do is um let's say a task is like
25:56rewrite the code base, or do really
25:59in-depth data analysis over some really
26:01complicated data, or maybe like build a
26:04very complex feature that takes multiple
26:06stages,
26:07and maybe dozens of pull requests.
26:09And so what it's going to do is it's
26:10going to start a bunch of agents to do
26:12kind of like the first pass.
26:14Based on that, it might do a second step
26:16where it has another set of agents that
26:18verify the work, or that summarize the
26:21work.
26:22Then it might do like a third stage,
26:24where it'll fan out again.
26:26So it'll kind of productively
26:27orchestrate a bunch of different agents.
26:29So my background is functional
26:30programming.
26:31And so the way that we designed this is
26:33is essentially an algebra for agents.
26:36So there's a way to run agents in
26:38sequence. There's a way to run agents in
26:40parallel. And Claude has different tools
26:43in order to orchestrate these agents
26:45inside of the sandbox, to use tokens
26:47efficiently, to do really, really
26:50complex work. It's kind of
26:52cool and something that just hasn't
26:55really been written about a lot. Like
26:56this is actually like a new form of test
26:58time compute. Like when we talk about
27:00the scaling laws and kind of we talk
27:01about the model getting more intelligent
27:03over time,
27:04historically,
27:06it's been a function of the size of the
27:07neural net,
27:09the amount of training data, and the
27:10number of flops that you put in to the
27:12training.
27:13And then recently, we also added test
27:15time compute. So this is essentially a
27:16fancy way a researcher way of saying how
27:18many tokens does it generate.
27:21And now
27:22dynamic workflows are essentially a new
27:24way to orchestrate test time compute.
27:27And it's a new way to kind of really,
27:29really ramp up the amount of test time
27:30compute that you use to do a really hard
27:32task.
27:34So this all very long way to say this is
27:36one way to launch thousands of agents in
27:38a way that is productive and efficient.
27:40A second way to do it is loops and
27:43routines.
27:44Loop is essentially a cron job that's
27:46running locally for Quad. Routine is the
27:48same thing, but it's running the Quad in
27:50the cloud.
27:51So you can close your laptop. And this
27:54is like slightly different because for a
27:55dynamic workflow, it's one task and you
27:58break it up into chunks.
27:59For loops and routines, it's one task
28:01that is repetitive, that doesn't share
28:03context, but it might share memory.
28:05And you kind of do this like over and
28:07over. You can do it like maybe every
28:08hour, every 5 minutes, every day.
28:11And so the thing that we've started
28:12doing is um we actually have Quad
28:14maintaining itself now.
28:16And the way we do this is we have a
28:18Slack channel where we just had Quad
28:21start a bunch of different routines to
28:23maintain its own code base. And we
28:25actually do this for the CLI, for the
28:27iOS app, for the Android app, uh for the
28:29desktop app.
28:30And you for example, one routine is
28:32clean up dead code.
28:34This is a single prompt. It's like one
28:35sentence. Quad runs this every day.
28:38It'll look for dead code across all the
28:39code bases using static and dynamic
28:42analysis. We didn't prompt that. It just
28:43kind of figured it out.
28:45And it'll put up a pull request every
28:46day to the weak the dead code.
28:48Another example is shipping experiments
28:51that should go out. Um so the
28:53experiment's already out to 100%. It'll
28:56delete it from the code base and it will
28:57just ship it. Another one is writing
29:00tests for areas of the code base that
29:01need test coverage. Another one is
29:04deleting tests that don't need to be
29:06there cuz you know, they were kind of
29:07useless tests added by older models or
29:09added by people at some point. One that
29:11one that I really love is this um
29:14I forgot what we called it. I think we
29:15called it abstraction police.
29:17And the idea is there are often in a big
29:20code base, there's kind of the same
29:21abstraction and it appears multiple
29:23times and if you kind of squint it
29:24actually maybe should just be the same
29:26abstraction, but kind of over time for
29:28whatever reason you rebuilt it multiple
29:31ways in different parts of the code
29:32base. So Quad kind of goes out every day
29:35across all our code bases, it finds
29:37these nearly duplicated abstractions and
29:39it unifies them. And so now we have
29:41every day maybe 20 or 30 of these
29:43routines. It's running across all of our
29:45code bases and
29:47it's not totally there yet, but we're on
29:50the path to fully automating the
29:51maintenance of our apps by doing this.
29:54And this is again hundreds of agents
29:57running every day, sometimes thousands
29:58of agents every day. It's doing the work
30:00of you know, dozens or hundreds of
30:02engineers. This is kind of what it used
30:04to take to do this kind of work.
30:06And this means that engineers can just
30:08like do the thing they actually want to
30:09do, which is ship new product and talk
30:11to users and do stuff that's actually
30:14fun.
30:15>> I guess nice conclusion from this, which
30:17you have mentioned in the past that
30:19basically coding is solved, right?
30:22You have mentioned this.
30:24Um
30:25I'm curious now that effectively
30:27everyone can write software, what
30:29separates the exceptional builders
30:34from the rest? What what are the
30:35qualities now that everyone can ship
30:37code?
30:38>> I I would give like one caveat. So
30:40coding is solved for the kind of coding
30:42that I do.
30:43It's not solved for everyone.
30:45You know, there's still code bases that
30:46are like super deep systems code bases
30:49where quad still struggles. There's
30:50distributed systems where quad still
30:52struggles. There's really kind of in the
30:55weeds UI verification, like something is
30:56off by pixel or something. Quad is still
30:58not perfect at this. Like Opus 5 was a
31:00big leap in vision and computer use, but
31:03it's still not perfect.
31:04Um but I I'm actually curious for people
31:07here, maybe raise your hand if 100% of
31:10your code is written using agents. You
31:12don't write any code by hand anymore.
31:16It's pretty good. Okay, how about more
31:18than 50%?
31:22Slightly less hands, maybe about the
31:23same.
31:24Yeah.
31:25So I think it's like it's getting there.
31:27So it's kind of getting to this to you
31:29know, to being solved for more and more
31:30kinds of code. And that's kind of cool.
31:32When I think about the people that are
31:34the best at using quad,
31:37I think there's a certain mindset that
31:39you can bring that's really effective.
31:41And it's really about being empirical.
31:45So forget all of the things that you
31:47learned about past models. Forget
31:49everything that you've learned about
31:51computer science theory in class.
31:54Look at the model,
31:55try to do a task, see where it
31:58struggles, and then based on that
31:59adjust. So it's just like very much
32:01become it's not a theoretical science,
32:04it's become an empirical science.
32:07So I think people that are really good
32:08at this, that are really good at kind of
32:09forgetting their priors, letting go of
32:12you know, this like maybe idea that
32:13didn't work before and just being open
32:14to trying it again.
32:16This is the kind of skill that's just
32:17very, very successful now.
32:20>> Now my last question is given everything
32:23that we talked about, if there's someone
32:26here that's studying CS,
32:28and you you learned to program before
32:30this era of AI agent coding,
32:34what should students still learn the
32:36hard way, like the old way.
32:39>> So for me,
32:40I learned computer science practically.
32:45I learned it by teaching myself to code
32:47in order to solve problems.
32:49Whenever I was doing this, I was doing
32:51it to solve a particular problem that I
32:54had.
32:55So I actually first learned to code on a
32:57TI-83 calculators.
32:59Um this is back in middle school and um
33:02I ended up actually writing a guide on
33:04the internet for programming TI-83
33:06calculators. It's still up on the
33:07internet somewhere.
33:08Um and it was uh it was basic. That that
33:10was my first language.
33:12And I I learned how to program on a
33:14calculator so I could just like get
33:15better at my math tests
33:17by uh by cheating on the test.
33:21>> [applause]
33:25>> So it it it was about something
33:27practical, you know, that like to me as
33:28a middle schooler that was kind of like
33:29the most practical thing I could think
33:30of. And I ended up getting good grades
33:33and then I got this little serial cable
33:34to give the you know, the programs to my
33:36classmates and they got really good
33:37grades. And then the math got a little
33:39bit harder. Um it wasn't something that
33:42I could solve in basic anymore. So I
33:43kind of went from this like, you know,
33:45like maybe algebra solver that was
33:47written in basic and I had to solve
33:49harder problems. And um you know, like
33:52once we got into calculus, I had to
33:54learn assembly so I can write
33:56a better solver so I could cheat better
33:57on the test now that it was calculus.
34:00And so for me, programming has always
34:02been very practical and I think this is
34:04always my advice for people in school is
34:06learn not just the computer science.
34:08This is like intellectually fascinating
34:10and it's really really interesting to
34:12know, but learn how to apply it. And
34:13often this is about building startups.
34:15It's about building products. It's about
34:17developing your own design sense,
34:19developing your business sense, learning
34:21how to how to do data science, learning
34:23how to talk to users. There are all
34:25these other skills and when you combine
34:27it with computer science and
34:29engineering, that's where it becomes
34:30really really valuable. So those are the
34:32hard skills that I would still be doing
34:34by hand.
34:36>> So, if I'm hearing and summarizing,
34:38start with making something you want
34:40first for yourself, and then level up
34:43and make something people want.
34:45>> Yes.
34:47>> And we just have one last special
34:49announcement for us. You want to one one
34:51last thing?
34:53>> Yeah, so um
34:54for everyone here today,
34:57uh you are getting Max 20X.
35:03>> [cheering]
35:08[cheering]
35:09[laughter]
35:15>> Pretty good.
35:16>> [applause]
35:17>> So, look for look for a code in your
35:19email. And uh I can't wait to see you
35:21what you build.
35:22>> We'll send an email.
35:26>> [applause]
35:26[cheering]
35:30[applause]
35:31>> So, I'm curious. Someone in this room
35:32should be building something that runs
35:34hopefully multiple months and thousands
35:36of agents now that you have the account
35:39to do it. And with that, thank you so
35:41much, Forrest.
35:42>> Thank you.
35:43>> Thank you.
35:43>> [cheering]
35:47[applause]

About this transcript

This page contains the full transcript of Boris Cherny: We Cut 80% of Claude Code’s Prompt by Y Combinator, generated from the public captions YouTube serves with the video. The transcript has 6,868 words across 1,096 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.