YT YouTube2Text

YouTube2Text → Transcripts → LIVE: Poteto (creator of pstack) on shipping 1,000's of PR's a month at SpaceX

LIVE: Poteto (creator of pstack) on shipping 1,000's of PR's a month at SpaceX — Transcript

by Matt Pocock · 12,075 words · 1,698 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00So, hello folks. I've got another treat
0:02for you today. Last time on this kind of
0:05podcasty thing, I suppose, we had Uncle
0:07Bob and we talked about software
0:09quality. We talked about agents. We
0:10talked about lots of cool stuff. Now, we
0:13have uh an incredible guest, someone who
0:15I'm delighted to welcome on, who's been
0:18exploding on Twitter recently about
0:21software factories, um about increasing
0:24the quality of your work, about
0:26increasing your velocity and climbing
0:28the trust ladder with agents so that you
0:30can ship more and more and more. And it
0:32is potato. Welcome. Thank you so much
0:35for joining.
0:36>> Thanks for having me. Yeah, very excited
0:38to be here. Yeah, big fan of yours
0:41>> and a huge fan of yours. I think people
0:43have been talking about this like it's
0:44like the meeting of the skill minds, the
0:47skill Mount Olympus or something because
0:49both of us have very popular skill
0:51libraries. Um I've not, as I was saying
0:53before we started, I've not used a ton
0:55of yours and like I want to get all of
0:58the juice out of your brain so that I
0:59can go and use it properly and use it
1:01better. And I think where I want to
1:04start with this is you gave a talk um
1:06pretty recently like um about 10 days
1:08ago and posted on X which went
1:10absolutely nuts as about how I shipped
1:132,500 PRs last month to production got
1:16about 3 million views or something on X
1:19and I watched it and I loved it and I
1:21recommended it and I kind of want to run
1:24this as almost like a Q&A of that talk
1:26basically of giving you because it just
1:29I just had tons of questions about it
1:31and I wanted to dive into it. And I
1:33think where I want to start is you talk
1:36about a trust ladder with agents where
1:39you as you trust agents more, you can
1:42get them to do better and better things
1:44and or scale them to up to use more and
1:47more agents. So what is your story of
1:50how you climbed the trust ladder and how
1:53did that work when like you got SpaceX
1:55and
1:56>> started climbing more and more?
1:58So I think this the the journey sort of
2:00began even before I joined cursor uh
2:03which is now SpaceX AI. Uh so the story
2:07is um
2:09after Meta so I I used to work at Meta
2:12on the React team. Uh I took a month off
2:15uh because I was feeling kind of burnt
2:16out and of course when what what do you
2:19do when you're burnt out? You go and
2:20start a new side project. Um and so I
2:23started a side project. you know, I was
2:25uh of course using AI to to write code.
2:28Uh but then I started to realize uh you
2:31know, I was spending like so many hours
2:32just micromanaging one agent, right? And
2:36you know, at the time, this was back in
2:38February, maybe February, early February
2:41or January, you know, people were really
2:43obsessed with this idea of like
2:44orchestration. This was like, you know,
2:46before, you know, things like cursor,
2:48you know, like the agents window was had
2:50become popular. So people were still in
2:53like like 2 land you know in their
2:55terminal and they were all talking about
2:56okay here you know I built a custom
2:58orchestrator right and so of course I
3:01had I was a bit nerd sniped by that and
3:03you know as I was building my toy
3:05project uh I got nerd sniped by oh how
3:08do I make my AI coding setup more
3:10efficient and so you know I I kind of
3:13started the journey there where I just
3:16you know took a step back and realized
3:18you know I was spending all this time
3:20micromanaging a single agent you know I
3:23was creating skills and I was like
3:25finding it quite difficult to measure
3:27the output or the the result the impact
3:30of the skill as well so I was kind of
3:32flying blind but I was you know
3:34iterating really fast um and um so that
3:39project eventually sort of became the
3:42basis of PAC even though I didn't know
3:44it at the time um and a lot of some
3:48tricks I had learned like building that
3:50early set of skills. Actually, it's
3:52still open source if you want to if
3:54anybody wants to take a look. It's on my
3:56GitHub like potato
3:58noodle n o d l e. Um, and in there you
4:02will see some skills and a brain
4:04directory. And so I was really
4:06interested in this idea of how do I, you
4:09know, extract my own ability, if that
4:12makes sense, and give it to the agent,
4:14right? cuz I was I I realized that you
4:16know all I was trying to do was trying
4:18to teach the agent to write code more
4:19like me you know do do you do do
4:22workflows more like me. So you know the
4:24skills were like an entry point to doing
4:27that.
4:28Um and then you know after I joined
4:31cursor uh I was starting to work on the
4:33agents window and uh it had a lot of
4:36performance issues. Uh it was it was it
4:39was pretty laggy. Uh and so since I had
4:41experience working in React, I was asked
4:43like, "Hey, do you want to come and help
4:45out uh with the agents window?" Um and
4:48so the the this beginning of the cursor
4:52journey was very manual. Uh I was deep
4:56in like looking at like flame graphs and
4:59heap snapshots and trying to see like
5:01why exactly is the app so slow. Uh but
5:04then coming back to the same realization
5:06like you know I was sort of the
5:07bottleneck. I was doing everything
5:09manually. I was sort of the meat proxy
5:11in a way, right? I was the meat proxy
5:13between my agent and Chrome DevTools. Uh
5:16and I was like really annoyed by that.
5:18>> And what month of the year is that?
5:20Let's say where are we in the timeline?
5:22>> Uh so I joined Cursor in March. So this
5:25was like early early April probably
5:28early April is when you know uh I joined
5:31and I didn't have any skills, right? I
5:32had I I sort of abandoned my personal
5:35skills because I didn't think they'd be
5:36relevant anymore. Uh but then working on
5:39the agents window uh and now working on
5:42grockbot uh I sort of realized that a
5:45lot of the lessons I had learned from
5:48those skill time building the the
5:49initial set of skills were very relevant
5:52especially around things like
5:54verification
5:55uh you know being very rigorous in your
5:57work um because I think from my
6:02experience even the the frontier ones
6:05tend to
6:07tend to take shortcuts. Uh they tend to
6:10do the easy thing. Uh so uh a lot of the
6:14skills that I've built have been around
6:16how do I make the easy thing the right
6:19thing? You know, how do I make that the
6:20best thing?
6:22>> The idea of sort of distilling your
6:25expertise and turning what you do every
6:28day into processes, that's something
6:30that feels super familiar to me. That's
6:32exactly what I've been doing with the
6:34skills. And I suppose there's something
6:37in that which is a lot of people think
6:39domain expertise is getting less useful
6:41now as people uh start to rely more on
6:44AI where what do you think about that
6:47just as a sort of vibe check before we
6:49start talking
6:51>> I actually feel like domain expertise is
6:52is more important than ever you know uh
6:56I think I wrote this on my ex at some
6:58point but you know at times I sometimes
7:01think of you know AI as is like
7:05especially as the models get smarter and
7:06smarter and more capable and the
7:08frontier models are just getting so good
7:10like I love Opus 5.5 by the way um uh
7:15you know as the models get really really
7:17really good it almost becomes like the
7:19bottleneck is no longer the agent right
7:22it becomes your ability to express your
7:25intent and your goals in a clear way
7:28that the agent can understand and
7:31actually carry out and That's why I
7:33think you know like people with a lot of
7:35domain expertise are extremely have a
7:38have a huge advantage in my opinion
7:41especially if you're a little bit like
7:42you know tech technoc curious you know
7:45so I I think of people like you know
7:47like uh like a doctor or a lawyer or you
7:50know someone who who has a deep
7:52expertise in a particular
7:54non-engineering domain and if they're
7:56actually just a little bit techsavvy and
7:59they can figure out how to use agents
8:00they can actually build really really
8:02great products, right? If they if they
8:05have a clear enough vision in their head
8:07and they can articulate it in a way that
8:09the agent can build it, you know, I
8:12think that that is really the the
8:15bottleneck these days is is like the
8:18transfer of your intent, right, and your
8:20vision to the agent.
8:23>> Yeah. I've been obsessed with language
8:25basically since agents um dropped. are
8:27just obsessed 100% and thinking
8:30constantly about the the composition of
8:32words, how I can make things sharper,
8:35what um what might be hidden in the
8:37phrases that I'm using. And it's and
8:40finding what I love is when you find a
8:42word that the agent then hooks on to and
8:45then goes, "Okay, I'm going to reinforce
8:47that word. I'm going to reuse that in my
8:49thinking traces." You know, I found that
8:51with um TDD was an early example of
8:53that. a lot of chat about TDD recently
8:55of like, you know, people say, should
8:57you use TDD with agents? Doesn't matter.
8:59What you're doing is you're getting the
9:00agent to think about TDD, getting it to
9:03write tests, getting it to prioritize
9:04things in a different way than it did
9:06before. And that's why sort of grilling,
9:08I think, works effectively. Grilling is
9:10a
9:11>> Yeah. Yeah. It it draws those words out
9:13of you, right? or or at least it helps
9:16the agent understand your thinking so
9:19that they can propose those words to you
9:20and you can pick up and say yes exactly
9:22that.
9:23>> Uh I've actually copied some of the the
9:25tips that you've shared as well where
9:27you know one of my favorite ones that
9:28you've shared recently or or not or like
9:30maybe in the past couple weeks is about
9:33uh reducing or eliminating tautological
9:36tests. Like one of my pet peeves of
9:38agents is like all of the useless tests
9:40that they write. And so, you know, that
9:43was one thing where, you know, the word
9:45tutology, right, is is is I guess, you
9:47know, not many people necessarily know
9:49that if if especially if English isn't
9:50your first language, but there's a lot
9:53of meaning to that word. And it's like
9:56it's almost like compressed, right? Like
9:58you compress a lot of intent and meaning
10:00into words. And so I I I totally agree
10:04with you. I think language I've always
10:06been interested in language actually uh
10:08like programming languages natural human
10:11languages and how they came to be and
10:13it's so interesting that now with agents
10:15it's sort of like this meeting of
10:18natural language with programming
10:20language but it's all it's all language
10:21out of the hood it's all communication
10:23>> totally I did a drama degree right so
10:25you know I've been thinking about
10:26language and Shakespeare and stuff for a
10:28long time and so this all feels very
10:30familiar
10:31>> um so okay there's sort
10:34before we get into like because I think
10:36the thing I want from you is like
10:38software factory stuff, right? Software
10:40factory is the big buzzword. Software
10:42factory is the thing that I'm thinking
10:43about too. I'm sort of releasing a
10:45course in that direction too.
10:47>> And it's this sort of scaling yourself
10:50up to un unrealistic numbers of PRs
10:53basically or PR numbers that sound
10:56ridiculous to people who don't
10:57understand how this works. So where I
10:59want to get to is sort of from people
11:01who are doing kind of like one to five
11:03agents today up to, you know, hundreds
11:06of agents running at once and how that
11:08sort of functions. And so I'd love to
11:10hear about your metaphor of the Michelin
11:12Kitchen instead of the software factory
11:15because I think that says a bit about
11:16the way you think about this stuff.
11:18>> Yes.
11:20Yeah. I I I've I've never really liked
11:22the term software factory. Not because
11:25you know it's not accurate but I think I
11:28think the a lot of people when they
11:29think factory right they don't
11:31necessarily equate that with quality or
11:34craft right things which are very
11:36important to me and a lot of people and
11:39technologists who work you know building
11:41products we care about the user
11:44experience we care about the things
11:45we're building. So while so while I do
11:48think software factory is an apt term,
11:51it also I guess maybe conjures up
11:53negative, you know, maybe sometimes
11:55negative connotations. So Michelin
11:58Kitchen is the thing that I've sort of
12:00landed on where it's much more I feel
12:02like it's much more aspirational and uh
12:05I like the metaphor a lot cuz you know I
12:06like food. I'm called potato of course
12:09and I like cooking and I see a lot of
12:11parallels right like with food right
12:14when you're cooking a meal for yourself
12:16for example it's both utilitarian like
12:19you're trying to just feed yourself
12:20right and and survive uh but it can
12:23actually be transformed into art right
12:25and that's what what a Michelin starred
12:28chef or even just a chef or a cook can
12:31do with food is take something very
12:33ordinary and turn it into a delicious
12:36meal that you know takes you back to
12:38your childhood days or something like
12:39that. Um and so it almost like mirrors
12:43that trust letter that I talk about
12:45where uh you can sort of imagine your
12:48own journey as a home cook, right? Uh as
12:50a home cook, you are doing all of the
12:52food, the cooking yourself. You cut all
12:55the vegetables, you do all the prep
12:57work, you do all the cleanup, you know,
12:59you are the one man or one woman show
13:03really. Um, and it's an interesting
13:07thought experiment like, okay, if you
13:09were to cook a meal and then you add
13:11people, right, your your your partner
13:14trying to your brother, your sister, and
13:16now suddenly you have your whole family
13:18in the kitchen. I think most people
13:19would get very stressed by that, right?
13:21The thought of, oh, so many people are
13:23just mocking around in my kitchen. They
13:24have no no idea where all the utensils
13:26are.
13:27>> I have a max capacity of one person in
13:28the kitchen. Yeah, absolutely. So, I
13:31feel like that that's really apt because
13:33when you ask yourself that question of
13:35how do I go from being a solo cook,
13:37right, to having an army or even not not
13:41even an army but a few sue chefs, right,
13:44that that are helping me in the kitchen.
13:45How do I think about dividing the work
13:48in a way that makes sense? You know, I'm
13:50not dividing work just for the sake of
13:52it, but in a way that actually makes the
13:55sum the to the better than, you know,
13:57the total of its parts. And so the
14:00Michelin kitchen metaphor to me like
14:02works really well in that regard because
14:05you know as a chef you're you know if
14:08you become a chef you're in a position
14:10where you're not necessarily cooking all
14:12the food yourself anymore but you are
14:14thinking you're almost like the tech
14:17lead right for the kitchen where uh you
14:20know chefs have to think about you know
14:21not just cooking but they have to
14:23basically organize the whole kitchen and
14:25they're like the CEO of the kitchen they
14:27have to think about when do you order
14:29ingredients, how do you store them, how
14:30do you prepare them, when do they have
14:32to be prepared, you know, it's a whole
14:34it's a whole job, right? That's not just
14:36cooking. Um, and I think that again it
14:39mirrors so much of how engineers write
14:42code today where you are not writing the
14:45code yourself anymore. You have agents,
14:47right? But you as the human are still
14:49responsible for the final outcome,
14:51right? your name still is associated
14:53with the work that you do, your
14:55reputation and you know so how you set
14:58up your kitchen right and how you set up
15:01your skills your environment your
15:03codebase I think are ultimately the new
15:07ingredients that go into um building
15:10product
15:12>> yeah I think what I love about your
15:14approach is the amount of focus that you
15:16put into the environment that the agent
15:18operates in right because I think a lot
15:20of people they think, right, the agent
15:22is good. I'm probably not going to be
15:25able to make it better. Let's just trust
15:27what these magic model people have put
15:29into the harness and the model
15:31combination. Uh, there's nothing I can
15:33really do, like I can't mess about with
15:35claw codes internals or something or
15:36whatever you're using. Um, but what I
15:40love about your approach, and it's
15:42something I advocate for too, is that
15:43you can change the environment the agent
15:45operates in, right? you can make changes
15:48in the codebase and also give it tools
15:51for verification as well and allow it to
15:54verify its own work. So the thing I I
15:56loved about watching that talk is the
15:58amount of focus you put in verification
16:01and like that is the lever that you can
16:03start to generate trust. Can you talk
16:05about that and what that concretely
16:07looks like? Let's start like looking at
16:09practical ways that people can improve
16:10their own processes, their own kitchens.
16:14Yeah, I've I've I've said this a lot
16:16actually that you know even if you don't
16:18use PAC or you know your skills I think
16:21that the single most important skill
16:24that should be in your toolkit is
16:27verification because without
16:29verification and for for by the way for
16:32those watching who don't know what that
16:33means it's this idea that you can give
16:35you can sort of give your agent uh hands
16:38and eyes in a way that's the the analogy
16:42I
16:43where the agent is able to run the code,
16:46right? And actually uh interact with it
16:49like a normal human user would and also
16:52do things like you know debug it, you
16:54know, take traces and snapshots. Um and
16:59uh funnily enough like that was actually
17:00the first skill I built when I joined
17:01Cursor. uh that gave me a lot of that
17:04was that was the thing that actually
17:05started to let me ascend the trust
17:07ladder a little bit in a way that some
17:09of the other skills I had looked at or
17:12built had not really let me do because
17:15no matter how good you know some of the
17:17other skills were like the how skill,
17:19the why skill, the unsop skill were, I
17:23was still relying on me right as the
17:25proxy between my agent and the output.
17:29So that you know if the agent can't
17:30actually see the result of its work
17:32there's no way it can actually iterate
17:35right and so this is where people start
17:36to talk about loops this idea of a loop
17:39and really I think the term loop you
17:41know seems kind of uh almost abstract
17:45like people like what what what is a
17:47loop what is an agent loop but really to
17:49me like the most important part of a
17:51loop that allows it to be a loop is the
17:54verification part because the agent is
17:56able to to verify by its own work and uh
18:01you know that takes you out of the
18:02equation where now I can actually do
18:04something like so the very one of the
18:06very first use cases I had for
18:08verification was you know like the
18:10performance work that I was doing on
18:11cursors agent window and I want I wanted
18:14to get to a point where I could do
18:15something called hill climbing uh which
18:17is a term that I I think the labs uh
18:20talk about a lot which is this idea that
18:22you know you have some kind of rubric or
18:25a way to judge or score something And
18:28now because you have a loop, you can
18:30have an agent continually try to make
18:33improvements to that. Uh I think
18:35Carpathy, Andre Carpathy also famously
18:38released uh something called auto
18:40research that has a lot of these ideas.
18:43Um but yeah, verification I would say is
18:46probably the most important skill in PAC
18:49uh and many other you know tool sets. Uh
18:53and I think it's the most important
18:55thing to focus on. So a lot of the a lot
18:59of I spent a lot of time actually you
19:01know tuning the verification the
19:04creative verification skill um and also
19:07internally the the we have so many
19:09verification skills now like every app
19:11that cursor has or spaceexai has has a
19:15uh verification skill that is
19:17automaintained as well
19:19>> uh and it's become critical
19:21infrastructure for our team because
19:22everybody uses it
19:24>> and you went pretty far with that too
19:26right like you had a um in your talk I
19:28saw that you actually built a custom CLI
19:30for that too. So what does that CLI do?
19:32Like how does it execute things and why
19:34did you I mean that's proof of how deep
19:36you're going right of how much you're
19:37pushing that.
19:39>> Yeah. So this is actually a tip I
19:42learned early on where
19:45um I guess you know back in January or
19:49or late last year the thing that people
19:51were concerned about was context window,
19:54right? That was the big the big topic at
19:56the time was how do I you know manage
19:58the context window because you know
20:00compaction summarization wasn't really
20:02that good yet and people were always
20:05people had this there was almost this
20:07meme in the community that you know once
20:10your agent summarized or compacted once
20:13it would become sort of stupid right for
20:15the rest of your session. So there was a
20:18lot of thinking around like
20:20you know being very efficient with your
20:22context usage and so that was actually
20:25the inspiration for some of the uh the
20:28CLI work inside of the verification
20:31skills. I guess now it's less so about
20:34context because uh you know agents are
20:38much better or harnesses have gotten a
20:40lot better with summarization.
20:42Um, I still think there's some benefits
20:45to, you know, uh, having a clean context
20:49window. Uh, so the CLI is really just
20:51more of a way for me to take the
20:53deterministic parts of what the skill
20:56does and encode that into a script or
20:58CLI to reduce to kind of take away the
21:03judgment that would otherwise
21:05unnecessarily be used because with
21:08judgment so I also think of you know
21:10agents and skills in sort of like it's
21:13like a gradient
21:15you have some parts of the work that are
21:18entirely judge measurement based right
21:20you know something that requires thought
21:22you know putting together multiple
21:24pieces of context thinking um and then
21:27you have the more deterministic parts
21:28like I don't know if you wanted to uh
21:31refactor some code right from one
21:34pattern to another that's very
21:35mechanical right you don't you don't
21:37need an agent to think about it and come
21:40up with it in a novel way each time
21:43right and so that that was really the
21:45inspiration for the CLI and you'll see
21:47this in a lot of the other skills that I
21:48built is like I try to extract out the
21:52deterministic parts and turn that into
21:54code and just leave only the parts that
21:56actually require judgment to the agent.
22:00So in a way I think of the seal as kind
22:01of like your wrapper, right? It's a
22:03wrapper with some light instructions
22:04around how to use these custom tools
22:06that are inside of the skill. Um but
22:10yeah, I don't think the CL is really
22:11that interesting in its own really. It's
22:13not like a novel piece of software. It's
22:16just something that interacts with like
22:17Playright and the Chrome DevTools
22:20protocol and calls a bunch of APIs. It's
22:22like it's just a bunch of glue.
22:24>> No, it's fascinating because it's a way
22:25of hiding information from the skill,
22:27right? It's a way of conserving the
22:28skill, keeping the skill quite small, I
22:31imagine, and then you're able to
22:32delegate more of the complicated
22:33deterministic stuff into a script within
22:36the skill. So it's almost you're
22:38compressing information and making the
22:41agent do more consistent things more
22:44consistently.
22:45>> Yeah,
22:45>> that's fascinating.
22:47And it it also helps I guess if you care
22:49about context window it it does help
22:52because now the agent doesn't need to uh
22:54you know re reinvent uh things cuz uh
22:58one thing I had noticed early on when we
23:00didn't have a CLI
23:02was that uh well the the agent would try
23:05to verify it work but it would basically
23:07rebuild the world each time and then
23:09every agent did it differently and I was
23:11starting to notice like that's very
23:12inefficient right I was wasting it it
23:14was actually not just about context
23:16usage but also speed, right? Like
23:18because now an agent had to actually go
23:19off and write the scripts or the CLI and
23:22test it and you know and it doesn't work
23:24and the last agent did it and it worked
23:26but it discarded it. So it was just very
23:28obvious at that point like I should just
23:30turn this into a CLI and put that inside
23:32of the skill uh so that every agent that
23:35uses it now benefits from that same
23:38piece. Um but I I also think like you
23:42know it's a good push for people to
23:43think about is how much of your skills
23:46and rules could actually be
23:48deterministic.
23:50Um that's like another core thing or or
23:53one of my core principles that I like to
23:55think about is yeah how do I uh make
23:59very efficient use of determinism and
24:02non-determinism
24:04and you know let Asians shine at the
24:07non-deterministic parts right because
24:09that's what they're trained to do. Um
24:11and the other parts which are much more
24:14mechanical or you know straightforward
24:16can be just pure determinism. Um, and
24:19you'll see this as well for things like
24:21doing migrations. Um, which is another
24:24big thing that I've I've talked about
24:26is, you know, going from one technology
24:29to another, especially one that is
24:32better for agents, right? And a lot of
24:34how you can do that migration is, I
24:36think, through things like scripts and
24:38CLIs, like the deterministic parts like
24:40code mods, you know, like crawling the
24:44abstract syntax tree and transforming
24:46code literally mechanically, right? like
24:48a script does it for you instead of the
24:50agent.
24:52>> Totally makes sense. I I mean I think
24:55what there's another thing there which
24:57is you're taking stuff away from the
25:00agent and you're kind of putting it in
25:03the environment too a little bit which
25:05is let's say you have a a thing that you
25:08notice the agent always gets wrong. You
25:10want to make that um just impossible
25:13within the environment. And that sort of
25:15comes down to code quality as well. I
25:18mean, I talk about a lot like having a
25:22what a good codebase means, right? What
25:24is a good codebase? And there's a
25:26definition I like which is a a good
25:28codebase is a codebase that's easy to
25:30make changes in, right? Easy to um
25:34change stuff without things screwing up.
25:36And that means that you have a lot of
25:37guard rails that you have a lot of um
25:40the agent or the human is constrained to
25:42very narrow paths. And that's again
25:44something you talk about in your talk.
25:46>> And you talk about this not only on the
25:48kind of sort of automated checks side of
25:52things. So linting and type checking
25:53blah blah blah but also in the way you
25:55design abstractions. And you guys even I
25:58think built a framework uh for your
25:59agent to work in too.
26:01>> I think what I'd love to hear is you
26:04obviously think of that as very
26:06important, right? And that's how
26:08important is that compared to other
26:10things you could be doing like building
26:11features or shipping work.
26:14Yeah, I think that's a um I almost feel
26:17like the new job of the engineer is
26:19really to to spend time on the
26:21environment. Um I almost actually wrote
26:23a tweet about this yesterday, but I but
26:25I didn't. But I think that
26:29I think that you you know if you if you
26:31haven't really spent time, you know,
26:33building trust in your agents and
26:34building skills and tools, you can get
26:37stuck in this mode where you're very low
26:39on that trust ladder, right? you don't
26:41have a lot of trust in your agents work.
26:43And so the only way to cope in that when
26:47you're in that situation is just to kind
26:48of lock in and micromanage your agents.
26:51And that's very time consuming. And when
26:53you're stuck in that mode, you don't
26:55really have the luxury to think about,
26:58you know,
27:00uh higher level things like like making
27:03yourself more productive. In the same
27:05way that uh I guess analogy would be
27:09like if you've never taken the time to
27:11learn like your tools right as a
27:13developer when you were writing code
27:14yourself and you know you've never heard
27:17of VS Code, you've never heard of Vim,
27:19you only knew about Notepad
27:22uh and you had hadn't even heard about
27:23Git. That's sort of the analogy. It's
27:25like you you haven't spent the time
27:27sharpening your own knives, right? And
27:29so, of course, if you have a dull knife,
27:32then everything's going to take a long
27:33time. Um, and you're going to be you're
27:36just going to be and and especially if
27:38you know deadlines are looming, then you
27:41don't have the now you're stuck in this
27:42rut, right? Where where you you you
27:45don't have sharp knives, you don't have
27:47good tools, but you're under all this
27:49pressure to ship, right? And so, all you
27:51can do is just focus on that. But I do
27:54think that, you know, if you can find
27:55yourself the time to actually spend time
27:58thinking about your setup, it's again
28:00going back to the cooking, you know,
28:02analogy,
28:04it's like uh, you know, if you, for
28:06example, if if cutting cutting cutting
28:09the garlic is like super slow, right?
28:11There are garlic mashers, right? You can
28:13you can buy and you put it in the thing
28:14and you like squeeze it out, right? It's
28:17super fast. Uh, machines and tools were
28:20invented for a reason, right? And so if
28:22you're operating a Michelin kitchen and
28:25your your your cooks had no tools, then
28:27of course everything's going to be
28:29extremely inefficient, very very, you
28:31know, every every every cook is going to
28:33make something up of their own. So I
28:35think the tools and the determinism to
28:37me are you know taking that part away
28:41and and just like you said about
28:42constraints as well. It's the
28:44constraints are are to me as well like
28:48uh actually a slight tangent on that is
28:50uh I think we should talk about
28:53TypeScript cuz like we we actually both
28:55share like a background in Typescript
28:57where you know you obviously have done a
28:58lot of work with TypeScript and total
29:00TypeScript and you know you're a leader
29:02in that space and I uh had adopted
29:05TypeScript pretty early and I had given
29:07like a talk or two at Typescript conf uh
29:10many years ago and so one of the the
29:13talk that I did actually was about type
29:15systems and constraining the
29:18constraining types. Like one of my most
29:20favorite things about Typescript is
29:22actually type narrowing, right? This
29:23idea that you go from a very broad type,
29:26right? That could be anything and then
29:28you through type guards and you know
29:31type narrowing and you know runtime
29:33checks you can actually narrow the space
29:35and say like oh this isn't just a string
29:38this is a very special type of string.
29:40It's a constant, right? like I but I I
29:42determine that through the type system
29:45and in a way it's like uh there's a lot
29:47of parallels I think to that with
29:49constraints in your codebase where is it
29:52you're you're constraining the space
29:55right if you if you think about category
29:57theory as well you know you're
29:58constraining the the number of possible
30:01types right that can can exist
30:05and you're saying there's only one type
30:07right and for for us like that framework
30:10that I'm called Dune. Uh it's not an
30:13open source framework. It's the the way
30:15I describe it to people. It's it's kind
30:17of like a internal Nex.js for our
30:20Electron apps. Uh but it comes with a
30:23lot of really really restrictive lit
30:25rules and the codebase is designed in a
30:28way that there's really only one way to
30:30do something. So we make use a we make
30:32use of a lot of conventional
30:35patterns. So like features all go into a
30:38specific directory. Well, every feature
30:41has its own directory. As an example,
30:43you know, there's like a a thing that
30:44discovers features like through a
30:46registry and like crawling the codebase
30:48and stuff like that. But this
30:51conventional pattern and the lint rules
30:55make for an environment where it's
30:57actually very hard to write bad code.
31:01And that sort of frees up the it both
31:04frees up your own mental uh you know
31:09capacity as well as the agent sort of
31:11doesn't have to think about that anymore
31:14where it's just like oh there's only
31:15there's I should just if I want to add a
31:17new feature it just goes in the feature
31:18the new feature directory and all the
31:20code goes in there and I'm not going to
31:22append to a god file right that was
31:25really actually the inspiration for
31:26those feature directories is the very
31:29first couple of versions of Grockbot
31:32were composed of like eight god files
31:36which were like at least 10,000 lines
31:38long if not longer and so I kind of had
31:40to break it up into smaller pieces.
31:43Uh but it was just observing you know
31:46actually that's another important part
31:47is observing how agents fail and then
31:50every time you see a mistake every time
31:52you see something that could be done
31:54better you think you step back and think
31:57how do I turn this into a lint rule? How
31:58do I make it so that the code base makes
32:01this impossible? Right? And it comes
32:03back to me for my you know my background
32:05learning Typescript and types uh type
32:08systems is how do I constrain the space
32:12so that you know I know precisely what
32:15I'm working with and I think yeah
32:16there's a lot of parallels there.
32:18>> Totally makes sense. And don't I mean
32:21it's funny that you mentioned TypeScript
32:22and Goth files in the same sentence
32:23because Typescript famously has a
32:2525,000line type uh file.
32:30Although I don't know if they've
32:30rewritten that and go as they probably
32:32have, haven't they? Um,
32:34okay. So, environment is important. You
32:38should watch your agent like a hawk to
32:39make sure that any mistakes it makes.
32:42You turn them into things in the
32:44environment. And the benefit of the
32:45environment is you're not overloading
32:47your agent, right, in terms of rules, in
32:49terms of things it has to remember. It's
32:51just in the environment. And so it
32:53stumbles into the rules and exactly um
32:56you know bounces off them and hits them
32:57at the right moment.
32:59>> So okay, we still haven't talked about
33:01the 2,500 PRs. Where do those come from?
33:03Like how do you you've built your trust
33:06ladder, you've worked on your
33:07environment, and you understand, okay,
33:09um I now want to scale up. So what are
33:13the mechanics of that scaling? Are you
33:16um initiating 2500
33:19like chats per month? That can't be
33:21right. So there must be are there any
33:23kind of automated triggers that trigger
33:26stuff in your repo? Like how do you get
33:28the software factory kind of triggering
33:30work by itself?
33:32>> Right.
33:34Um I'll definitely say that the
33:36prerequisite to you know something like
33:40a very high volume of of pull requests
33:43um is the environment. you know, the the
33:46stuff we just talked about where I
33:48definitely would not have been able to
33:49do this if I had not spent the time, you
33:52know, thinking about the kitchen, right,
33:53and the knives and the tools for my
33:55Asians. And so, in a way, I I think of
33:58this as
34:00I've spent the time building one kitchen
34:02and one restaurant. And now I I'm in a
34:05position where I don't actually have to
34:07be there anymore because the
34:10environment, you know, that the same
34:12analogy, right? It works really well.
34:14Yeah. You open chain of restaurants,
34:16right? That's
34:16>> Yeah. Exactly. Yeah. Exactly. It's like
34:19you're Gordon Ramsay and you know, you
34:21you've taught your executive chef like
34:23all the tricks of the of coming up with
34:25great menu. Uh and like the kitchen is
34:28set up really well. Everything's just
34:30perfect and you're now in a position
34:32where you can open your second your
34:33third restaurant. And I guess I I sort
34:37of see each project that I work on, like
34:39each big chat is sort of like a
34:41restaurant, right? and and I'm I'm I
34:44have multiple of them operating at the
34:46same time and I'm sort of like
34:48helicoptering between them sometimes
34:51some more than others depending on how
34:53in the loop I am but yeah definitely I
34:55think there's there there are external
34:59triggers and context that those projects
35:02don't have that
35:05for a long time I was the proxy for that
35:08so uh the best example I have is like
35:10you know you have a project that's
35:11working on a feature
35:13uh or you're trying to fix a bug and
35:15you're getting bug reports, but the bug
35:17reports are going to things like Slack
35:19or linear or X, right? And these are
35:24external systems that aren't connected
35:26to your inner loop. So, I like to talk
35:28about this outer loop and the inner
35:30loop.
35:32Uh I don't know if I'm using the
35:33definition correctly but to me my inner
35:35loop is like basically my engineers my
35:38agent engineers working on the code to
35:41an building towards an intent or
35:45snapshot of my intent right and the
35:48thing about that is that the snapshot
35:50can go stale right new information comes
35:52to light that I then have to be the
35:54proxy of and you know transfer that
35:57context to my agent so you know if you
35:59if you don't have these triggers pulling
36:02information back into the interloop,
36:04then you sort of have to play that role
36:07where you're you're off, you know, in
36:09Slack or X or or whatever and you're
36:12gathering context, right? You're getting
36:14context about bug reports, about feature
36:16requests, about, you know, something
36:19someone said about, you know, our
36:20backend infrastructure has some
36:22limitation, you know, all that
36:24information, you have to f that across
36:27to your agent. So that's where I think
36:29like tools like Grogbot are really good
36:32because they help you automate the outer
36:35loop as well. And when you connect those
36:37two loops, it's very very powerful
36:39because now all of a sudden your agents
36:41have the ability to get context for this
36:44for themselves, right? If for example
36:48uh you know either through just as a
36:51simple example like maybe you have the
36:52Slack MCP, right? Or you have uh your
36:56own harness, right, that you've built a
36:58Slack subscription into for a particular
37:00Slack channel. Now all of a sudden you
37:03can tell your agents, okay, subscribe to
37:05the Slack channel. Every time there's a
37:08uh, you know, bug report about
37:11something, go off and triage that thing,
37:14right? Go reproduce the issue, right?
37:16Using the verification skills that we've
37:18already spent time building and all of
37:20those other skills that we've set up so
37:22that I have a lot of trust, right? I
37:24have a lot of trust that these agents
37:25can actually go off and understand the
37:28bug, you know, uh verify that the bug
37:31actually still exists on main and it
37:34wasn't
37:35something about you maybe the users
37:38setup or their data or maybe I don't
37:40know they didn't install a dependency or
37:42something like that like uh basically I
37:45think uh creating that yeah creating
37:49those two loops and connecting them is
37:52really a very important part of the job
37:53these days. Um, especially if you are
37:56thinking about how to scale yourself.
37:58So, a big theme here is really just like
38:01always thinking about like what where am
38:04I the bottleneck in this process? Why do
38:07my agents need me, you know, to answer
38:09this question? I I always like to think
38:11about that. And so I try to think about
38:13how do I actually get the agent to
38:15answer its own question, right? But not
38:17by hallucinating, not by guessing, but
38:19actually real data. And you know, a lot
38:22of people talk about this idea of a
38:23company brain, right, or a context
38:25graph. I feel like those terms are
38:27unnecessarily
38:29complex uh or even abstract. To me, it's
38:33just about um how do I take information
38:37that my agent needs that I would
38:39otherwise have to go and pass it myself
38:42and just teach it how to do it, right?
38:44And that removes me from the equation.
38:47And so how I arrive at 2,000 or however
38:50many PRs is the fact that I have all
38:53these loops set up, right? And so uh it
38:57allows me to open chain restaurants,
39:00right? I can I can really parallels
39:02myself. So yeah, I'm not sitting there
39:04creating 2,500 chats, right? Of course,
39:06it's really like these projects are um
39:10actually cursor has a new feature called
39:12projects which are these like
39:13coordinator agents. Um and so the
39:17coordination co coordinator agents are
39:19really good at sort of delegating and
39:21not doing work of their own but they
39:23manage and supervise like almost a list
39:25of tasks and they spawn sub agents to go
39:28and do them. And so I'm just constantly
39:30feeding context or teaching the agents
39:32how to get their own context and then
39:33they're going off and doing the work for
39:35me. Uh and really the big the last thing
39:38I'll say to this is like the big unlock
39:40for me for getting to 2,000 PRs is
39:43starting from the question and working
39:45backwards of how do I get to the point
39:48where my agent can merge its own code?
39:51Because
39:53the obvious thing people ask me when
39:55when they when I tell them, "Oh, I
39:56shipped 2,000 and 2,500 pull requests
39:59last month." They'll be like, "How did
40:00you review that?" Right? That that's a
40:02lot of PRs to review. Like your team
40:04must hate you.
40:05>> Do do you mind if we go there in a
40:07second? Because a good question about
40:09that.
40:09>> Yeah. Yeah. Yeah.
40:10>> I want to like this analogy is great. I
40:12want to like deepen it a bit which is
40:15before if you're like manually
40:17initiating all those chats it's like
40:18you're bringing the orders to your chefs
40:20manually right whereas if you've got an
40:22agent sort of like doing the expo then
40:25you're able to sort of run it yourself
40:26itself what is what does that concretely
40:30look like then you've got these sort of
40:31grock bots that are um subscribing to
40:33channels pulling in Slack messages and
40:36you it sounds like have a couple of
40:37coordinator agents or like chief of
40:39staff agents that like monitor that or
40:43something like when you look at your
40:45computer to manage your agents, what
40:47does it look like?
40:49>> Yeah. So, so uh this is I guess somewhat
40:53confusing but we're working on you know
40:54simplifying and unifying but so uh
40:58there's graphbot uh which or you know
41:00you can use other tools of course as
41:02well but I I largely think of these
41:04tools as like your outer loop. These are
41:06tools like you know Grabbot that have
41:08connectors right these are connectors I
41:10guess they a lot of people call them
41:12personal agents um but they're
41:14connectors to things like your email
41:16your calendar slack uh plaid I don't
41:20know like all these different services
41:22and they are a great source of pulling
41:24context in to your work so the same way
41:29that a human like you know if I were if
41:31I was a manager and I was leading a team
41:34of engineers years. Um, you know, like
41:37when I used to work in Netflix, one of
41:39the biggest things that managers would
41:40talk about was this idea of context not
41:43control, which funnily enough, you know,
41:46has so much uh has so much uh carry over
41:49to the agents world. Uh, of you know,
41:52you you know, you you of course can
41:55drive to an outcome you want by control,
41:57right? Like by micromanaging, but what
42:00you want is to provide context instead,
42:02right? like teach the agent, teach your
42:04engineers how to be self-sufficient and
42:06then you don't have to micromanage them.
42:09>> Um, and so I see a lot of parallels
42:11there. Uh, but yeah, graphbot. So,
42:13concretely, I have some graph bots that
42:16look at my Slack channels, look at my X,
42:19uh, or my emails, uh, or linear, and
42:23they're just constantly they have
42:24routines that subscribe. So they're
42:26constantly watching and I have I I'll
42:29tell them things like you know uh I'll
42:31watch for issues with uh bugs in the
42:34graphbot desktop app as an example. Uh
42:37and whenever you find that send it to my
42:40cursor project. So one of the really
42:42cool things about grabbot is it connects
42:43to cursor. So cursor has uh like I I
42:47just mentioned this new feature called
42:49projects. And a project is really a uh
42:52again like a you get a coordinator agent
42:55that's in the cloud. It has its own
42:57computer and all it really does is like
42:59it's a manager of agents. It's like your
43:01executive chef, right? Your your chief
43:03of staff. It doesn't do the work itself.
43:06It delegates and orchestrates and
43:08manages the work of other sub agents to
43:12you know that report to your chief your
43:16chief uh of staff. And it basically is
43:20responsible for driving the work forward
43:22and managing things and uh passing
43:25context to them.
43:27>> So if you get a sudden burst of issues,
43:28let's say you get 30 issues at once in
43:30one payload or something or very quickly
43:32the coordinator agent can figure it out
43:34and delegate.
43:34>> Yeah, exactly. It gets like uh you know
43:3630 the 30 or so payloads and spawns a
43:39sub agent or a single coordinator agent.
43:41It can actually do a bunch of different
43:42topologies of agents and it will sort of
43:46figure out the best way to uh you know
43:48efficiently distribute the tasks to your
43:52team of agents. Um so I use uh cursor
43:57projects a lot um and I also use grapot
44:01a lot and cursor projects are my inner
44:03loop and grabbot is my outer loop.
44:05Grabbot takes all the context, external
44:08context, gives it to the projects
44:11because it can actually just send
44:12messages to those projects, right? You
44:14don't even have to open cursor. You can
44:16just tell your grabbot, okay, create a
44:18project, right, for these series of
44:20tasks. They're all related, right? Maybe
44:23as an example, you know, you've had a uh
44:26a big burst of issues that are all about
44:28performance, right? Your app is slow uh
44:31and they're all connected, right? Maybe
44:34some of them even have a similar fix,
44:37right? But and you can certainly go off
44:39and just spawn one agent per task, but
44:42then you've lost that sort of thread
44:43between them, right? And and you may
44:46duplicate work or you may not really
44:48think about the higher level problem.
44:49You know, sometimes when you you you
44:50solve bugs, you know, it helps to have
44:53multiple bug reports that are are
44:55slightly different because it helps you
44:57really, you know, zoom out and see
44:58actually, you know, the problem when I
45:00looked at this one report, I thought the
45:01bug was here, but actually when when I
45:03see the other multitude of bugs is
45:05actually up here,
45:07>> right?
45:07>> Yeah. Got you. So that that's why you
45:09have so many agents in that loop then,
45:10right? Because it's not just you have um
45:13like you have a bug report comes in, you
45:15spawn a single agent to look at that bug
45:17report. that a that single agent will be
45:19duplicating work with other um other
45:21agents, right? Because if there are
45:23multiple bug reports coming in through
45:24the same thing, that can be duplicated
45:25work.
45:26>> Yeah.
45:26>> Fascinating.
45:28>> That's really fascinating. Okay. And so
45:30this just this endless series of
45:32triggers um coming from real users
45:34reporting real reports um builds up this
45:38sort of and accelerates the factory sort
45:40of adds more orders in. Other than bug
45:43reports, are there any other sources
45:44that you use for like um accelerating
45:48for pushing these PRs?
45:51>> Uh well, funnily enough, it's some of it
45:54comes from uh reading the code, too. So,
45:57I guess I have sort of uh well, so to
46:01clarify that, you know, the 2,500 PRs,
46:03they're not obviously like 2,500
46:06features, right? they are a lot of the
46:08work actually is spent on gardening like
46:12another term that I really love. Uh
46:15where
46:17so I guess this is more important when
46:19you have a big team of engineers human
46:21engineers that you work with where and
46:25also this goes back a little bit to what
46:26I was talking about with the
46:27environment. You know, setting up a
46:28really good environment that doesn't
46:29just help you and your agents, but
46:31everybody on your team, right? Think of
46:33a new hire who doesn't have a lot of
46:34context on all of your engineering
46:36practices joining your your team. And if
46:39you have a really good environment, they
46:41can be productive from day one, right?
46:43They can they don't have to like, you
46:44know, make open a bunch of lowquality
46:46PRs. They can start, you know, they can
46:49start just turning out really good code.
46:52Um,
46:54and uh, yeah, I think I I sort of lost
46:57my train of thought.
46:58>> I've got a I've got a followup, which is
47:00what's what are the mechanics of like
47:02triggering
47:03>> like how when when do you trigger a a
47:06agent
47:08to go and look at the code, right?
47:10Because some people might say, "Oh,
47:11let's just do that every hour or
47:13something or like on a chron job or
47:15what's
47:16>> Oh, yeah. Yeah. Yeah. Yeah. Uh I saw
47:19some of your recent tweets as well about
47:20like you know the some of the tweets
47:22you've been doing which are great for
47:24setting up your routines. Uh I have some
47:27routines like that as well. Um so uh one
47:32of them is uh
47:35like looking through just another simple
47:38example is you know React has a lot of
47:41foot guns. Um, so, uh, as as I'm sure
47:44you're aware. And so I have an agent
47:46that's just constantly looking for band
47:48patterns. And the interesting thing
47:51about that one is that I don't actually
47:52tell it to fix the issue first. I tell
47:54it to append it to a document. And then
47:57every couple of days I look at it and I
47:59see actually these are all the same
48:00thing, you know, and so that gives me,
48:03you know, you almost want like a buffer,
48:05a queue. Sometimes that's actually more
48:07effective than just spawning off a
48:09couple of like a lot of sub agents to
48:11fix every single thing because when you
48:13are in kind of pure execution mode and
48:17just trying to like you know f uh you
48:20know execute on the orders that are
48:21coming in very fast you sometimes miss
48:23the big picture. So sometimes having a
48:26buffer forces you to think about the big
48:28picture because you you you have these
48:31artifacts and things that you can look
48:33at as a human um and sort of use your
48:36own human judgment to or I guess you can
48:39use an agent to do that as well. But you
48:41give the agent and yourself a way to
48:44identify patterns, right, that you might
48:46otherwise miss if you're just only
48:49solving each bug at a time. And that's
48:52also really the benefit of having
48:53something like a chief of staff agent is
48:56uh it can see the forest right uh in
49:00addition to actually doing the
49:02execution.
49:04>> Fascinating. That's I mean my brain is
49:07exploding a bit there with the sort of
49:08chief of staff at the software factory.
49:10I might have to change some of the
49:12course that I'm filming next week.
49:13That's
49:15>> uh all right. Let's talk about let's
49:16talk about review, right? because this
49:18is the reply that you get, you know, is
49:20>> did you read did you taste all 2500 of
49:24those dishes as they swept past you?
49:27>> And I assume the answer is a variety is
49:30a version of no.
49:34>> Yeah, I think you you you don't want to
49:36be in a position where you're not
49:37tasting your food ever again. Uh but you
49:40also, you know, for scale, you cannot be
49:42tasting every single dish that comes out
49:44of your kitchen, especially if you have
49:46multiple restaurants. So it becomes more
49:48about sampling right and thinking about
49:50the processes in the same way that you
49:53know if I guess maybe this is where the
49:55the the factory analogy is a bit more
49:58apt is you know as a quality supervisor
50:01on a factory you you can't look at every
50:04single item you sample right you take
50:06you you you go in there every day and
50:09you look at the quality of the pull
50:11requests you look at the code that the
50:12agents are writing and you scrutinize it
50:16very rig rigorously and you think about
50:19all the inefficiencies, the bad patterns
50:21that the agents are doing and then you
50:24think about how to course correct the
50:26environment, right? Not not that single
50:28agent. Uh because if maybe if it if it
50:33was a one-off incident, it's fine. You
50:35know that maybe there's nothing to fix
50:36there. But if you actually notice that
50:39multiple agents are are having the same
50:41issue, right? They're taking the same
50:43shortcut. they're they're propagating
50:45the same workaround everywhere. Uh
50:48that's a sign that you should go off and
50:50think about how to uh amend your kitchen
50:53or your factory, right? Like thinking
50:55about your skills, your constraints,
50:57your lints, your type systems um and
51:02setting or adjusting it so that that
51:04problem doesn't happen again. And when
51:06you do that enough times, then you get
51:08to a place where the codebase is again
51:10like the environment is so constrained
51:13and so it guides you so well that you
51:16can just you can just step away, right?
51:19That's the dream. And I'll I'll
51:20definitely say it um it's very hard to
51:23get to this point. I don't want to sell
51:26this as like, you know, something that
51:27you can just do easily by using PAC.
51:30Like it takes a lot of time and effort
51:32to think about your code and where you
51:35see your agents failing and thinking
51:38very thoughtfully, intentionally and
51:40setting up guard rails and constraints
51:43so that they do the right thing by
51:45default.
51:47>> And you're not like if to go back to the
51:50software factory analogy, this isn't a
51:52dark factory, right? This the lights are
51:54on, right?
51:55>> It kind of is. Yeah, actually.
51:56>> Is it?
51:57>> Yeah. Well, it's dark in the sense that
51:59so um it's dark in the sense that well I
52:02think if my agents are merging their own
52:04pull requests it's sort of become dark
52:07where I go to sleep my agents now work
52:0924/7
52:11uh I have I have the equivalent of like
52:14more than 10 chiefs of staff right each
52:16working on a different area like for
52:18example I have one that's working on
52:20performance of the Grockbot desktop app
52:23I have one that's working on uh fixing
52:25bugs that users report I have one that's
52:28exploring rewriting it in a different
52:31language just for fun, you know, like
52:32what if what if, you know, just
52:34reimagining what what it would be if it
52:35was like a native app. It's just a toy.
52:38Um, but the idea is like yeah, I uh I
52:41when you spend the time setting up your
52:43environment, I've gotten to a point
52:45where I review the pull request after
52:47it's landed, right? I I tell my agents
52:50full autopilot is is something that you
52:52can do in in PAC and that will trigger
52:56off this very intense rigorous
52:58verification loop where it will spawn a
53:00bunch of verifier agents for every pull
53:03request and it will fuzz right fuzzing
53:06meaning that it will actually run the
53:07application. It's going to click around
53:09and try to use it like a real human.
53:11look for regressions, look for bugs in
53:13your implementation and um it will try
53:17to find issues with the thing and then
53:19it will fix it itself. It'll do that
53:21again and eventually get the PR to a
53:24state where it can land.
53:26Uh so it does it does it is quite token
53:29intensive. You can tune this of course.
53:32Uh so you know instead of like 10
53:34verifier agents you might do like one,
53:37right? Or you just tell the agent to
53:39verify it's done work. But yeah, the key
53:41thing is the verification part is really
53:44the key piece that gives me a lot of
53:46confidence
53:48that I guess verification plus the
53:49environment, right? It's these the
53:51combination of these two things that
53:52allow me to step away and say agents go
53:54off and merge your thing. I'll review it
53:56in the morning by looking at my commit
53:58history
53:59>> and if I see problems, I go and course
54:02correct,
54:02>> right? And I'll go and revert or modify,
54:06add new link rules and whatever. Um, so
54:10it does it does take time to get to that
54:12point, but once you get it, oh, it's so
54:14it feels so magical. Uh, I I I tell
54:17people like I'm sleeping so much better
54:19now because, you know, it took it the
54:22very first day I turned on the sort of
54:24dark factory was very scary because I
54:26was like, "Ooh, what if I call the SE,
54:28right? What if I break something
54:29overnight?"
54:30>> Uh, and it took a lot of it took a lot
54:34of uh bravery, I think, to do that, but
54:37>> somehow I did it. And yeah, now I'm in a
54:39place where my Asians are are merging
54:42their own code while I sleep.
54:43>> It sounds like
54:44>> I think it's dark in that sense.
54:46>> Yes, it's dark sometimes, right? You do
54:48sometimes.
54:48>> That's true. That's true.
54:49>> Because I think of a dark factory is
54:51like almost like if you take the
54:53original definition of Kapathy's vibe
54:55coding, right, which is the code almost
54:57doesn't exist. You forget that code
54:59might be a thing. I think your approach
55:01is totally different from that, which is
55:03that code and the environment is
55:06essential. And if the code in the
55:07environment are bad, then you will get
55:09bad outputs. Garbage in, garbage out. So
55:12I I I think this is a this is a
55:15different thing. It's like, you know,
55:18the I don't know, maybe there's a dimmer
55:20switch or something, right? Like, you
55:21know, some parts of dark, some parts
55:23were light. This is why the maybe the
55:24the kitchen is a better analogy
55:27>> the restaurant because you know even as
55:29a as a as a
55:32as a restaurant restaurant you still
55:35might go to your restaurants every now
55:37and then to take take a peek in taste
55:39the food right
55:40>> uh I think that's
55:41>> the idea of sampling instead of blocking
55:42I think is really important
55:44>> I think what would you say to people who
55:46are in I guess you're obviously in a
55:49pretty security conscious environment
55:50where you're working very security
55:52conscious
55:53>> mh Um maybe there are folks working in
55:57like um medical applications or law or
56:00finance or something. I think of the
56:03like some PRs are kind of like two-way
56:05doors which is you can merge it and then
56:08revert it, right? It's cheap back
56:09through. But there are some PRs that are
56:11one-way doors, right? That will cause
56:12data loss of some kind that will
56:16>> do something that can't be easily walked
56:18back. How do you deal with situations
56:21where most of your PRs, let's say, are
56:23one-way doors? Like, is this something
56:25you just wouldn't recommend or like what
56:27do you think?
56:28>> Yeah, I think that's a really good
56:30question. I think that
56:32it all comes back to me to the quality
56:34of the verification that you're able to
56:37um get out of your agent. And I think
56:41for domains where the work is
56:44verifiable,
56:46this is easier, right? and the the
56:48oneway doors become two-way doors in a
56:51sense.
56:53But I guess I don't know if you're
56:54working on something that is like
56:56extremely
56:57is very hard to verify programmatically
57:00then I think yeah you're definitely in a
57:02position where it's very hard to get to
57:04that point. Um so I do think like yeah
57:07verifiability of the domain is an
57:10important aspect to be able to do this.
57:13Um and software engineering is just one
57:15of those things where it's quite
57:16verifiable in in a lot of cases maybe
57:19not totally um you know like other
57:22domains like mathematics I think are
57:24another example of not all of it of
57:26course but some aspects of mathematics
57:30can be verifiable
57:32if you write a proof for example um and
57:35so yeah I think it's a great question
57:38that I don't really have the answer to
57:40and I think that this is something the
57:41industry and us as engineers will have
57:43to figure out is you know my sort of uh
57:48hope and prediction for the future is
57:49that we'll see more and more interesting
57:52new agentoriented programming languages
57:56and one of the most fascinating ones
57:57that I've seen so far is this one called
57:59bend bend d bend um and that language is
58:04one where it kind of marries programming
58:08with proofs right there used to be a
58:12time, you know, where you actually had
58:14to write your proofs in a different
58:16language. And proofs, by the way, for
58:18those uh who who aren't familiar is this
58:20idea of uh that you can sort of formally
58:24verify that some code is correct
58:26mathematically, right? Especially if
58:29you've written your code in a very
58:31functional programming way.
58:34uh but for the longest time you had to
58:36do that in a separate language like lean
58:38or tla+ or uh I'm blanking on some of
58:42the other other examples but uh like
58:45languages like that where you would
58:46construct the mathematical proof and
58:48then use a solver essentially to det
58:52that that you've covered all the cases
58:54you don't have like a race condition or
58:56or whatever.
58:58So yeah, I think
59:02trying to sum up the question, I think
59:03yeah, if you are in a position where you
59:05can figure out how your agents can truly
59:07verify the work in a way that gives you
59:10confidence, you can actually, you know,
59:13uh have the PRs merge cuz if it
59:16compiles, right, if it if the proofs
59:18show you that it's correct, then why
59:21wouldn't you just merge it? Um but of
59:24course, yeah, not all the means are
59:26verifiable. Yeah, it's a tough one. Um,
59:31okay. I think we've got to think about
59:34wrapping up because we are nearly on the
59:35hour. Have you Have you got something
59:36after this? I mean, I've got something
59:38before I give my son dinner, but
59:40>> I I can go a bit longer after you.
59:42>> Okay, let's let's go five minutes longer
59:43then. Um I think I just want to have one
59:46more question which is
59:49I think I want to ask how you see Pstack
59:55and how you see skills in general like
59:57in terms of we talked about this before
1:00:00we went on air which is like people
1:00:02think of as like my skills versus your
1:00:05skills and how do you combine frameworks
1:00:08together? How do you use Pstack with my
1:00:11stuff? like what should you take from
1:00:13each one and because I think I see
1:00:16skills as sort of just derived from
1:00:19process basically like they're just
1:00:21processes turned into words and I would
1:00:25love to know
1:00:27how you recommend people take Pstack and
1:00:29take my stuff as well and turn it into
1:00:31their own processes.
1:00:35I think you shared a tip actually today
1:00:36that I thought was actually very
1:00:38relevant, which is this idea that you go
1:00:40off and look at your previous
1:00:42transcripts, right? And you sort of mine
1:00:44for information of, you know, your own
1:00:48look through your own your own prompts,
1:00:50right, to the agents where you correct
1:00:51them where you have to constantly
1:00:53intervene and uh you know take that
1:00:57higher level learning and turn that into
1:00:59a a reusable skill, right? So that
1:01:00agents stop repeating that mistake. I
1:01:04think that uh PAC and your skills are
1:01:08very complimementaryary. I I totally
1:01:10agree with you that they're like a skill
1:01:12is really much just process. I mean it's
1:01:14just at the end of the day a skill is
1:01:15just English or or language. It's just
1:01:18markdown.
1:01:19>> Um and I think you can you can
1:01:21definitely weave them, combine them in a
1:01:24way that makes sense to you. But I do
1:01:27think that uh everyone should have their
1:01:30own set of knives, right?
1:01:33I I keep going back to the the the
1:01:35cooking analogy, but it's so apt because
1:01:37like, you know, every chef when they go
1:01:39to a different job, right, when they go
1:01:41to a different restaurant, they carry
1:01:42they bring their knives with them. The
1:01:44tools go with them, right? And so trust
1:01:46to me is really about trust in your own
1:01:48tools. And when you spend the time
1:01:51sharpening them and understanding them
1:01:52really, really well, you can do great
1:01:54things. And everybody's skills and tool
1:01:56set is going to look different. you
1:01:58know, someone might find a lot of
1:02:00success combining, you know, like your
1:02:03grill me with docs, uh, or wayfinder
1:02:05skill with some of the execution skills
1:02:07in Pstack as an example. Some people
1:02:10might use more of your skills, some
1:02:11people might use more of my skills. I
1:02:13think at the end end of the day, it
1:02:15really just comes back to how much do
1:02:17you trust, you know, me and Matt, right?
1:02:19Like if you if you trust us both, of
1:02:21course, use our skills, but I also
1:02:23encourage you to, you know, look at your
1:02:25own transcripts. Um um tell the agent to
1:02:29look through, you know, some of the all
1:02:31of the patterns that you've used, the
1:02:33the times you've had to intervene, you
1:02:36know, suggest turning them into lint
1:02:38rules or new skills, right? The the past
1:02:41chats I I often say is like a a treasure
1:02:44trove of context because that, you know,
1:02:46it's it's like the process materialized,
1:02:51right? Like it's the real process. It's
1:02:53not an abstract idea in your head. it's
1:02:55the actual thing right and you can
1:02:57actually see how it happened in practice
1:02:59and extract so much information from
1:03:01that and there's so much so that I
1:03:02actually turn I have a skill in pac
1:03:04called recall which is exactly that um
1:03:07where this was a pattern where
1:03:10you know I was working in I was working
1:03:12on a similar problem so specifically I
1:03:14was working on virtualization for the
1:03:17cursor application and there were a lot
1:03:18of bugs and so you know every time I
1:03:21started a new chat I was like ah this
1:03:22there's so much good context from the
1:03:24last one So, you know, I want to bring
1:03:25it over to the new chat. How do I do
1:03:27that? And that's where the transcript
1:03:29came, uh, you know, looking at the past
1:03:31transcript came about. And then recall
1:03:33was just a way for me to collapse
1:03:35collapse and compress that workflow into
1:03:37a skill. So that I didn't have to just
1:03:39say I didn't have to write a long essay
1:03:42every time. Go look at all these chats,
1:03:43right? And, you know, blah blah blah.
1:03:45So, I I largely think of skills,
1:03:47especially as agents get more capable as
1:03:50really encoding workflows. you know
1:03:52skills from last year were really more
1:03:54about like almost like implementation
1:03:56details like here are the exact script
1:03:59commands you know you should use right I
1:04:02think with the latest models you can
1:04:04just delete those parts and just really
1:04:06focus on the workflow right until it
1:04:09it's more the skill becomes more like a
1:04:11series of steps
1:04:13a series of your process uh and I think
1:04:16over time we'll see that skills get
1:04:18smaller and smaller you know more
1:04:20compact Um, and yeah, they're very
1:04:24compatible. Or you can, you know, if you
1:04:26want, why not read our skills, right,
1:04:29and com and combine them in your of your
1:04:31own, right? Combine Wfinder with potato
1:04:34mode and make your own custom mode,
1:04:36right? Like like skills are the the
1:04:38thing I love about skills that is are
1:04:40that they're so malleable. You can do
1:04:42anything you want. It's just language.
1:04:44>> Absolutely. There's nothing magical in
1:04:46them, right? They're just words. And
1:04:48>> Exactly. If if there is any magic in
1:04:49them, it's just the words chosen and the
1:04:52phrases used and the thinking that's
1:04:55been done to turn those like take
1:04:59abstract process and turn them into
1:05:01language. And once that thinking has
1:05:04been done, then it's just there. It's
1:05:05available. It's on the surface and you
1:05:06just nick it. Um Lauren, thank you so
1:05:10much. This has been glorious.
1:05:11>> Yeah, this has been super fun. I really
1:05:14enjoyed talking to you. Hope we can do
1:05:16it again. I'd love to do it again. I'd
1:05:19love to do it again. Absolutely. Um
1:05:20yeah, we'll check in in uh
1:05:22>> I don't know. Yeah, Monday. Let's do it.
1:05:25>> Yeah, let's do it. Part two.
1:05:28>> Well, thank you so much. I'm going to
1:05:29close the stream here. Laura and I will
1:05:31uh chat a little bit and stay here. But
1:05:33thank you guys so much for watching. The
1:05:34glorious.

About this transcript

This page contains the full transcript of LIVE: Poteto (creator of pstack) on shipping 1,000's of PR's a month at SpaceX by Matt Pocock, generated from the public captions YouTube serves with the video. The transcript has 12,075 words across 1,698 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.