LIVE: Poteto (creator of pstack) on shipping 1,000's of PR's a month at SpaceX — Transcript
Full transcript
- 0:00So, hello folks. I've got another treat
- 0:02for you today. Last time on this kind of
- 0:05podcasty thing, I suppose, we had Uncle
- 0:07Bob and we talked about software
- 0:09quality. We talked about agents. We
- 0:10talked about lots of cool stuff. Now, we
- 0:13have uh an incredible guest, someone who
- 0:15I'm delighted to welcome on, who's been
- 0:18exploding on Twitter recently about
- 0:21software factories, um about increasing
- 0:24the quality of your work, about
- 0:26increasing your velocity and climbing
- 0:28the trust ladder with agents so that you
- 0:30can ship more and more and more. And it
- 0:32is potato. Welcome. Thank you so much
- 0:35for joining.
- 0:36>> Thanks for having me. Yeah, very excited
- 0:38to be here. Yeah, big fan of yours
- 0:41>> and a huge fan of yours. I [laughter]
- 0:42think people have been talking about
- 0:44this like it's like the meeting of the
- 0:45skill minds, the skill Mount Olympus or
- 0:49something because both of us have very
- 0:50popular skill libraries. Um I've not, as
- 0:52I was saying before we started, I've not
- 0:54used a ton of yours and like I want to
- 0:57get all of the juice out of your brain
- 0:59so that I can go and use it properly and
- 1:01use it better. And I think where I want
- 1:04to start with this is you gave a talk um
- 1:06pretty recently like um about 10 days
- 1:08ago and posted on X which went
- 1:10absolutely nuts as about how I shipped
- 1:132,500 PRs last month to production got
- 1:16about 3 million views or something on X
- 1:19and I watched it and I loved it and I
- 1:21recommended it and I kind of want to run
- 1:24this as almost like a Q&A of that talk
- 1:26basically of giving you because it just
- 1:29I just had tons of questions about it
- 1:31and I wanted to dive into it. And I
- 1:33think where I want to start is you talk
- 1:36about a trust ladder with agents where
- 1:39you as you trust agents more, you can
- 1:42get them to do better and better things
- 1:44and or scale them to up to use more and
- 1:47more agents. So what is your story of
- 1:50how you climbed the trust ladder and how
- 1:53did that work when like you got SpaceX
- 1:55and
- 1:56>> started climbing more and more?
- 1:58So I think this the the journey sort of
- 2:00began even before I joined cursor uh
- 2:03which is now SpaceX AI. Uh [snorts] so
- 2:06the story is um
- 2:09after Meta so I I used to work at Meta
- 2:12on the React team. [snorts] Uh I took a
- 2:14month off uh because I was feeling kind
- 2:16of burnt out and of course when what
- 2:19what do you do when you're burnt out?
- 2:20You go and start a new side project. Um
- 2:22and so I started a side project. you
- 2:24know, I was uh of course using AI to to
- 2:27write code. Uh [snorts] but then I
- 2:29started to realize uh you know, I was
- 2:31spending like so many hours just
- 2:33micromanaging one agent, right? And you
- 2:36know, at the time, this was back in
- 2:38February, maybe February, early February
- 2:41or January, you know, people were really
- 2:43obsessed with this idea of like
- 2:44orchestration. This was like, you know,
- 2:46before, you know, things like cursor,
- 2:48you know, like the agents window was had
- 2:50become popular. So people were still in
- 2:53like like 2 land you know in their
- 2:55terminal and they were all talking about
- 2:56okay here you know I built a custom
- 2:58orchestrator right and so of course I
- 3:01had I was a bit nerd sniped by that
- 3:03[snorts] and you know as I was building
- 3:04my toy project uh I got nerd sniped by
- 3:08oh how do I make my AI coding setup more
- 3:10efficient and so you know I I kind of
- 3:13started the journey there where I just
- 3:16you know took a step back and realized
- 3:18you know I was spending all this time
- 3:20micromanaging a single agent you know I
- 3:23was [snorts] creating skills and I was
- 3:24like finding it quite difficult to
- 3:26measure the output or the the result the
- 3:29impact of the skill as well so I was
- 3:32kind of flying blind but I was you know
- 3:34iterating really fast um and um so that
- 3:39project eventually sort of became the
- 3:42basis of PAC even though I didn't know
- 3:44it at the time um and a lot of some
- 3:48tricks I had learned like building that
- 3:50early set of skills. Actually, it's
- 3:52still open source if you want to if
- 3:54anybody wants to take a look. It's on my
- 3:56GitHub like potato
- 3:58noodle n o d l e. Um, and in there you
- 4:02will see some skills and a brain
- 4:04directory. [snorts] And so I was really
- 4:06interested in this idea of how do I, you
- 4:09know, extract my own ability, if that
- 4:12makes sense, and give it to the agent,
- 4:14right? cuz I was I I realized that you
- 4:16know all I was trying to do was trying
- 4:18to teach the agent to write code more
- 4:19like me you know do do you do do
- 4:22workflows more like me. So you know the
- 4:24skills were like an entry point to doing
- 4:27that.
- 4:28Um and then you know after I joined
- 4:31cursor uh I was starting to work on the
- 4:33agents window and uh it had a lot of
- 4:36performance issues. Uh it was it was it
- 4:39was pretty laggy. Uh and so since I had
- 4:41experience working in React, I was asked
- 4:43like, "Hey, do you want to come and help
- 4:45out uh with the agents window?" Um and
- 4:48so the the this beginning of the cursor
- 4:52journey was very manual. [snorts] Uh I
- 4:55was deep in like looking at like flame
- 4:58graphs and heap snapshots and trying to
- 5:00see like why exactly is the app so slow.
- 5:03Uh but then coming back to the same
- 5:05realization like you know I was sort of
- 5:07the bottleneck. I was doing everything
- 5:09manually. I was sort of the meat proxy
- 5:11in a way, right? I was the meat proxy
- 5:13between my agent and Chrome DevTools. Uh
- 5:16and I was like really annoyed by that.
- 5:18>> And what month of the year is that?
- 5:20Let's say where are we in the timeline?
- 5:22>> Uh so I joined Cursor in March. So this
- 5:25was like early early April probably
- 5:28early April is when [snorts] you know uh
- 5:30I joined and I didn't have any skills,
- 5:32right? I had I I sort of abandoned my
- 5:35personal skills because I didn't think
- 5:36they'd be relevant anymore. Uh but then
- 5:39working on the agents window uh and
- 5:41[snorts] now working on grockbot uh I
- 5:44sort of realized that a lot of the
- 5:46lessons I had learned from those skill
- 5:48time building the the initial set of
- 5:50skills were very relevant especially
- 5:52around things like verification
- 5:55uh you know being very rigorous in your
- 5:57work um [clears throat] because I think
- 6:01from my experience even the the frontier
- 6:04ones tend to
- 6:07tend to take shortcuts. Uh they tend to
- 6:10do the easy thing. Uh so uh a lot of the
- 6:14skills that I've built have been around
- 6:16how do I make the easy thing the right
- 6:19thing? You know, how do I make that the
- 6:20best thing?
- 6:22>> The idea of sort of distilling your
- 6:25expertise and turning what you do every
- 6:28day into processes, that's something
- 6:30that feels super familiar to me. That's
- 6:32exactly what I've been doing with the
- 6:34skills. And I suppose there's something
- 6:37in that which is a lot of people think
- 6:39domain expertise is getting less useful
- 6:41now as people uh start to rely more on
- 6:44AI where what do you think about that
- 6:47just as a sort of vibe check before we
- 6:49start talking [laughter]
- 6:51>> I actually feel like domain expertise is
- 6:52is more important than ever you know uh
- 6:56I think I wrote this on my ex at some
- 6:58point but you know at times I sometimes
- 7:01think of you know AI as is like
- 7:05especially as the models get smarter and
- 7:06smarter and more capable and the
- 7:08frontier models are just getting so good
- 7:10like I [snorts] love Opus 5.5 by the way
- 7:13um uh you know as the models get really
- 7:17really really good it almost becomes
- 7:19like the bottleneck is no longer the
- 7:21agent right it becomes your ability to
- 7:24express your intent and your goals in a
- 7:27clear way that [snorts] the agent can
- 7:29understand and actually carry out and
- 7:33That's why I think you know like people
- 7:34with a lot of domain expertise are
- 7:37extremely have a have a huge advantage
- 7:39in my opinion especially if you're a
- 7:42little bit like you know tech technoc
- 7:44curious you know so I I think of people
- 7:46like you know like uh like a doctor or a
- 7:49lawyer or you know someone who who has a
- 7:52deep expertise in a particular
- 7:54non-engineering domain and [snorts] if
- 7:56they're actually just a little bit
- 7:57techsavvy and they can figure out how to
- 7:59use agents they can actually build
- 8:01really really great products, right? If
- 8:04they if they have a clear enough vision
- 8:06in their head and they can articulate it
- 8:09in a way that the agent can build it,
- 8:11you know, I think that that is really
- 8:14the the bottleneck these days is is like
- 8:17the transfer of your intent, right, and
- 8:20your vision to the agent.
- 8:23>> Yeah. I've been obsessed with language
- 8:25basically since agents um dropped. are
- 8:27just obsessed 100% and thinking
- 8:30constantly about the the composition of
- 8:32words, how I can make things sharper,
- 8:35what um what might be hidden in the
- 8:37phrases that I'm using. And it's and
- 8:40finding what I love is when you find a
- 8:42word that the agent then hooks on to and
- 8:45then goes, "Okay, I'm going to reinforce
- 8:47that word. I'm going to reuse that in my
- 8:49thinking traces." You know, I found that
- 8:51with um TDD was an early example of
- 8:53that. a lot of chat about TDD recently
- 8:55of like, you know, people say, should
- 8:57you use TDD with agents? Doesn't matter.
- 8:59What you're doing is you're getting the
- 9:00agent to think about TDD, getting it to
- 9:03write tests, getting it to prioritize
- 9:04things in a different way than it did
- 9:06before. And that's why sort of grilling,
- 9:08I think, works effectively. Grilling is
- 9:10a
- 9:11>> Yeah. Yeah. It it draws those words out
- 9:13of you, right? or or at least it helps
- 9:16the agent understand your thinking so
- 9:19that they can propose those words to you
- 9:20and you can pick up and say yes exactly
- 9:22that.
- 9:23>> Uh I've actually copied some of the the
- 9:25tips that you've shared as well where
- 9:27you know one of my favorite ones that
- 9:28you've shared recently or or not or like
- 9:30maybe in the past couple weeks is
- 9:32[snorts] about uh reducing or
- 9:34eliminating tautological tests. Like one
- 9:37of my pet peeves of agents is like all
- 9:39of the useless tests that they write.
- 9:41And so, you know, that was one thing
- 9:44where, you know, the word tutology,
- 9:45right, is is is I guess, you know, not
- 9:47many people necessarily know that if if
- 9:49especially if English isn't your first
- 9:51language, but there's a lot of meaning
- 9:53to that word. And it's like it's almost
- 9:56like compressed, right? Like you
- 9:58compress a lot of intent and meaning
- 10:00into words. And so I I I totally agree
- 10:04with you. I think language I've always
- 10:06been interested in language actually uh
- 10:08like programming languages natural human
- 10:11languages and how they came to be and
- 10:13it's so interesting that now with agents
- 10:15it's sort of like this meeting of
- 10:18natural language with programming
- 10:20language but it's all it's all language
- 10:21out of the hood it's all communication
- 10:23>> totally I did a drama degree right so
- 10:25you know I've been thinking about
- 10:26language and Shakespeare and stuff for a
- 10:28long time [laughter] and so this all
- 10:29feels very familiar
- 10:31>> um so okay there's sort
- 10:34before we get into like because I think
- 10:36the thing I want from you is like
- 10:38software factory stuff, right? Software
- 10:40factory is the big buzzword. Software
- 10:42factory is the thing that I'm thinking
- 10:43about too. I'm sort of releasing a
- 10:45course in that direction too.
- 10:47>> And it's this sort of scaling yourself
- 10:50up to un unrealistic numbers of PRs
- 10:53basically or PR numbers that sound
- 10:56ridiculous to people who don't
- 10:57understand how this works. So where I
- 10:59want to get to is sort of from people
- 11:01who are doing kind of like one to five
- 11:03agents today up to, you know, hundreds
- 11:06of agents running at once and how that
- 11:08sort of functions. And so I'd love to
- 11:10hear about your metaphor of the Michelin
- 11:12Kitchen instead of the software factory
- 11:15because I think that says a bit about
- 11:16the way you think about this stuff.
- 11:18>> Yes.
- 11:20Yeah. I I I've I've never really liked
- 11:22the term software factory. Not because
- 11:25you know it's not accurate but I think I
- 11:28think the a lot of people when they
- 11:29think factory right they don't
- 11:31necessarily equate that with quality or
- 11:34craft right things which are very
- 11:36important to me and a lot of people and
- 11:39technologists who work you know building
- 11:41products we care about the user
- 11:44experience we care about the things
- 11:45we're building. So while so while I do
- 11:48think software factory is an apt term,
- 11:51it also I guess maybe conjures up
- 11:53negative, you know, maybe sometimes
- 11:55negative connotations. So Michelin
- 11:58Kitchen is the thing that I've sort of
- 12:00landed on where it's much more I feel
- 12:02like it's much more aspirational and uh
- 12:05I like the metaphor a lot cuz you know I
- 12:06like food. I'm called potato of course
- 12:09[snorts] and I like cooking [laughter]
- 12:10and I see a lot of parallels right like
- 12:13with food right when you're cooking a
- 12:15meal for yourself for example
- 12:17[clears throat] it's both utilitarian
- 12:19like you're trying to just feed yourself
- 12:20right and and survive uh but it can
- 12:23actually be transformed into art right
- 12:25and that's what what a Michelin starred
- 12:28chef or even just a chef or a cook can
- 12:31do with food is take something very
- 12:33ordinary and turn it into a delicious
- 12:36meal that you know takes you back to
- 12:38your childhood days or something like
- 12:39that. Um and so it almost like mirrors
- 12:43that trust letter that I talk about
- 12:45where uh you can sort of imagine your
- 12:48own journey as a home cook, right? Uh as
- 12:50a home cook, you are doing all of the
- 12:52food, the cooking yourself. You cut all
- 12:55the vegetables, you do all the prep
- 12:57work, you do all the cleanup, you know,
- 12:59you are the one man or one woman show
- 13:03really. Um, and it's an interesting
- 13:07thought experiment like, okay, if you
- 13:09were to cook a meal and then you add
- 13:11people, right, your your your partner
- 13:14trying to your brother, your sister, and
- 13:16now suddenly you have your whole family
- 13:18in the kitchen. I think most people
- 13:19would get very stressed by that, right?
- 13:21The thought of, oh, so many people are
- 13:23just mocking around in my kitchen. They
- 13:24have no no idea where all the utensils
- 13:26are.
- 13:27>> I have a max capacity of one person in
- 13:28the kitchen. Yeah, absolutely. So, I
- 13:31feel like that that's really apt because
- 13:33when you ask yourself that question of
- 13:35how do I go from being a solo cook,
- 13:37right, to having an army or even not not
- 13:41even an army but a few sue chefs, right,
- 13:44that that are helping me in the kitchen.
- 13:45How do I think about dividing the work
- 13:48in a way that makes sense? You know, I'm
- 13:50not dividing work just for the sake of
- 13:52it, but in a way that actually makes the
- 13:55sum the to the better than, you know,
- 13:57the total of its parts. And so the
- 14:00Michelin kitchen metaphor to me like
- 14:02works really well in that regard because
- 14:05you know as a chef you're you know if
- 14:08you become a chef you're in a position
- 14:10where you're not necessarily cooking all
- 14:12the food yourself anymore but you are
- 14:14thinking you're almost like the tech
- 14:17lead right for the kitchen where uh you
- 14:20know chefs have to think about you know
- 14:21not just cooking but they have to
- 14:23basically organize the whole kitchen and
- 14:25they're like the CEO of the kitchen they
- 14:27have to think about when do you order
- 14:29ingredients, how do you store them, how
- 14:30do you prepare them, when do they have
- 14:32to be prepared, you know, it's a whole
- 14:34it's a whole job, right? That's not just
- 14:36cooking. Um, and I think that again it
- 14:39mirrors so much of how engineers write
- 14:42code today where you are not writing the
- 14:45code yourself anymore. You have agents,
- 14:47right? But you as the human are still
- 14:49responsible for the final outcome,
- 14:51right? your name still is associated
- 14:53with the work that you do, your
- 14:55reputation and you know so how you set
- 14:58up your kitchen right and how you set up
- 15:01your skills your environment your
- 15:03codebase I think are ultimately the new
- 15:07ingredients that go into um building
- 15:10product
- 15:12>> yeah I think what I love about your
- 15:14approach is the amount of focus that you
- 15:16put into the environment that the agent
- 15:18operates in right because I think a lot
- 15:20of people they think, right, the agent
- 15:22is good. I'm probably not going to be
- 15:25able to make it better. Let's just trust
- 15:27what these magic model people have put
- 15:29into the harness and the model
- 15:31combination. Uh, there's nothing I can
- 15:33really do, like I can't mess about with
- 15:35claw codes internals or something or
- 15:36whatever you're using. Um, but what I
- 15:40love about your approach, and it's
- 15:42something I advocate for too, is that
- 15:43you can change the environment the agent
- 15:45operates in, right? you can make changes
- 15:48in the codebase and also give it tools
- 15:51for verification as well and allow it to
- 15:54verify its own work. So the thing I I
- 15:56loved about watching that talk is the
- 15:58amount of focus you put in verification
- 16:01and like that is the lever that you can
- 16:03start to generate trust. Can you talk
- 16:05about that and what that concretely
- 16:07looks like? Let's start like looking at
- 16:09practical ways that people can improve
- 16:10their own processes, their own kitchens.
- 16:14Yeah, I've I've I've said this a lot
- 16:16actually that you know even if you don't
- 16:18use PAC or you know your skills I think
- 16:21that the single most important skill
- 16:24that should be in your toolkit is
- 16:27verification because without
- 16:29verification and for for by the way for
- 16:32those watching who don't know what that
- 16:33means it's this idea that you can give
- 16:35you can sort of give your agent uh hands
- 16:38and eyes in a way that's the the analogy
- 16:42I
- 16:43where the agent is able to run the code,
- 16:46right? And actually
- 16:48>> [snorts]
- 16:48>> uh interact with it like a normal human
- 16:50user would and also do things like you
- 16:53know debug it, you know, take traces and
- 16:56snapshots. Um and uh funnily enough like
- 17:00that was actually the first skill I
- 17:01built when I joined Cursor. uh that gave
- 17:03me a lot of that was that was the thing
- 17:05that actually started to let me ascend
- 17:07the trust ladder a little bit in a way
- 17:09that some of the other skills I had
- 17:11looked at or built had not really let me
- 17:14do because no matter how good you know
- 17:17some of the other skills were like the
- 17:19how skill, the why skill, the unsop
- 17:21skill were, I was still relying on me
- 17:24right as the proxy between my agent and
- 17:27the output. So that you know if the
- 17:30agent can't actually see the result of
- 17:31its work there's no way it can actually
- 17:34iterate right and so this is where
- 17:36people start to talk about loops this
- 17:38idea of a loop and really I think the
- 17:40term loop you know seems kind of uh
- 17:43almost abstract like people like what
- 17:46what what is a loop what is an agent
- 17:47loop but really to me like the most
- 17:50important part of a loop that allows it
- 17:52to be a loop is the verification part
- 17:55because the agent is able to to verify
- 17:58by its own work and uh you know that
- 18:01takes you out of the equation where now
- 18:04I can actually do something like so the
- 18:06very one of the very first use cases I
- 18:07had for verification was you know like
- 18:09the performance work that I was doing on
- 18:11cursors agent window and I want I wanted
- 18:14to get to a point where I could do
- 18:15something called hill climbing uh which
- 18:17is a term that I I think the labs uh
- 18:20talk about a lot which is this idea that
- 18:22you know you have some kind of rubric or
- 18:25a way to judge or score something And
- 18:28now because you have a loop, you can
- 18:30have an agent continually try to make
- 18:33improvements to that. Uh I think
- 18:35Carpathy, Andre Carpathy also famously
- 18:38released uh something called auto
- 18:40research that has a lot of these ideas.
- 18:43Um but yeah, verification I would say is
- 18:46probably the most important skill in PAC
- 18:49uh and many [clears throat] other you
- 18:51know tool sets. Uh and I think it's the
- 18:55most important thing to focus on. So a
- 18:57lot of the a lot of I spent a lot of
- 19:00time actually you know tuning the
- 19:02verification the creative verification
- 19:05skill um and also internally the the we
- 19:09have so many verification skills now
- 19:10like every app that cursor has or
- 19:12spaceexai has has a uh verification
- 19:16skill that is automaintained as well
- 19:19[clears throat]
- 19:19>> uh and it's become critical
- 19:21infrastructure for our team because
- 19:22everybody uses it
- 19:24>> and you went pretty far with that too
- 19:26right like you had a um in your talk I
- 19:28saw that you actually built a custom CLI
- 19:30for that too. So what does that CLI do?
- 19:32Like how does it [clears throat] execute
- 19:33things and why did you I mean that's
- 19:35proof of how deep you're going right of
- 19:37how much you're pushing that.
- 19:39>> Yeah. So this is actually a tip I
- 19:42learned early on where
- 19:45um I guess you know back in January or
- 19:49or late last year the thing that people
- 19:51were concerned about was context window,
- 19:54right? That was the big the big topic at
- 19:56the time was how do I you know manage
- 19:58the context window because you know
- 20:00compaction summarization wasn't really
- 20:02that good yet [snorts] and people were
- 20:05always people had this there was almost
- 20:07this meme in the community that you know
- 20:09once your agent summarized or compacted
- 20:12once it would become sort of stupid
- 20:15right for the rest of your session. So
- 20:17there was a lot of thinking around like
- 20:20you know being very efficient with your
- 20:22context usage and so that was actually
- 20:25the inspiration for some of the uh the
- 20:28CLI work inside of the verification
- 20:31skills. I guess now it's less so about
- 20:34context because uh you know agents are
- 20:38much better or harnesses have gotten a
- 20:40lot better with summarization.
- 20:42Um, I still think there's some benefits
- 20:45to, you know, uh, having a clean context
- 20:49window. Uh, so the CLI is really just
- 20:51more of a way for me to take the
- 20:53deterministic parts of what the skill
- 20:56does and encode that into a script or
- 20:58CLI to reduce to kind of take away the
- 21:03judgment that would otherwise
- 21:05unnecessarily be used because with
- 21:08judgment so I also think of you know
- 21:10agents and skills in sort of like it's
- 21:13like a gradient
- 21:15you have some parts of the work that are
- 21:18entirely judge measurement based right
- 21:20you know something that requires thought
- 21:22you know putting together multiple
- 21:24pieces of context thinking um and then
- 21:27you have the more deterministic parts
- 21:28like I don't know if you wanted to uh
- 21:31refactor some code right from one
- 21:34pattern to another that's very
- 21:35mechanical right you don't you don't
- 21:37need an agent to think about it and come
- 21:40up with it in a novel way each time
- 21:43right and so that that was really the
- 21:45inspiration for the CLI and you'll see
- 21:47this in a lot of the other skills that I
- 21:48built is like I try to extract out the
- 21:52deterministic parts and turn that into
- 21:54code and just leave only the parts that
- 21:56actually require judgment to the agent.
- 22:00So in a way I think of the seal as kind
- 22:01of like your wrapper, right? It's a
- 22:03wrapper with some light instructions
- 22:04around how to use these custom tools
- 22:06that are inside of the skill. Um but
- 22:10yeah, I don't think the CL is really
- 22:11that interesting in its own really. It's
- 22:13not like a novel piece of software. It's
- 22:16just something that interacts with like
- 22:17Playright and the Chrome DevTools
- 22:20protocol and calls a bunch of APIs. It's
- 22:22like it's just a bunch of glue.
- 22:24>> No, it's fascinating because it's a way
- 22:25of hiding information from the skill,
- 22:27right? It's a way of conserving the
- 22:28skill, keeping the skill quite small, I
- 22:31imagine, and then you're able to
- 22:32delegate more of the complicated
- 22:33deterministic stuff into a script within
- 22:36the skill. So it's almost you're
- 22:38compressing information and making the
- 22:41agent do more consistent things more
- 22:44consistently.
- 22:45>> Yeah,
- 22:45>> that's fascinating.
- 22:47And it it [clears throat] also helps I
- 22:49guess if you care about context window
- 22:50it it does help because now the agent
- 22:53doesn't need to uh you know re reinvent
- 22:56uh things cuz uh one thing I had noticed
- 22:59early on when we didn't have a CLI
- 23:02was that uh well the the agent would try
- 23:05to verify it work but it would basically
- 23:07rebuild the world each time and then
- 23:09every agent did it differently and I was
- 23:11starting to notice like that's very
- 23:12inefficient right I was wasting it it
- 23:14was actually not just about context
- 23:16usage but also speed, right? Like
- 23:18because now an agent had to actually go
- 23:19off and write the scripts or the CLI and
- 23:22test it and you know and it doesn't work
- 23:24and the last agent did it and it worked
- 23:26but it discarded it. So it was just very
- 23:28obvious at that point like I should just
- 23:30turn this into a CLI and put that inside
- 23:32of the skill uh so that every agent that
- 23:35uses it now benefits from that same
- 23:38piece. Um but I I also think like you
- 23:42know it's a good push for people to
- 23:43think about is how much of your skills
- 23:46and rules could actually be
- 23:48deterministic.
- 23:50Um that's like another core thing or or
- 23:53one of my core principles that I like to
- 23:55think about is yeah how do I uh make
- 23:59very efficient use of determinism and
- 24:02non-determinism
- 24:04and you know let Asians shine at the
- 24:07non-deterministic parts right because
- 24:09that's what they're trained to do. Um
- 24:11and the other parts which are much more
- 24:14mechanical or you know straightforward
- 24:16can be just pure determinism. Um, and
- 24:19you'll see this as well for things like
- 24:21doing migrations. Um, which is another
- 24:24big thing that I've I've talked about
- 24:26is, you know, going from one technology
- 24:29to another, especially one that is
- 24:32better for agents, right? And a lot of
- 24:34how you can do that migration is, I
- 24:36think, through things like scripts and
- 24:38CLIs, like the deterministic parts like
- 24:40code mods, you know, like crawling the
- 24:44abstract syntax tree and transforming
- 24:46code literally mechanically, right? like
- 24:48a script does it for you instead of the
- 24:50agent.
- 24:52>> Totally makes sense. I I mean I think
- 24:55what there's another thing there which
- 24:57is you're taking stuff away from the
- 25:00agent and you're kind of putting it in
- 25:03the environment too a little bit which
- 25:05is let's say you have a a thing that you
- 25:08notice the agent always gets wrong. You
- 25:10want to make that um just impossible
- 25:13within the environment. And that sort of
- 25:15comes down to code quality as well. I
- 25:18mean, I talk about a lot like having a
- 25:22what a good codebase means, right? What
- 25:24is a good codebase? And there's a
- 25:26definition I like which is a a good
- 25:28codebase is a codebase that's easy to
- 25:30make changes in, right? Easy to um
- 25:34change stuff without things screwing up.
- 25:36And that means that you have a lot of
- 25:37guard rails that you have a lot of um
- 25:40the agent or the human is constrained to
- 25:42very narrow paths. And that's again
- 25:44something you talk about in your talk.
- 25:46>> And you talk about this not only on the
- 25:48kind of sort of automated checks side of
- 25:52things. So linting and type checking
- 25:53blah blah blah but also in the way you
- 25:55design abstractions. And you guys even I
- 25:58think built a framework uh for your
- 25:59agent to work in too.
- 26:01>> I think what I'd love to hear is you
- 26:04obviously think of that as very
- 26:06important, right? And that's how
- 26:08important is that compared to other
- 26:10things you could be doing like building
- 26:11features or shipping work.
- 26:14Yeah, I think that's a um I almost feel
- 26:17like the new job of the engineer is
- 26:19really to to spend time on the
- 26:21environment. Um I almost actually wrote
- 26:23a tweet about this yesterday, but I but
- 26:25I didn't. But I think that
- 26:29I think that you you know if you if you
- 26:31haven't really spent time, you know,
- 26:33building trust in your agents and
- 26:34building skills and tools, you can get
- 26:37stuck in this mode where you're very low
- 26:39on that trust ladder, right? you don't
- 26:41have a lot of trust in your agents work.
- 26:43And so the only way to cope in that when
- 26:47you're in that situation is just to kind
- 26:48of lock in and micromanage your agents.
- 26:51And that's very time consuming. And when
- 26:53you're stuck in that mode, you don't
- 26:55really have the luxury to think about,
- 26:58you know,
- 27:00uh higher level things like like making
- 27:03yourself more productive. In the same
- 27:05way that uh I guess analogy would be
- 27:09like if you've never taken the time to
- 27:11learn like your tools right as a
- 27:13developer when you were writing code
- 27:14yourself and you know you've never heard
- 27:17of VS Code, you've never heard of Vim,
- 27:19you only knew about Notepad [laughter]
- 27:22uh and you had hadn't even heard about
- 27:23Git. That's sort of the analogy. It's
- 27:25like you you haven't spent the time
- 27:27sharpening your own knives, right? And
- 27:29so, of course, if you have a dull knife,
- 27:32then everything's going to take a long
- 27:33time. Um, and you're going to be you're
- 27:36just going to be and and especially if
- 27:38you know deadlines are looming, then you
- 27:41don't have the now you're stuck in this
- 27:42rut, right? Where where you you you
- 27:45don't have sharp knives, you don't have
- 27:47good tools, but you're under all this
- 27:49pressure to ship, right? And so, all you
- 27:51can do is just focus on that. But I do
- 27:54think that, you know, if you can find
- 27:55yourself the time to actually spend time
- 27:58thinking about your setup, it's again
- 28:00going back to the cooking, you know,
- 28:02analogy,
- 28:04it's like uh, you know, if you, for
- 28:06example, if if cutting cutting cutting
- 28:09the garlic is like super slow, right?
- 28:11There are garlic mashers, right? You can
- 28:13you can buy and you put it in the thing
- 28:14and you like squeeze it out, right? It's
- 28:17super fast. Uh, machines and tools were
- 28:20invented for a reason, right? And so if
- 28:22you're operating a Michelin kitchen and
- 28:25your your your cooks had no tools, then
- 28:27of course everything's going to be
- 28:29extremely inefficient, very very, you
- 28:31know, every every every cook is going to
- 28:33make something up of their own. So I
- 28:35think the tools and the determinism to
- 28:37me are you know taking that part away
- 28:41and and just like you said about
- 28:42constraints as well. It's the
- 28:44constraints are are to me as well like
- 28:48uh actually a slight tangent on that is
- 28:50uh I think we should talk about
- 28:53TypeScript cuz like we we actually both
- 28:55share like a background in Typescript
- 28:57where you know you obviously have done a
- 28:58lot of work with TypeScript and total
- 29:00TypeScript and you know you're a leader
- 29:02in that space and I uh had adopted
- 29:05TypeScript pretty early and I had given
- 29:07like a talk or two at Typescript conf uh
- 29:10many years ago and so one of the the
- 29:13talk that I did actually was about type
- 29:15systems and constraining the
- 29:18constraining types. Like one of my most
- 29:20favorite things about Typescript is
- 29:22actually type narrowing, right? This
- 29:23idea that you go from a very broad type,
- 29:26right? That could be anything and then
- 29:28you through type guards and you know
- 29:31type narrowing and you know runtime
- 29:33checks you can actually narrow the space
- 29:35and say like oh this isn't just a string
- 29:38this is a very special type of string.
- 29:40It's a constant, right? like I but I I
- 29:42determine that through the type system
- 29:45and in a way it's like uh there's a lot
- 29:47of parallels I think to that with
- 29:49constraints in your codebase where is it
- 29:52you're you're constraining the space
- 29:55right if you if you think about category
- 29:57theory as well you know you're
- 29:58constraining the the number of possible
- 30:01types right that can can exist
- 30:05and you're saying there's only one type
- 30:07right and for for us like that framework
- 30:10that I'm called Dune. Uh it's not an
- 30:13open source framework. It's the the way
- 30:15I describe it to people. It's it's kind
- 30:17of like a internal Nex.js for our
- 30:20Electron apps. Uh but it comes with a
- 30:23lot of really really restrictive lit
- 30:25rules and the codebase is designed in a
- 30:28way that there's really only one way to
- 30:30do something. So we make use a we make
- 30:32use of a lot of conventional
- 30:35patterns. So like features all go into a
- 30:38specific directory. Well, every feature
- 30:41has its own directory. As an example,
- 30:43you know, there's like a a thing that
- 30:44discovers features like through a
- 30:46registry and like crawling the codebase
- 30:48and stuff like that. But this
- 30:51conventional pattern and the lint rules
- 30:55make for an environment where it's
- 30:57actually very hard to write bad code.
- 31:01And that sort of frees up the it both
- 31:04frees up your own mental uh you know
- 31:09capacity as well as the agent sort of
- 31:11doesn't have to think about that anymore
- 31:14where it's just like oh there's only
- 31:15there's I should just if I want to add a
- 31:17new feature it just goes in the feature
- 31:18the new feature directory and all the
- 31:20code goes in there and I'm not going to
- 31:22append to a god file right that was
- 31:25really actually the inspiration for
- 31:26those feature directories is the very
- 31:29first couple of versions of Grockbot
- 31:32were composed of like eight god files
- 31:36which were like at least 10,000 lines
- 31:38long if not longer and so I kind of had
- 31:40to break it up into smaller pieces.
- 31:43Uh but it was just observing you know
- 31:46actually that's another important part
- 31:47is observing how agents fail and then
- 31:50every time you see a mistake every time
- 31:52you see something that could be done
- 31:54better you think you step back and think
- 31:57how do I turn this into a lint rule? How
- 31:58do I make it so that the code base makes
- 32:01this impossible? Right? And it comes
- 32:03back to me for my you know my background
- 32:05learning Typescript and types uh type
- 32:08systems is how do I constrain the space
- 32:12so that you know I know precisely what
- 32:15I'm working with and I think yeah
- 32:16there's a lot of parallels there.
- 32:18>> Totally makes sense. And don't I mean
- 32:21it's funny that you mentioned TypeScript
- 32:22and Goth files in the same sentence
- 32:23because Typescript famously has a
- 32:2525,000line type uh file.
- 32:29>> [laughter]
- 32:30>> Although I don't know if they've
- 32:30rewritten that and go as they probably
- 32:32have, haven't they? Um,
- 32:34okay. So, environment is important. You
- 32:38should watch your agent like a hawk to
- 32:39make sure that any mistakes it makes.
- 32:42You turn them into things in the
- 32:44environment. And the benefit of the
- 32:45environment is you're not overloading
- 32:47your agent, right, in terms of rules, in
- 32:49terms of things it has to remember. It's
- 32:51just in the environment. And so it
- 32:53stumbles into the rules and exactly um
- 32:56you know bounces off them and hits them
- 32:57at the right moment.
- 32:59>> So okay, we still haven't talked about
- 33:01the 2,500 PRs. Where do those come from?
- 33:03Like how do you you've built your trust
- 33:06ladder, you've worked on your
- 33:07environment, and you understand, okay,
- 33:09um I now want to scale up. So what are
- 33:13the mechanics of that scaling? Are you
- 33:16um initiating 2500
- 33:19like chats per month? That can't be
- 33:21right. So there must be are there any
- 33:23kind of automated triggers that trigger
- 33:26stuff in your repo? Like how do you get
- 33:28the software factory kind of triggering
- 33:30work by itself?
- 33:32>> Right.
- 33:34Um I'll definitely say that the
- 33:36prerequisite to you know something like
- 33:40a very high volume of of pull requests
- 33:43um is the environment. you know, the the
- 33:46stuff we just talked about where I
- 33:48definitely would not have been able to
- 33:49do this if I had not spent the time, you
- 33:52know, thinking about the kitchen, right,
- 33:53and the knives and the tools for my
- 33:55Asians. And so, in a way, I I think of
- 33:58this as
- 34:00I've spent the time building one kitchen
- 34:02and one restaurant. And now I I'm in a
- 34:05position where I don't actually have to
- 34:07be there anymore because the
- 34:10environment, you know, that the same
- 34:12analogy, right? It works really well.
- 34:14[laughter] Yeah. You open chain of
- 34:15restaurants, right? That's
- 34:16>> Yeah. Exactly. Yeah. Exactly. It's like
- 34:19you're Gordon Ramsay and you know, you
- 34:21you've taught your executive chef like
- 34:23all the tricks of the of coming up with
- 34:25great menu. Uh and like the kitchen is
- 34:28set up really well. Everything's just
- 34:30perfect and you're now in a position
- 34:32where you can open your second your
- 34:33third restaurant. And [snorts]
- 34:36I guess I I sort of see each project
- 34:38that I work on, like each big chat is
- 34:41sort of like a restaurant, right? and
- 34:43and I'm I'm I have multiple of them
- 34:45operating at the same time and I'm sort
- 34:48of like helicoptering between them
- 34:50sometimes some more than others
- 34:52depending on how in the loop I am but
- 34:55yeah definitely I think there's there
- 34:56there are external triggers and context
- 35:00that those projects don't have that
- 35:05for a long time I was the proxy for that
- 35:08so uh the best example I have is like
- 35:10you know you have a project that's
- 35:11working on a feature
- 35:13uh or you're trying to fix a bug and
- 35:15you're getting bug reports, but the bug
- 35:17reports are going to things like Slack
- 35:19or linear or X, right? And these are
- 35:24external systems that aren't connected
- 35:26to your inner loop. So, I like to talk
- 35:28about this outer loop and the inner
- 35:30loop.
- 35:32Uh I don't know if I'm using the
- 35:33definition correctly but to me my inner
- 35:35loop is like basically my engineers my
- 35:38agent engineers working on the code to
- 35:41an building towards an intent or
- 35:45snapshot of my intent right and the
- 35:48thing about that is that the snapshot
- 35:50can go stale right new information comes
- 35:52to light that I then have to be the
- 35:54proxy of and you know transfer that
- 35:57context to my agent so you know if you
- 35:59if you don't have these triggers pulling
- 36:02information back into the interloop,
- 36:04then you sort of have to play that role
- 36:07where you're you're off, you know, in
- 36:09Slack or X or or whatever and you're
- 36:12gathering context, right? You're getting
- 36:14context about bug reports, about feature
- 36:16requests, about, you know, something
- 36:19someone said about, you know, our
- 36:20backend infrastructure has some
- 36:22limitation, you know, all that
- 36:24information, you have to f that across
- 36:27to your agent. So that's where I think
- 36:29like tools like Grogbot are really good
- 36:32because they help you automate the outer
- 36:35loop as well. And when you connect those
- 36:37two loops, it's very very powerful
- 36:39because now all of a sudden your agents
- 36:41have the ability to get context for this
- 36:44for themselves, right? If for example
- 36:48uh you know either through just as a
- 36:51simple example like maybe you have the
- 36:52Slack MCP, right? Or you have uh your
- 36:56own harness, right, that you've built a
- 36:58Slack subscription into for a particular
- 37:00Slack channel. Now all of a sudden you
- 37:03can tell your agents, okay, subscribe to
- 37:05the Slack channel. Every time there's a
- 37:08uh, you know, bug report about
- 37:11something, go off and triage that thing,
- 37:14right? Go reproduce the issue, right?
- 37:16Using the verification skills that we've
- 37:18already spent time building and all of
- 37:20those other skills that we've set up so
- 37:22that I have a lot of trust, right? I
- 37:24have a lot of trust that these agents
- 37:25can actually go off and understand the
- 37:28bug, you know, uh verify that the bug
- 37:31actually still exists on main and it
- 37:34wasn't
- 37:35something about you maybe the users
- 37:38setup or their data or maybe I don't
- 37:40know they didn't install a dependency or
- 37:42something like that like uh basically I
- 37:45think uh creating that yeah creating
- 37:49those two loops and connecting them is
- 37:52really a very important part of the job
- 37:53these days. Um, especially if you are
- 37:56thinking about how to scale yourself.
- 37:58So, a big theme here is really just like
- 38:01always thinking about like what where am
- 38:04I the bottleneck in this process? Why do
- 38:07my agents need me, you know, to answer
- 38:09this question? I I always like to think
- 38:11about that. And so I try to think about
- 38:13how do I actually get the agent to
- 38:15answer its own question, right? But not
- 38:17by hallucinating, not by guessing, but
- 38:19actually real data. And you know, a lot
- 38:22of people talk about this idea of a
- 38:23company brain, right, or a context
- 38:25graph. I feel like those terms are
- 38:27unnecessarily
- 38:29complex uh or even abstract. To me, it's
- 38:33just about um how do I take information
- 38:37that my agent needs that I would
- 38:39otherwise have to go and pass it myself
- 38:42and just teach it how to do it, right?
- 38:44And that removes me from the equation.
- 38:47And so how I arrive at 2,000 or however
- 38:50many PRs is the fact that I have all
- 38:53these loops set up, right? And so uh it
- 38:57allows me to open chain restaurants,
- 39:00right? I can I can really parallels
- 39:02myself. So yeah, I'm not sitting there
- 39:04creating 2,500 chats, right? Of course,
- 39:06it's really like these projects are um
- 39:10actually cursor has a new feature called
- 39:12projects which are these like
- 39:13coordinator agents. Um and so the
- 39:17coordination co coordinator agents are
- 39:19really good at sort of delegating and
- 39:21not doing work of their own but they
- 39:23manage and supervise like almost a list
- 39:25of tasks and they spawn sub agents to go
- 39:28and do them. And so I'm just constantly
- 39:30feeding context or teaching the agents
- 39:32how to get their own context and then
- 39:33they're going off and doing the work for
- 39:35me. Uh and really the big the last thing
- 39:38I'll say to this is like the big unlock
- 39:40for me for getting to 2,000 PRs is
- 39:43starting from the question and working
- 39:45backwards of how do I get to the point
- 39:48where my agent can merge its own code?
- 39:51Because
- 39:53the obvious thing people ask me when
- 39:55when they when I tell them, "Oh, I
- 39:56shipped 2,000 and 2,500 pull requests
- 39:59last month." They'll be like, "How did
- 40:00you review that?" Right? That that's a
- 40:02lot of PRs to review. Like your team
- 40:04must hate you.
- 40:05>> Do do you mind if we go there in a
- 40:07second? Because a good question about
- 40:09that.
- 40:09>> Yeah. Yeah. Yeah.
- 40:10>> I want to like this analogy is great. I
- 40:12want to like deepen it a bit which is
- 40:15before if you're like manually
- 40:17initiating all those chats it's like
- 40:18you're bringing the orders to your chefs
- 40:20manually right whereas if you've got an
- 40:22agent sort of like doing the expo then
- 40:25you're able to sort of run it yourself
- 40:26itself what is what does that concretely
- 40:30look like then you've got these sort of
- 40:31grock bots that are um subscribing to
- 40:33channels pulling in Slack messages and
- 40:36you it sounds like have a couple of
- 40:37coordinator agents or like chief of
- 40:39staff agents that like monitor that or
- 40:43something like when you look at your
- 40:45computer to manage your agents, what
- 40:47does it look like?
- 40:49>> Yeah. So, [clears throat] so uh this is
- 40:52I guess somewhat confusing but we're
- 40:53working on you know simplifying and
- 40:55unifying but so uh there's graphbot uh
- 40:59which or you know you can use other
- 41:01tools of course as well but I I largely
- 41:03think of these tools as like your outer
- 41:05loop. These are tools like you know
- 41:07Grabbot that have connectors right these
- 41:09are connectors I guess they a lot of
- 41:11people call them personal agents um but
- 41:14they're connectors to things like your
- 41:16email your calendar slack uh plaid I
- 41:20don't know like all these different
- 41:21services and they are a great source of
- 41:24pulling context in to your work so the
- 41:29same way that a human like you know if I
- 41:31were if I was a manager and I was
- 41:34leading a team of engineers years. Um,
- 41:37you know, like when I used to work in
- 41:38Netflix, one of the biggest things that
- 41:40managers would talk about was this idea
- 41:42of context not control, which funnily
- 41:45enough, you know, has so much uh has so
- 41:48much uh carry over to the agents world.
- 41:51Uh, of you know, you you know, you you
- 41:54of course can drive to an outcome you
- 41:56want by control, right? Like by
- 41:58micromanaging, but what you want is to
- 42:00provide context instead, right? like
- 42:02teach the agent, teach your engineers
- 42:04how to be self-sufficient and then you
- 42:06don't have to micromanage them.
- 42:09>> Um, and so I see a lot of parallels
- 42:11there. Uh, but yeah, graphbot. So,
- 42:13concretely, I have some graph bots that
- 42:16look at my Slack channels, look at my X,
- 42:19uh, or my emails, uh, or linear, and
- 42:23they're just constantly they have
- 42:24routines that subscribe. So they're
- 42:26constantly watching and I have I I'll
- 42:29tell them things like you know uh I'll
- 42:31watch for issues with uh bugs in the
- 42:34graphbot desktop app as an example. Uh
- 42:37and whenever you find that send it to my
- 42:40cursor project. So one of the really
- 42:42cool things about grabbot is it connects
- 42:43to cursor. So cursor has uh like I I
- 42:47just mentioned this new feature called
- 42:49projects. And a project is really a uh
- 42:52again like a you get a coordinator agent
- 42:55that's in the cloud. It has its own
- 42:57computer and all it really does is like
- 42:59it's a manager of agents. It's like your
- 43:01executive chef, right? Your your chief
- 43:03of staff. It doesn't do the work itself.
- 43:06It delegates and orchestrates and
- 43:08manages the work of other sub agents to
- 43:12you know that report to your chief your
- 43:16chief uh of staff. And it basically is
- 43:20responsible for driving the work forward
- 43:22and managing things and uh passing
- 43:25context to them.
- 43:27>> So if you get a sudden burst of issues,
- 43:28let's say you get 30 issues at once in
- 43:30one payload or something or very quickly
- 43:32the coordinator agent can figure it out
- 43:34and delegate.
- 43:34>> Yeah, exactly. It gets like uh you know
- 43:3630 the 30 or so payloads and spawns a
- 43:39sub agent or a single coordinator agent.
- 43:41It can actually do a bunch of different
- 43:42topologies of agents and it will sort of
- 43:46figure out the best way to uh you know
- 43:48efficiently distribute the tasks to your
- 43:52team of agents. Um so I use uh cursor
- 43:57projects a lot um and I also use grapot
- 44:01a lot and cursor projects are my inner
- 44:03loop and grabbot is my outer loop.
- 44:05Grabbot takes all the context, external
- 44:08context, gives it to the projects
- 44:11because it can actually just send
- 44:12messages to those projects, right? You
- 44:14don't even have to open cursor. You can
- 44:16just tell your grabbot, okay, create a
- 44:18project, right, for these series of
- 44:20tasks. They're all related, right? Maybe
- 44:23as an example, you know, you've had a uh
- 44:26[clears throat] a big burst of issues
- 44:28that are all about performance, right?
- 44:29Your app is slow uh and they're all
- 44:33connected, right? Maybe some of them
- 44:34even have a similar fix, right? But and
- 44:38you can certainly go off and just spawn
- 44:40one agent per task, but then you've lost
- 44:42that sort of thread between them, right?
- 44:45And and you may duplicate work or you
- 44:47may not really think about the higher
- 44:48level problem. You know, sometimes when
- 44:50you you you solve bugs, you know, it
- 44:52helps to have multiple bug reports that
- 44:54are are slightly different because it
- 44:56helps you really, you know, zoom out and
- 44:58see actually, you know, the problem when
- 44:59I looked at this one report, I thought
- 45:01the bug was here, but actually when when
- 45:03I see the other multitude of bugs is
- 45:05actually up here,
- 45:07>> right?
- 45:07>> Yeah. Got you. So that that's why you
- 45:09have so many agents in that loop then,
- 45:10right? Because it's not just you have um
- 45:13like you have a bug report comes in, you
- 45:15spawn a single agent to look at that bug
- 45:17report. that a that single agent will be
- 45:19duplicating work with other um other
- 45:21agents, right? Because if there are
- 45:23multiple bug reports coming in through
- 45:24the same thing, that can be duplicated
- 45:25work.
- 45:26>> Yeah.
- 45:26>> Fascinating. [snorts]
- 45:28>> That's really fascinating. Okay. And so
- 45:30this just this endless series of
- 45:32triggers um coming from real users
- 45:34reporting real reports um builds up this
- 45:38sort of and accelerates the factory sort
- 45:40of adds more orders in. Other than bug
- 45:43reports, are there any other sources
- 45:44that you use for like um accelerating
- 45:48for pushing these PRs?
- 45:51>> Uh well, funnily enough, it's some of it
- 45:54comes from uh reading the code, too. So,
- 45:57I guess I have sort of uh well, so to
- 46:01clarify that, you know, the 2,500 PRs,
- 46:03they're not obviously like 2,500
- 46:06features, right? they are a lot of the
- 46:08work actually is spent on gardening like
- 46:12another term that I really love. Uh
- 46:15where
- 46:17so I guess this is more important when
- 46:19you have a big team of engineers human
- 46:21engineers that you work with where and
- 46:25also this goes back a little bit to what
- 46:26I was talking about with the
- 46:27environment. You know, setting up a
- 46:28really good environment that doesn't
- 46:29just help you and your agents, but
- 46:31everybody on your team, right? Think of
- 46:33a new hire who doesn't have a lot of
- 46:34context on all of your engineering
- 46:36practices joining your your team. And if
- 46:39you have a really good environment, they
- 46:41can be productive from day one, right?
- 46:43They can they don't have to like, you
- 46:44know, make open a bunch of lowquality
- 46:46PRs. They can start, you know, they can
- 46:49start just turning out really good code.
- 46:52Um,
- 46:54and uh, yeah, I think I I sort of lost
- 46:57my train of thought.
- 46:58>> I've got a I've got a followup, which is
- 47:00what's what are the mechanics of like
- 47:02triggering
- 47:03>> like how when when do you trigger a a
- 47:06agent
- 47:08to go and look at the code, right?
- 47:10Because some people might say, "Oh,
- 47:11let's just do that every hour or
- 47:13something or like on a chron job or
- 47:15what's
- 47:16>> Oh, yeah. Yeah. Yeah. Yeah. Uh I saw
- 47:19some of your recent tweets as well about
- 47:20like you know the some of the tweets
- 47:22you've been doing which are great for
- 47:24setting up your routines. Uh I have some
- 47:27routines like that as well. Um so uh one
- 47:32of them is uh
- 47:35like looking through just another simple
- 47:38example is you know [clears throat]
- 47:40React has a lot of foot guns. Um, so,
- 47:43uh, as as I'm sure you're aware. And so
- 47:45I have an agent that's just constantly
- 47:47looking for band patterns. And the
- 47:50interesting thing about that one is that
- 47:52I don't actually tell it to fix the
- 47:53issue first. I tell it to append it to a
- 47:55document. And then every couple of days
- 47:58I look at it and I see actually these
- 48:00are all the same thing, you know, and so
- 48:02that gives me, you know, you almost want
- 48:04like a buffer, a queue. Sometimes that's
- 48:07actually more effective than just
- 48:09spawning off a couple of like a lot of
- 48:10sub agents to fix every single thing
- 48:13because when you are [clears throat] in
- 48:15kind of pure execution mode and just
- 48:17trying to like you know f uh you know
- 48:20execute on the orders that are coming in
- 48:21very fast you sometimes miss the big
- 48:24picture. So sometimes having a buffer
- 48:26forces you to think about the big
- 48:28picture because you you you have these
- 48:31artifacts and things that you can look
- 48:33at as a human um and sort of use your
- 48:36own human judgment to or I guess you can
- 48:39use an agent to do that as well. But you
- 48:41give the [clears throat] agent and
- 48:42yourself a way to identify patterns,
- 48:46right, that you might otherwise miss if
- 48:48you're just only solving each bug at a
- 48:51time. And that's also really the benefit
- 48:53of having something like a chief of
- 48:54staff agent is uh it can see the forest
- 48:59right uh in addition to actually doing
- 49:02the execution.
- 49:04>> Fascinating. That's I mean my brain is
- 49:07exploding a bit there with the sort of
- 49:08chief of staff at the software factory.
- 49:10I might have to change some of the
- 49:12course that I'm filming next week.
- 49:13That's [laughter]
- 49:15>> uh all right. Let's talk about let's
- 49:16talk about review, right? because this
- 49:18is the reply that you get, you know, is
- 49:20>> did you read did you taste all 2500 of
- 49:24those dishes as they swept past you?
- 49:27>> And I assume the answer is a variety is
- 49:30a version of no.
- 49:34>> Yeah, I think you you you don't want to
- 49:36be in a position where you're not
- 49:37tasting your food ever again. Uh but you
- 49:40also, you know, for scale, you cannot be
- 49:42tasting every single dish that comes out
- 49:44of your kitchen, especially if you have
- 49:46multiple restaurants. So it becomes more
- 49:48about sampling right and thinking about
- 49:50the processes in the same way that you
- 49:53know if I guess maybe this is where the
- 49:55the the factory analogy is a bit more
- 49:58apt is you know as a quality supervisor
- 50:01on a factory you you can't look at every
- 50:04single item you sample right you take
- 50:06you you you go in there every day and
- 50:09you look at the quality of the pull
- 50:11requests you look at the code that the
- 50:12agents are writing and you scrutinize it
- 50:16very rig rigorously and you think about
- 50:19all the inefficiencies, the bad patterns
- 50:21that the agents are doing and then you
- 50:24think about how to course correct the
- 50:26environment, right? Not not that single
- 50:28agent. Uh because if maybe if it if it
- 50:33was a one-off incident, it's fine. You
- 50:35know that maybe there's nothing to fix
- 50:36there. But if you actually notice that
- 50:39multiple agents are are having the same
- 50:41issue, right? They're taking the same
- 50:43shortcut. they're they're propagating
- 50:45the same workaround everywhere. Uh
- 50:48that's a sign that you should go off and
- 50:50think about how to uh amend your kitchen
- 50:53or your factory, right? Like thinking
- 50:55about your skills, your constraints,
- 50:57your lints, your type systems um and
- 51:02setting or adjusting it so that that
- 51:04problem doesn't happen again. And when
- 51:06you do that enough times, then you get
- 51:08to a place where the codebase is again
- 51:10like the environment is so constrained
- 51:13and so it guides you so well that you
- 51:16can just you can just step away, right?
- 51:19That's the dream. And I'll I'll
- 51:20definitely say it um it's very hard to
- 51:23get to this point. I don't want to sell
- 51:26this as like, you know, something that
- 51:27you can just do easily by using PAC.
- 51:30Like it takes a lot of time and effort
- 51:32to think about your code and where you
- 51:35see your agents failing and thinking
- 51:38very thoughtfully, intentionally and
- 51:40setting up guard rails and constraints
- 51:43so that they do the right thing by
- 51:45default.
- 51:47>> And you're not [clears throat] like if
- 51:49to go back to the software factory
- 51:51analogy, this isn't a dark factory,
- 51:53right? This the lights are on, right?
- 51:55>> It kind of is. Yeah, actually.
- 51:56>> Is it?
- 51:57>> Yeah. Well, it's dark in the sense that
- 51:59so um it's dark in the sense that well I
- 52:02think if my agents are merging their own
- 52:04pull requests it's sort of become dark
- 52:07where I go to sleep my agents now work
- 52:0924/7
- 52:11uh I have I have the equivalent of like
- 52:14more than 10 chiefs of staff right each
- 52:16working on a different area like for
- 52:18example I have one that's working on
- 52:20performance of the Grockbot desktop app
- 52:23I have one that's working on uh fixing
- 52:25bugs that users report I have one that's
- 52:28exploring rewriting it in a different
- 52:31language just for fun, you know, like
- 52:32what if what if, you know, just
- 52:34reimagining what what it would be if it
- 52:35was like a native app. It's just a toy.
- 52:38Um, but the idea is like yeah, I uh I
- 52:41when you spend the time setting up your
- 52:43environment, I've gotten to a point
- 52:45where I review the pull request after
- 52:47it's landed, right? I I tell my agents
- 52:50full autopilot is is something that you
- 52:52can do in in PAC and that will trigger
- 52:56off this very intense rigorous
- 52:58verification loop where it will spawn a
- 53:00bunch of verifier agents for every pull
- 53:03request and it will fuzz right fuzzing
- 53:06meaning that it will actually run the
- 53:07application. It's going to click around
- 53:09and try to use it like a real human.
- 53:11look for regressions, look for bugs in
- 53:13your implementation and um it will try
- 53:17to find issues with the thing and then
- 53:19it will fix it itself. It'll do that
- 53:21again and eventually get the PR to a
- 53:24state where it can land.
- 53:26Uh so it does it does it is quite token
- 53:29intensive. You can tune this of course.
- 53:32Uh so you know instead of like 10
- 53:34verifier agents you might do like one,
- 53:37right? Or you just tell the agent to
- 53:39verify it's done work. But yeah, the key
- 53:41thing is the verification part is really
- 53:44the key piece that gives me a lot of
- 53:46confidence
- 53:48that I guess verification plus the
- 53:49environment, right? It's these the
- 53:51combination of these two things that
- 53:52allow me to step away and say agents go
- 53:54off and merge your thing. I'll review it
- 53:56in the morning by looking at my commit
- 53:58history
- 53:59>> and if I see problems, I go and course
- 54:02correct,
- 54:02>> right? And I'll go and revert or modify,
- 54:06add new link rules and whatever. Um, so
- 54:10it does it does take time to get to that
- 54:12point, but once you get it, oh, it's so
- 54:14it feels so magical. Uh, I I I tell
- 54:17people like I'm sleeping so much better
- 54:19now because, you know, it took it the
- 54:22very first day I turned on the sort of
- 54:24dark factory was very scary because I
- 54:26was like, "Ooh, what if I call the SE,
- 54:28right? What if I break something
- 54:29overnight?"
- 54:30>> Uh, and it took a lot of it took a lot
- 54:34of uh bravery, I think, to do that, but
- 54:37>> somehow I did it. And yeah, now I'm in a
- 54:39place where my Asians are are merging
- 54:42their own code while I sleep.
- 54:43>> It sounds like
- 54:44>> I think it's dark in that sense.
- 54:46>> Yes, it's dark sometimes, right? You do
- 54:48sometimes.
- 54:48>> That's true. That's true.
- 54:49>> Because I think of a dark factory is
- 54:51like almost like if you take the
- 54:53original definition of Kapathy's vibe
- 54:55coding, right, which is the code almost
- 54:57doesn't exist. You forget that code
- 54:59might be a thing. I think your approach
- 55:01is totally different from that, which is
- 55:03that code and the environment is
- 55:06essential. And if the code in the
- 55:07environment are bad, then you will get
- 55:09bad outputs. Garbage in, garbage out. So
- 55:12I I I think this is a this is a
- 55:15different thing. It's like, you know,
- 55:18the I don't know, maybe there's a dimmer
- 55:20switch or something, right? Like, you
- 55:21know, some parts of dark, some parts
- 55:23were light. This is why the maybe the
- 55:24the kitchen is a better analogy
- 55:27>> the restaurant because you know even as
- 55:29a as a as a
- 55:32as a [clears throat] restaurant
- 55:33restaurant you still might go to your
- 55:36restaurants every now and then to take
- 55:37take a peek in taste the food right
- 55:40>> uh I think that's
- 55:41>> the idea of sampling instead of blocking
- 55:42I think is really important
- 55:44>> I think what would you say to people who
- 55:46are in I guess you're obviously in a
- 55:49pretty security conscious environment
- 55:50where you're working very security
- 55:52conscious
- 55:53>> mh [clears throat] Um maybe there are
- 55:56folks working in like um medical
- 55:58applications or law or finance or
- 56:01something. I think of the like some PRs
- 56:04are kind of like two-way doors which is
- 56:06you can merge it and then revert it,
- 56:08right? It's cheap back through. But
- 56:10there are some PRs that are one-way
- 56:11doors, right? That will cause data loss
- 56:13of some kind that will
- 56:16>> do something that can't be easily walked
- 56:18back. How do you deal with situations
- 56:21where most of your PRs, let's say, are
- 56:23one-way doors? Like, is this something
- 56:25you just wouldn't recommend or like what
- 56:27do you think?
- 56:28>> Yeah, I think that's a really good
- 56:30question. I think that
- 56:32it all comes back to me to the quality
- 56:34of the verification that you're able to
- 56:37um get out of your agent. And I think
- 56:41for domains where the work is
- 56:44verifiable,
- 56:46this is easier, right? and the the
- 56:48oneway doors become two-way doors in a
- 56:51sense.
- 56:53But I guess I don't know if you're
- 56:54working on something that is like
- 56:56extremely
- 56:57is very hard to verify programmatically
- 57:00then I think yeah you're definitely in a
- 57:02position where it's very hard to get to
- 57:04that point. Um so I do think like yeah
- 57:07verifiability of the domain is an
- 57:10important aspect to be able to do this.
- 57:13Um and software engineering is just one
- 57:15of those things where it's quite
- 57:16verifiable in in a lot of cases maybe
- 57:19not totally um you know like other
- 57:22domains like mathematics I think are
- 57:24another example of not all of it of
- 57:26course but some aspects of mathematics
- 57:30can be verifiable
- 57:32if you write a proof for example um and
- 57:35so yeah I think it's a great question
- 57:38that I don't really have the answer to
- 57:40and I think that this is something the
- 57:41industry and us as engineers will have
- 57:43to figure out is you know my sort of uh
- 57:48hope and prediction for the future is
- 57:49that we'll see more and more interesting
- 57:52new agentoriented programming languages
- 57:56and one of the most fascinating ones
- 57:57that I've seen so far is this one called
- 57:59bend bend d bend um and that language is
- 58:04one where it kind of marries programming
- 58:08with proofs right there used to be a
- 58:12time, you know, where you actually had
- 58:14to write your proofs in a different
- 58:16language. And proofs, by the way, for
- 58:18those uh who who aren't familiar is this
- 58:20idea of uh that you can sort of formally
- 58:24verify that some code is correct
- 58:26mathematically, right? Especially if
- 58:29you've written your code in a very
- 58:31functional programming way.
- 58:34uh but for the longest time you had to
- 58:36do that in a separate language like lean
- 58:38or tla+ or uh I'm blanking on some of
- 58:42the other other examples but uh like
- 58:45languages like that where you would
- 58:46construct the mathematical proof and
- 58:48then use a solver essentially to det
- 58:52that that you've covered all the cases
- 58:54you don't have like a race condition or
- 58:56or whatever.
- 58:58So yeah, I think
- 59:02trying to sum up the question, I think
- 59:03yeah, if you are in a position where you
- 59:05can figure out how your agents can truly
- 59:07verify the work in a way that gives you
- 59:10confidence, you can actually, you know,
- 59:13uh have the PRs merge cuz if it
- 59:16compiles, right, if it if the proofs
- 59:18show you that it's correct, then why
- 59:21wouldn't you just merge it? Um but of
- 59:24course, yeah, not all the means are
- 59:26verifiable. Yeah, it's a tough one. Um,
- 59:31okay. I think we've got to think about
- 59:34wrapping up because we are nearly on the
- 59:35hour. Have you Have you got something
- 59:36after this? I mean, I've got something
- 59:38before I give my son dinner, but
- 59:40>> I I can go a bit longer after you.
- 59:42>> Okay, let's let's go five minutes longer
- 59:43then. Um I think I just want to have one
- 59:46more question which is
- 59:49I think I want to ask how you see Pstack
- 59:55and how you see skills in general like
- 59:57in terms of we talked about this before
- 1:00:00we went on air which is like people
- 1:00:02think of as like my skills versus your
- 1:00:05skills and how do you combine frameworks
- 1:00:08together? How do you use Pstack with my
- 1:00:11stuff? like what should you take from
- 1:00:13each one and because I think I see
- 1:00:16skills as sort of just derived from
- 1:00:19process basically like they're just
- 1:00:21processes turned into words and I would
- 1:00:25love to know
- 1:00:27how you recommend people take Pstack and
- 1:00:29take my stuff as well and turn it into
- 1:00:31their own processes.
- 1:00:35I think you shared a tip actually today
- 1:00:36that I thought was actually very
- 1:00:38relevant, which is this idea that you go
- 1:00:40off and look at your previous
- 1:00:42transcripts, right? And you sort of mine
- 1:00:44for information of, you know, your own
- 1:00:48look through your own your own prompts,
- 1:00:50right, to the agents where you correct
- 1:00:51them where you have to constantly
- 1:00:53intervene and uh you know take that
- 1:00:57higher level learning and turn that into
- 1:00:59a a reusable skill, right? So that
- 1:01:00agents stop repeating that mistake. I
- 1:01:04think that uh PAC and your skills are
- 1:01:08very complimementaryary. I I totally
- 1:01:10agree with you that they're like a skill
- 1:01:12is really much just process. I mean it's
- 1:01:14just at the end of the day a skill is
- 1:01:15just English or or language. It's just
- 1:01:18markdown.
- 1:01:19>> Um and I think you can you can
- 1:01:21definitely weave them, combine them in a
- 1:01:24way that makes sense to you. But I do
- 1:01:27think that uh everyone should have their
- 1:01:30own set of knives, right?
- 1:01:33I I keep going back to the the the
- 1:01:35cooking analogy, but it's so apt because
- 1:01:37like, you know, every chef when they go
- 1:01:39to a different job, right, when they go
- 1:01:41to a different restaurant, they carry
- 1:01:42they bring their knives with them. The
- 1:01:44tools go with them, right? And so trust
- 1:01:46to me is really about trust in your own
- 1:01:48tools. And when you spend the time
- 1:01:51sharpening them and understanding them
- 1:01:52really, really well, you can do great
- 1:01:54things. And everybody's skills and tool
- 1:01:56set is going to look different. you
- 1:01:58know, someone might find a lot of
- 1:02:00success combining, you know, like your
- 1:02:03grill me with docs, uh, or wayfinder
- 1:02:05skill with some of the execution skills
- 1:02:07in Pstack as an example. Some people
- 1:02:10might use more of your skills, some
- 1:02:11people might use more of my skills. I
- 1:02:13think at the end end of the day, it
- 1:02:15really just comes back to how much do
- 1:02:17you trust, you know, me and Matt, right?
- 1:02:19Like if you if you trust us both, of
- 1:02:21course, use our skills, but I also
- 1:02:23encourage you to, you know, look at your
- 1:02:25own transcripts. Um um tell the agent to
- 1:02:29look through, you know, some of the all
- 1:02:31of the patterns that you've used, the
- 1:02:33the times you've had to intervene, you
- 1:02:36know, suggest turning them into lint
- 1:02:38rules or new skills, right? The the past
- 1:02:41chats I I often say is like a a treasure
- 1:02:44trove of context because that, you know,
- 1:02:46it's it's like the process materialized,
- 1:02:51right? Like it's the real process. It's
- 1:02:53not an abstract idea in your head. it's
- 1:02:55the actual thing right and you can
- 1:02:57actually see how it happened in practice
- 1:02:59and extract so much information from
- 1:03:01that and there's so much so that I
- 1:03:02actually turn I have a skill in pac
- 1:03:04called recall which is exactly that um
- 1:03:07where this was a pattern where
- 1:03:10you know I was working in I was working
- 1:03:12on a similar problem so specifically I
- 1:03:14was working on virtualization for the
- 1:03:17cursor application and there were a lot
- 1:03:18of bugs and so you know every time I
- 1:03:21started a new chat I was like ah this
- 1:03:22there's so much good context from the
- 1:03:24last one So, you know, I want to bring
- 1:03:25it over to the new chat. How do I do
- 1:03:27that? And that's where the transcript
- 1:03:29came, uh, you know, looking at the past
- 1:03:31transcript came about. And then recall
- 1:03:33was just a way for me to collapse
- 1:03:35collapse and compress that workflow into
- 1:03:37a skill. So that I didn't have to just
- 1:03:39say I didn't have to write a long essay
- 1:03:42every time. Go look at all these chats,
- 1:03:43right? And, you know, blah blah blah.
- 1:03:45So, I I largely think of skills,
- 1:03:47especially as agents get more capable as
- 1:03:50really encoding workflows. you know
- 1:03:52skills from last year were really more
- 1:03:54about like almost like implementation
- 1:03:56details like here are the exact script
- 1:03:59commands you know you should use right I
- 1:04:02think with the latest models you can
- 1:04:04just delete those parts and just really
- 1:04:06focus on the workflow right until it
- 1:04:09it's more the skill becomes more like a
- 1:04:11series of steps
- 1:04:13a series of your process uh and I think
- 1:04:16over time we'll see that skills get
- 1:04:18smaller and smaller you know more
- 1:04:20compact Um, and yeah, they're very
- 1:04:24compatible. Or you can, you know, if you
- 1:04:26want, why not read our skills, right,
- 1:04:29and com and combine them in your of your
- 1:04:31own, right? Combine Wfinder with potato
- 1:04:34mode and make your own custom mode,
- 1:04:36right? Like like skills are the the
- 1:04:38thing I love about skills that is are
- 1:04:40that they're so malleable. You can do
- 1:04:42anything you want. It's just language.
- 1:04:44>> Absolutely. There's nothing magical in
- 1:04:46them, right? They're just words. And
- 1:04:48>> Exactly. If if there is any magic in
- 1:04:49them, it's just the words chosen and the
- 1:04:52phrases used and the thinking that's
- 1:04:55been done to turn those like take
- 1:04:59abstract process and turn them into
- 1:05:01language. And once that thinking has
- 1:05:04been done, then it's just there. It's
- 1:05:05available. It's on the surface and you
- 1:05:06just nick it. Um Lauren, thank you so
- 1:05:10much. This has been glorious.
- 1:05:11>> Yeah, this has been super fun. I really
- 1:05:14enjoyed talking to you. Hope we can do
- 1:05:16it again. I'd love to do it again. I'd
- 1:05:19love to do it again. Absolutely. Um
- 1:05:20yeah, we'll check in in uh
- 1:05:22>> I don't know. Yeah, Monday. Let's do it.
- 1:05:25>> Yeah, [laughter] let's do it. Part two.
- 1:05:28>> Well, thank you so much. I'm going to
- 1:05:29close the stream here. Laura and I will
- 1:05:31uh chat a little bit and stay here. But
- 1:05:33thank you guys so much for watching. The
- 1:05:34glorious.
About this transcript
This page contains the full transcript of LIVE: Poteto (creator of pstack) on shipping 1,000's of PR's a month at SpaceX by Matt Pocock, generated from the public captions YouTube serves with the video. The transcript has 12,134 words across 1,707 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.