SpaceX AI engineer Lauren Tan part 2 — Transcript
Full transcript
- 0:00Um, personally I think that uh the best use of you of AI is really as like a like a pair like someone
- 0:09you're pair programming with and not someone as not not a not a tool to just replace the act of
- 0:14writing code or worse outsourcing your thinking. And I think there's a tendency like, you know,
- 0:21because AI is so exciting, you want to put AI everywhere and be seen as someone who's very,
- 0:26you know, um, what do you call it? Like, uh, on the ball, I guess, with with AI. Uh,
- 0:33that you feel this pressure of like, you know, I'm just going to I got to increase my productivity. I
- 0:38got to ship like 100 PRs this week and barely understand what I'm doing and, you know,
- 0:42vive code my way to a million ARR or whatever. Um but I think you know um one of the things that
- 0:52I've personally seen is that although when I use AI I save some time you know writing the code I
- 0:59actually find myself spending more time reviewing what the AI did and like correcting it. So,
- 1:05you know, like I've I've tried the the thing that a lot of people say like, you know, you
- 1:09uh instead of prompting an AI to just build the feature outright, you sort of get it to uh build
- 1:15a spec for you first, then you review that spec, and then you do this whole thing, then it's like,
- 1:20uh, okay. So, I assume you can see my screen. Yes. Yeah, we're good. [clears throat] So yeah, today
- 1:25yeah I think I think the big theme for me as I've been using agents to write code and I'm sure a lot
- 1:31of you have had the same experience as well is how do you trust it? You know especially if you are an
- 1:36engineer that's been writing code for a very long time. You have a lot of opinions and lessons that
- 1:43you've learned about doing good engineering. And when you see agents just, you know, winging it
- 1:49and, you know, guessing, hallucinating, uh, you know, confidently stating that they found the
- 1:55smoking gun, uh, for the hundth time, uh, but it's actually not the real problem. You lose a lot of
- 2:01trust. And when you lose when you don't have much trust [clears throat] in your agents, I feel like
- 2:06you you really can't get the most out of them. And for me, the parallel is like with management. Uh
- 2:12so if I'm an a manager an engineering manager of a team and I have a bunch of you know I have a team
- 2:18of engineers uh on my team and I don't trust them then the mode of operation I'm going to be in is
- 2:25going to be like micromanagement right I'll have to spend a lot of time looking over my reports
- 2:30shoulders and checking that they're doing their work well you know that they're not shipping bugs
- 2:35to production and so I drew this chart because uh it's It's not it's not a very scientific chart but
- 2:43like this is how I imagine myself and my journey through using agents. So you know like fast
- 2:51forward or back forward uh or fast back uh fast backwards like a year or so when you know nobody
- 2:59was or not many people were using agents to code. Uh I think you uh you know get into this mode
- 3:06where you are in very heavily in the loop with one or several like a handful of agents and you find
- 3:15yourself just constantly fig uh you know trying to understand what your agents are doing uh and
- 3:20you're very very in loop. You're watching every single output. you are sitting there prompting
- 3:26um and you really can't parallelize beyond that because you don't again you don't have that trust
- 3:32right you can't go to a 100 agents uh like spawn 100 agents when you don't even trust the output
- 3:37of one agent so over the past 5 months I feel like I've really been able to uh like ascend this
- 3:47trust curve and now I'm at the point where uh I actually have this sounds kind of scary to say And
- 3:53it it makes me sound like a slop artist, but I I promise I'm not. But I actually have my agents now
- 3:59um autoemerging PRs for me. Uh which is like a wild thing to say, but um like I woke up today and
- 4:06there were like 20 PRs landed and I just reviewed them on main like they were already landed and
- 4:12they were good. Uh so how did I get to that point is basically what I wanted to talk about today.
- 4:19Uh and again like yeah feel free to jump in if you have questions. colon. Um but uh oh yeah, of
- 4:27course I got to show this this chart. [laughter] Uh where uh no do not trust to someone requested
- 4:36to control my computer. Uh I probably won't do that. Uh but yeah, so this chart I think I I'm I'm
- 4:43sharing this chart not to kind of like flex but to kind of show like the journey like so you can
- 4:49see like the curve like it sort of like inversely matches the contributions I've been able to land
- 4:54at cursor. So I joined five months ago and five months ago like I you know my first month I was
- 5:00like not very productive because I was obvious you know I was learning the codebase didn't know what
- 5:04the heck was going on and as I got more confident in in my agents uh I've really been able to kind
- 5:12of ramp up my productivity uh and again like yeah like last month I shipped a thousand PRs which is
- 5:19ridiculous. Uh, and then this month we're only on the 12th. I'm already at like almost 800 PRs
- 5:26landed. Uh, so the velocity is definitely high and you you I'm sure a lot of you will definitely
- 5:33be questioning like how how much of this code is actually good. Um, and I think yeah, like that's
- 5:38definitely fair to question. Um but uh yeah, I think I think if you set up your agents well,
- 5:48you can definitely get to a very similar level. Um and so I'm going to talk about how we do that.
- 5:56Uh so for me I think I'm curious like I guess call in your experience as well but uh for me
- 6:03I think the most important skill that you should have in your toolbox when you work with agents is
- 6:10verification. Uh and by verification I mean the ability for an agent to actually run the code
- 6:18uh or take CPU traces or heap snapshots or uh you know open an iOS simulator whatever you know
- 6:27however your application is exposed to your users. It can do the same thing and uh run it for real
- 6:35and actually test and verify it don't work because that's the thing that really closes the loop.
- 6:41uh it doesn't guarantee your agent writes good code uh but it allows them to at least
- 6:46write correct code uh which is a big a really big step forward for being able to trust your agents.
- 6:53Um I will I can share one example that we have uh within cursor. Uh oops where let me open this
- 7:14let's make this uh make full screen. There you go. Uh so for the for cursor's agent window
- 7:23uh so this is actually an interesting story but uh when I joined cursor five months ago
- 7:29uh they're actually uh well I was supposed to join a different team I was supposed to join like the
- 7:34cloud agents team uh but then since I have a lot of experience working on react and agents window
- 7:40is a react application uh I was uh I was asked to basically help out with the agent window work. Uh
- 7:50but um there wasn't really a lot of like skills to help me. So I just found myself that okay
- 7:58uh agents window is going to launch in like week right we have a really tight deadline and um there
- 8:05was uh you know I was just sitting there like okay I'm going to open up the performant the the chrome
- 8:09dev tools and just like take a trace look at it myself and try to make sense of this flame graph.
- 8:14And keep in mind I was just like in my first week so I had no idea what I was looking at. No idea
- 8:19where you know I mean I had some idea but you know the code base was completely fresh to me. Uh and
- 8:25I realized like my agent had no idea either you know like I would take a screenshot of the tree I
- 8:30would download trades I would send it to it and it be like yeah it kind of looks like this you know
- 8:34uh and it would like confidently state like it's this thing and then I try to fix that and turns
- 8:39out that's not the actual thing. So this was very very slow process and if you've ever done any like
- 8:46performance work yourself or you know just even development with an agent where you don't have a
- 8:50verification skill you are the verifier right you you're the bottleneck you you you tell your agent
- 8:57to do something and then it goes off and write some code then you open up your you know local
- 9:01dev build and then you start to say oh you know doesn't work then you got to copy paste screen uh
- 9:06you know screenshots or console errors or whatever uh and then your agent like slowly kind of like
- 9:12uh you know works with that and then tries to understand it and um fix the thing but
- 9:19then you're constantly just in the loop and and being a bottleneck. So there's really no way to
- 9:22parallelize. So the control glass skill is like one of the first skills I built uh for cursor. Uh
- 9:29and glass by the way is the code name for agents window that we use internally but it's just cursor
- 9:36I guess. Um, and so this skill uh is I guess the the the code itself is not super interesting. Your
- 9:43agent can very easily make one for you. Uh where if if you're building an electron app or a web app
- 9:49or even iOS uh applications, uh you can teach your agent how to use like the Chrome DevTools
- 9:56protocol or through uh Apple has some utilities as well for running the simulator and taking traces
- 10:03and controlling programmatic control as well. Uh so that's really useful. Uh but one thing I
- 10:11want to talk about is uh the this thing uh where is the read me? Uh so this skill comes with this
- 10:21very unique feature called or not feature uh unique file called a feature map. And so the
- 10:29story then is like I built this skill and so now the agent was able to uh actually run the agent
- 10:35window and take traces and whatnot. Uh but it had no idea what what the agent's window was. So
- 10:43um you know like someone would say like oh the the left sidebar is like laggy or something like
- 10:49that or you know the right side the the PR tab is not working and the agent would just be like kind
- 10:55of flailing around. it would spend a lot of time trying to like look up the code and you know where
- 10:59is this feature? How do I actually get to it on the UI which made it basically completely useless.
- 11:04Uh you know like we would I would run this skill locally and you know it would spawn a dev build
- 11:12uh but then it just be turnurning like I just try to click here. It wouldn't know how to get
- 11:17to things um and it was just an awful experience. Uh so who's putting arrows on my screen? Um, so,
- 11:26uh, yeah, this this feature map has been really useful, uh, because it teaches the agent how to
- 11:32get to all of the features that you have. Um, and in PAC, the plugin that I I've made, uh,
- 11:39if you search for PAC cursor on Google, you you'll find it. Uh, but there is a create verification
- 11:46skill in that plugin where it actually helps you set up something like this for yourself. Um,
- 11:51including the feature map. So it will actually explore the code and build up this initial feature
- 11:56map that tells your agent how to get to all of the different features that you have. Uh and this is
- 12:02extremely powerful because now like you have these user reports that come in uh you you can actually
- 12:08map even like a vague report or even a screenshot. So we have this uh internally at cursor where uh
- 12:16we have a slack channel with you know lots of people giving us feedback on the agents window
- 12:21and rockbot and whatnot. Uh, and often times the report is very bad, like very low quality,
- 12:28like someone just put very often we get like a screenshot like and then someone just says
- 12:32question mark question mark question mark like what is this? [laughter] And you know
- 12:35like the without this your agent is like I have no clue, right? But with a feature map like this,
- 12:41it has a lot more context and understanding of how to actually navigate how to get to all of
- 12:47the different features. Uh so like you know example like I guess like the sidebar like
- 12:52what is the sidebar uh you know like all the different sub features that are present in it
- 12:58um like from the user point of view here's where to how do I get to it all the different keyboard
- 13:03shortcuts uh even like the the what do you call it the DOM elements or yeah like the attributes
- 13:10that you use for selecting things through the CDP uh are all there. So uh again yeah this is
- 13:17like really really powerful uh for for agents uh and piece stack ships uh that create verification
- 13:25skill but also a maintain verification skill uh so you can keep this up to date. Cool. Yeah, I was
- 13:32just going to ask how you created that. So do you mind sharing a little bit more about um that that
- 13:36process in the context of Pstack and maybe just what Pstack is for the folks who aren't familiar?
- 13:41Yeah. So, Pstack is pretty interesting because uh well, first of all, the name is kind of goofy.
- 13:46Like the P the P in P stack is like potato potato stack because I um so uh uh there's a pretty
- 13:57uh famous person Gary Tan who is the CEO of Y Combinator and he's come up with this plugin
- 14:03called GStack uh Gary Stack and uh funnily enough we share the last name. We have no relations. Uh,
- 14:11but I thought it would be funny to kind of, you know, poke fun at Gary and make Pstack my version
- 14:16of of of his plug-in. Uh, but kind of tailor it to my own set of p engineering practices. Uh,
- 14:25but I honestly actually never set out to build PAC. Uh, it just started with a bunch of skills,
- 14:30right? Like I started with that control glass skill and then I started with another skill
- 14:35like called how which I also noticed through like observing agents. Um so like you know in
- 14:42the early days of me you know trying to climb this ladder I was like super in the loop and I
- 14:46was basically nitpicking my agents to an extreme degree. I was like I would tell it um you know
- 14:53this feature has stopped working. Here's a bug report like why isn't it working? And very often
- 15:00the agent would just like confidently state like, "Oh, it has to be this, right? It has to be this
- 15:05thing." And I noticed like when I looked at the actual tool calls, I noticed it wasn't actually
- 15:10reading the code that I thought should be affected. And that made me just extremely
- 15:16suspicious. And at that point, I was like, I'm not going to I can't trust any this agent anymore cuz
- 15:20it's just it's just completely hallucinating. And I think I think it's very easy to just, you know,
- 15:27like build up that distrust and not and kind of feel helpless like, you know, you don't know how
- 15:33to help your agents succeed. But like again, I think the the the management analogy is super
- 15:38helpful because like imagine if you were a manager of an engineering team and you had an engineer on
- 15:44your team who was a really good coder, no business context whatsoever. you know they they just you
- 15:49just hired them and they they onboarded you know like 5 seconds ago. Uh and so how do you actually
- 15:55teach that person to be effective? So how you do that is through a skill. uh skill being just
- 16:01you know it's just markdown right but you know it encodes a lot of information instructions a lot of
- 16:08uh you can really draw out a lot of intelligence from an agent by well some people on Twitter call
- 16:15it like you know pull the agent to a different latent space which is kind of like a fancy way
- 16:19of just saying like since uh you LLMs are sort of like they predict the next token uh when you give
- 16:26it some high quality tokens uh to begin with then you know it it can kind of pattern match on like
- 16:32a higher space that's you know smarter. Um so that's like a very interesting model there. But
- 16:40yeah I built PAC very very incrementally. Uh so uh started with just really observing how
- 16:46agents you know all the fail different failure modes of of that agents were having and every
- 16:51time I saw that I just okay I'm just going to make that a skill right like stop hallucinating
- 16:56actually go and search up look up the code use a lot of sub agents uh and yeah stop guessing
- 17:06yeah that makes sense one one kind of followup question here both from myself and from a bunch
- 17:10of people in the chat so I guess it's two two parts So one is like how do you maintain these
- 17:14skills? So like the product changes over time. Obviously there's a lot of people who are shipping
- 17:17against the codebase. So how do these skills get maintained? Uh and then second to that is like
- 17:23how do you know when your verification is is good enough? Uh like and you know you can trust that
- 17:28the ver verification loops that you've built are going to I guess you trust that the outputs uh
- 17:32when they're done. Uh yeah maybe I'll talk about um I think somewhat related. Maybe I'll start with
- 17:40this one first. So like how do I maintain these skills? So um if you're not familiar with this
- 17:49concept, an eval is essentially like a way to uh well I think the mental model I have is like it's
- 17:55like a unit test for an agent. Um and uh you can actually make your own eval. You don't need like a
- 18:02special framework for them. You can build you can you can build one depending on like you know how
- 18:08scientific and how rigorous you want to be. Uh, my screen is red. Yeah, there's a little button. Um,
- 18:15sorry. Do you mind like disabling the drawing or something? I I can't see my screen. Yeah, sorry.
- 18:21If you guys could not draw on the screen, that'd be great. But, um, there's a little button in the
- 18:24Is that a troll? Yeah, the little drop down. Um, how do I clear? Yeah, you got it. Perfect. Yeah.
- 18:36Yeah, evals are a way to unit test your skills basically. And actually in Pstack, we ship
- 18:43uh under potato mode there's a playbook if you search for it called eval playbook. Um and it's
- 18:49uh uh it's like not it's actually pretty pretty rigorous the way it's done. Uh but um essentially
- 18:57what I do is I spawn a lot of different sub agents. I have like my main coordinator agent
- 19:03uh come up with a rubric for uh what I want the skill to do. Um and then it spawns all these sub
- 19:11agents and it it creates individual directories for them uh which are cleverly named to not
- 19:18let the sub agent know that it's being evaluated because uh agents can actually tell and when they
- 19:24do they change their behavior. Uh, but it does a bunch of stuff like that to um essentially yeah
- 19:32like test whether or not the skill I'm making or changing is actually doing what I think it does.
- 19:38Um, and one of the really nice things about cursor is that we are we we support so many
- 19:43different models. So you can actually eval your skill across all sorts of different models. Um,
- 19:49and you know get a sense of how well it performs across that different matrix. um especially for
- 19:55the models that you use. Uh so I do this a lot. Every time I I modify a skill, I will run one of
- 20:02these uh like the Ebal playbook uh and make sure that you know it's actually leading to a result
- 20:08I want. Uh but I will say like maintaining skills is actually pretty hard. Uh it requires I think a
- 20:16lot of taste and observation. So you kind of need to be very good at being a backseat driver. You
- 20:23know what I mean? Like if you do pair if you ever done pair programming for example uh and you watch
- 20:28a coworker code and you just like you you could probably do this better you know you could do you
- 20:32know like why did you not do this right you you ask a lot of questions to your coworker and it's
- 20:37kind of a similar thing here you like you don't want to just be a passive observer of your agent
- 20:41you want to be very in the driver seat in the initial stages when you're building up your own
- 20:46set of skills uh you know obviously you can use something like PAC but if you're building your own
- 20:50set of skills it's very I think you opening up the all the tool calls and like reading the code and
- 20:57reading all the to uh the agent behavior and their thinking blocks is a really great way to see where
- 21:04they they fail, right? like what what you know where are they being done and then you can go and
- 21:10build a skill for that and then with verification how you trust it is it's I think it's also a very
- 21:16similar iteration loop uh where you know like I actually did the same process for verifying the
- 21:21verification skill where I actually get um so one thing that's interesting about eval is that you
- 21:28can sort of hill climb them meaning that uh your eval can produce a score right uh a score that you
- 21:36can get your coordinator to produce uh but also comp uh you can have a judge agent of a different
- 21:41model to uh kind of cross reference and make sure that the first model is not being biased,
- 21:48right? The model that's judging all of the sub aents that are running the thing. Uh but you can
- 21:53also like hill climb. So meaning that you can you can use like /loop in cursor and you can say okay
- 21:59keep looping on this eval right until everything is 10 out of 10 as an example. Uh and I did the
- 22:06same the basically the same approach with the control skill. And so I kind of it was very it
- 22:10was very hands-off actually. Uh so you know I uh I kind of built I built that skill that way like the
- 22:16CLI and that skill. Um and over time it's gotten really good. Uh but yeah, it was definitely not
- 22:23super smooth at the beginning. It required a lot of iteration and I think there's an analogy here
- 22:30for me which is um well I make this analogy later in a different slide on my drawing here. Uh but I
- 22:38think of it like uh you know as a as a engineer now you're sort of more like you uh like maybe a
- 22:46manager or the analogy I like is like you're like a a chef in a restaurant. uh you know,
- 22:51you're the head chef. Uh you're not cooking all the food yourself anymore. You have a team
- 22:56of cooks, right? You have line cooks, you have a sue chef, you have, you know, all these different
- 23:00stations. Um and it's your job to really design the environment. You know, you you're in charge of
- 23:06setting up the kitchen. You're in charge of, you know, like giving tasks to different people. So,
- 23:14um yeah, it's a very interesting way of working. Uh but yeah, that's that's how I've basically
- 23:20built uh these verification skills. Yeah, just just one fault there on like to go try to go
- 23:26one layer deeper. So are you let's say we wanted to build um an eval or a skill for for something
- 23:33and we wanted to kind of get better on its own which is is what I think you're suggesting. Uh
- 23:38are you doing that in like a work tree kind of isolated with like the sub aents and and then the
- 23:43reviewer agent and and all that? Is it happening like in some type of cloud hosted environment?
- 23:48like what's the the more the practical steps if I wanted to go do this uh and like set up
- 23:52a verification system for something? What would I what would I do or where would I start? Um
- 23:58I think that uh the best place to start is local because you can observe you can definitely observe
- 24:04what your agents are doing. So, uh, if you're building a verification skill for yourself,
- 24:09uh, I would definitely start local and just have your agent bring up the application,
- 24:14whether it's like a CLI or, uh, desktop app or whatever. And so, you can actually observe, right?
- 24:19You can see how the agent is interacting with the the application. You can see it, you know, how it
- 24:26calls like the different APIs that that allow it to interact with the uh the application. Um but
- 24:35uh for me personally uh I have basically been kind of all in mostly all in on cloud agents because
- 24:41they're extremely powerful. Uh and the really powerful thing about cursor is the the cloud
- 24:47agents actually where if you spend a little bit of time setting up your environment these control
- 24:52skills these verification skills pay a huge amount of dividends because it's not just something that
- 24:58makes you as a single engineer better. It actually levels up your whole team uh and even your whole
- 25:05company because uh you can actually start thinking about cloud agents. You can start thinking about
- 25:09automations that automatically do things like uh I I get I kind of talk about this a bit later
- 25:17but I'll just kind of get into it. Uh where where you know for example like I talk a lot about this
- 25:23agent we have called Benny, right? who who uh you know takes all of the bug reports that we get and
- 25:30it automatically goes off in the cloud, opens up a cloud uh it's you know its desktop. It runs cursor
- 25:37in its own computer and it uses the same control skills to interact with the application and try
- 25:43to reproduce the bug uh or the user report, right? And this is so so powerful because at once I can
- 25:49immediately I I get so much information from this automatically like here in this example you can
- 25:54see that uh the Benny actually reproduced the bug uh but it's already fixed on main. So it actually
- 26:02confirms that we fixed this problem already and all I need to do is just release another build
- 26:06of of cursor. Uh so that's like huge information there that I didn't have to go off and sit with
- 26:12an agent you know and spend an hour trying to figure like is this fixed? this is not fixed. So
- 26:17you you you gain back so much time. Uh but you know everybody on my team benefits from this.
- 26:22Everybody in the company benefits from this. Uh so definitely think that uh you know keeping these uh
- 26:29using cloud agents is super powerful. Uh but yeah it's like a journey. You have to trust it first
- 26:35right before you you get to this point. And that's it goes back to what I was saying here where you
- 26:40know it's very hard. It's it's almost impossible. And I would definitely encourage you not to try
- 26:45to jump from, you know, like if you're still in this zone, you don't want to jump to like I'm
- 26:51going to spawn hundred of thous or thousands of cloud agents right now because you're just going
- 26:56to waste a lot of tokens. Um and it's going to be extremely expensive. Yeah. So just to kind of
- 27:02recap so far, basically the if we wanted to go on the journey that you've kind of gone on, it would
- 27:06be to start with verification, building some some skills and some some ways of determining that the
- 27:12agents are producing at least like correct code. Whether like you said, whether it's good code or
- 27:16not is maybe a separate question, but like it's it's technically solving the problem by looking
- 27:20at, you know, stack traces, looking at, you know, the the actual behavior in the app and so on.
- 27:25Um, and then once we trust it locally, then we can start to think about scaling into the cloud and
- 27:29running more agents that are picking up signals I guess on their own, right? So whether that's like
- 27:33a bug report that comes in or something, they can go and pick it up and solve the problem and
- 27:38give us back a PR. And then maybe the last step is like automerging the PRs, which uh is where you're
- 27:43at, [laughter] maybe not where everyone is at. Um, and then reviewing the one main, but um, is
- 27:48that is that about right? Yeah, exactly. I think yeah, that's why I drew this this uh this curve,
- 27:53right? because that this this basically describes my journey of you know when I started barely could
- 27:59use a couple agents and I was just observing every single thing. I think there's really no shortcut
- 28:03for going from here to there because this is really about your personal level of trust in
- 28:08agents, right? Um obviously, you know, as a as an engineer, you don't want to just slop code
- 28:13into production. So, how do you actually build up that trust? Takes um a lot of uh I guess taste
- 28:20and judgment. Um but uh you know, like I think plugins like Pstack definitely kind of help you
- 28:27uh get up to speed much quicker. Uh and so I guess it's like if you trust me and you trust
- 28:33PAC then in by extension you can maybe trust your agents but if you don't trust me and I I
- 28:39definitely would not encourage people to blindly trust me uh uh you know if you build up your own
- 28:46set of skills that you can obviously you know take a look at PAC and kind of fork it make it
- 28:51your own improve the skills definitely encourage that uh but for me it's really all about it just
- 28:57keeps coming back to trust you know every one of us here in this chat have a different standard for
- 29:02engineering. Uh and there are different things that are important for us in our codebase and
- 29:08uh when you are able to encode all of that into skills and you can verify that your agent is
- 29:13actually doing them that allows you to really kind of ascend this curve and um uh you know
- 29:20start automating things. Uh there's another piece I wanted to talk about um if there's more. Yeah,
- 29:26go for it. I I'll pick up more questions as I go. Yeah, I think there's a third part to this
- 29:30which I haven't talked about yet which is kind of an interesting one which is like refactoring and
- 29:35rewriting. Like one of the uh I guess most controversial one of the most controversial
- 29:41topics in the industry I think is like should you rewrite your app or not? Um because I think
- 29:48engineers are very prone to this where especially when you join a company you come in and you see
- 29:53like the code base and you're like oh man this is [ __ ] like who wrote this code you know it's
- 29:58terrible I want to rewrite the whole thing is a very common inclination and I think a lot of you
- 30:04know before agents um and I guess arguably even now people will definitely discourage you from
- 30:09re rewriting stuff but I'm actually here to make a case for why you might want to consider it. Um
- 30:16because I think it really depends. Uh you know uh brownfield applications I think are actually in a
- 30:24pretty good spot especially if they're set up well already. Uh and like recently I've been talking to
- 30:29some people but uh you know I I was just observing I I just noticed this parallel which is that a lot
- 30:37of big tech company problems are now everybody's problems. Um, and the big tech company problem,
- 30:43you know, like when I was working at Meta, like we had this giant monor repo, we had like,
- 30:48I don't know, tens of thousands of engineers just, you know, like banging on their keyboards and and
- 30:53shipping code. And a lot of really great engineers at Meta. Uh, but, uh, I'll say like, you know,
- 31:01you'll be surprised that the code quality is actually not that good. Um, and so I often joke
- 31:06that like, you know, before AI slop, we had human slop. Um and so uh you know I think a lot of big
- 31:12tech infra like uh like what Meta has or Google you know you know really big tech companies are
- 31:19actually designed for that where you you're sort of like you're catering to the the you know like
- 31:24uh this sounds so bad to say but like the the least capable engineer on your team
- 31:28right you build you build frameworks you build conventions you build guard rails you know you
- 31:34restrict credentials so that you know your intern doesn't wipe your production database Um there's
- 31:41uh you know if you have that level of infra already I think your agents can actually already
- 31:46do a very solid job right because they have the the guard rails are already in place for agents
- 31:53to not cause havoc or not cause too much havoc uh in your codebase. Um and you can always add
- 31:59more you know guard rails. Uh but I think like green field applications especially are you know
- 32:04like the brand new applications are like the biggest risk in my opinion. Uh and also the
- 32:10greatest opportunity because you know if you vibe code a project uh a prototype um like we did for
- 32:16Grockbot you know Grockbot was spun up very very very quickly. Um and if you if you haven't heard
- 32:22of of Grockbot it's like our a new application we just launched yesterday. Uh it's it's really cool.
- 32:28uh lets you orchestrate your create like individual agents that have their own identity
- 32:33and you can kind of orchestrate them. It's super cool. Definitely check it out. U but yeah, that
- 32:37was it's like a very it was a very green field application like most prototypes are so like vibe
- 32:43coded very quickly. Humans were not reading the code at all. And uh I had this tweet recently uh
- 32:51where I said something about organic architecture. Um maybe I'll find it. Uh but the idea is that uh
- 33:01when you have a completely vibe coded application, you essentially have no guard rails whatsoever. So
- 33:07uh your agents when you give them a task, they will just solve it in whatever method is the
- 33:14most convenient. And over time you get into this uh situation where you have a codebase that is
- 33:20spiraling out of control because you don't understand it. uh your agents understand it
- 33:26I guess in a way but like they've built something that is you know optimized for short for shortcuts
- 33:32uh and uh you know it will you will suffer you'll have a lot of of issues with that application
- 33:40uh so I think starting your codebase with uh like very strong constraints is very much needed uh
- 33:48because like when you have a codebase that you can trust right when you have guardrails that actually
- 33:53help you uh uh help your agents write good code. You can get into the you know like into this part
- 34:00of the curve where I I where like I I said you know I woke up today and I had like 20 PRs merged
- 34:07u by my agents and that's because I invested a lot a lot of time uh over 600 PRs I I I calculated
- 34:14yesterday uh when I refactored all of Grockbot to this new architecture that I've been building. Um,
- 34:23and yeah, I've gotten to a point where I I don't really look I really don't look at
- 34:28the code anymore. And um, I say that not just, you know, to sell you tokens, but because I, you know,
- 34:35it it it took a lot of work to get to that point. I spent a lot of tokens to get the codebase to
- 34:40this point where I no longer have to look at it. Uh, but I'm very excited because, you know,
- 34:45of the potential where, you know, it's not just this doesn't just benefit me. It benefits everyone
- 34:50contributing to Grockbot and it also empowers you know designers and product managers and you know
- 34:57pe uh even GTM people to add features to Grockbot and I don't have to worry you know I don't have
- 35:03to to wake up at night in in the middle of the night and worry like oh [ __ ] someone's just
- 35:07merged a perf regression right I have a ton of constraints and CI like it's actually very
- 35:13annoying to write code in in graph web but like agents absorb all of that annoyance Um but yeah,
- 35:19I'm happy to talk about what exactly that is. Um yeah, I think one question um yeah, before we get
- 35:27into the this part here is just around that element of like what your your your CI looks
- 35:32like or maybe some of the constraints and then also like the average PR size. I saw a question
- 35:35about that earlier just to give people you know kind of a a glance. It doesn't have to be like
- 35:40mathematically average but just uh you know like what generally the size of the a PR is. um if it's
- 35:46only a couple lines of code or you know um yeah um I think it depends uh let me I'm trying to do
- 35:55this in a way where I'm not going to like you yeah you don't have to share the actual number like an
- 36:00actual average I think this is fine benchmark but like we have so okay this is not that interesting
- 36:07but uh well fun fact is that virtualization in grockbot and in uh cursor is actually powered by
- 36:15uh Pretext uh which is a sort of new library that someone's built. Um that's really interesting. You
- 36:22should you should check it out, but that's not really that important. Uh I think the average
- 36:26PR size I actually don't know. I I don't know if I want to click on these. Uh I probably can,
- 36:33but I would say like they can range anywhere from a few hundred lines or 50 lines to like a thousand
- 36:39depending on what the thing is doing. Uh, so like here I'm actually like deleting a bunch of files.
- 36:45So I expect that it's just like mostly deletion. Uh, but yeah, it kind of varies. There's no like
- 36:52Yeah, there's no like hard cap or hard limit. Are they're all like 50 line PR? There's no hard cap.
- 36:55Yeah, there's definitely no hard cap, but I I do encourage my agents to split up their work into
- 36:59multiple PRs. Uh I do that mostly because uh I like I like the idea of the I guess maybe this is
- 37:08much harder to do now as in the world of agents and you have like so many commits but I like
- 37:13the idea that you know the git history is a very rich source of context. Uh, and I like the I like
- 37:19each PR to sort of atomically describe what that small piece of thing is doing, which also makes
- 37:26it easier for me to revert changes and like figure out, you know, oh, I shipped a bug and it's just
- 37:30it's here, right? It's not in this 40,000 line PR where you who knows what landed in there. Uh, but
- 37:38I don't have a hard cap on PR size. Cool. And then um yeah, also quick question on like CI. So again,
- 37:46you don't have to go into like uh the screen share of like your CI does, but just generally would you
- 37:50describe what the CI kind of looks like uh or how strict it is? Uh yeah. So uh well specifically
- 37:58for Grockbot. So Dune is the is the sort of cheeky code code name for the architecture that
- 38:04we've built for Grockbot. Um the CI looks pretty annoying because there's checks for everything.
- 38:13So like literally I have um uh well if you've written any react for example you know you know
- 38:18that one of the biggest foot guns in react is use effect. Uh so in uh Dune and in graphbot we've
- 38:26banned use effect. So Dune is just you can the the mental model of what Dune is uh you can kind of
- 38:32think of it as like Nex.js JS for uh electron apps and it's designed for agents to write
- 38:38uh and it's like custom for you know our agent powered applications. Um so the CI checks are
- 38:45very like specific to that like you know don't use use effect it's it's it's banned like CI will fail
- 38:51uh and yell at you. We have like some of the more interesting ones that people might raise eyebrows
- 38:56is like I actually ban code comments as well uh which is very interesting. Uh, but I've noticed
- 39:03that 99% of the time agents just write code comments that kind of describe some historical
- 39:10thing that is actually totally irrelevant to the code. Um, like it will often say like, you know,
- 39:15oh, Lauren said you should never do this and it's now in in a code comment. I'm like, what? Like
- 39:19why? What? That was I didn't say that as like a durable, you know, global rule. I just meant like
- 39:24your this PR sucks and you should change that part. agents don't really understand us that
- 39:31well surprisingly uh and or they kind of assume too much and they kind of do things in like very
- 39:37stupid ways. So like yeah we just ban everything everything you can imagine like the agents are bad
- 39:43at we ban. Uh so one example that we actually suffer a lot in the agents window is we have
- 39:51uh you know you if you've used the agents window you've definitely seen performance issues and you
- 39:55know we're constantly trying to fix them. Uh but it's like a it's a never- ending struggle
- 40:01because there's so many pull requests that get merged. Every any one of them could just regress
- 40:06performance or stability or reliability. Uh you know the agents window doesn't have this
- 40:10architecture yet. I plan to do bring this learning back there and kind of refactor everything there.
- 40:17uh but uh it just regresses super often uh because uh there's just one example is like we have very
- 40:24poor um isolation between processes. So like on you know on on Electron you have a renderer
- 40:30thread that renders your UI but you also have like a main thread that you can run other code that you
- 40:35know doesn't need to block the renderer. Um, but we do a poor job of separating those things and so
- 40:42often times you just accidentally have code that gets pulled into running on the renderer thread
- 40:47and then all of a sudden you're competing with the the renderer that you know that has a very if you
- 40:53want like 60 fps you have to every frame that gets drawn has to be done in 16 milliseconds.
- 40:59So very very small you know deadline per frame uh if you want you know a very smooth product. Uh and
- 41:05when you start building bringing in accidentally bringing in you know things that are like very
- 41:10computationally heavy or they have a lot of IO uh then you just get into like a lot of jank your
- 41:17FPS really drops. You start uh you know losing frames. You get long tasks that take more than 16
- 41:23milliseconds and you just get this really choppy experience. So all of those patterns that we've
- 41:28learned basically building electron apps we've encoded into this framework and it becomes like
- 41:34a hard failure. So I literally in in grabbot we literally have a directory called electron
- 41:39main electron renderer and we have a import uh CI guess where we actually check the dependency graph
- 41:48to make sure you're not accidentally importing code from one directory to another. Uh so that's
- 41:54enforced by CI um as well as bug bots uh which is our which cursors um like code review tool
- 42:02that runs on CI uh you know in our agents MD it's everywhere like so I I I I um I have this thing
- 42:10here where I I talk about like um you know like there are multiple layers I think for building a
- 42:16good codebase. Uh obviously the codebase is one where uh if you have an architecture like this
- 42:22where it's extremely strict uh you know the the the way to build features is very conventional
- 42:28that's like the strongest strongest level of enforcement because agents just love to copy
- 42:33existing patterns. So uh one example of this in Rockbot is like we have this these concepts
- 42:39called like a feature and we have entry points and transcript cards like oh you know the cards
- 42:44that you see in the chat these are all like like nouns I guess in in the framework and so there's
- 42:50a very conventional way of creating them and so like a feature is all in in a single directory as
- 42:57an example and so all of the code that contributes to that feature lives in one directory so it's all
- 43:02collocated in one place makes It's super easy. You know, agents don't have to like uh grap around and
- 43:08try to figure out like where all the things are. It just looks at the feature and like,
- 43:12oh, okay, I'm working on the onboarding feature in Grockbot. Uh, I'm just going to work in this
- 43:17directory. And for 80% of the work, it's mostly just very encapsulated there. [clears throat] But,
- 43:23uh, like it's like very it's like designed again for you know like the dumbest agent like you don't
- 43:29have to think, right? the the the one of the key principles I have for this framework is like the
- 43:36shortest the shortest path is the best path. So uh because that plays exactly to how agents love to
- 43:43write code is like they like to take shortcuts really you know they they'll find the quickest
- 43:47way to solve the problem. So why not make that the best way to solve the problem? Uh so I I
- 43:55probably won't get into all the specific details. Um and uh the the this framework is really more
- 44:01of a collection of ideas and principles rather than something that will open source. Uh you can
- 44:06you can you know screenshot this I guess if you want and uh tell your agent to uh do some build
- 44:11build something like this for you too. Um yeah, but it's really all about the layers uh you know
- 44:18like the the codebase is one part with features uh and directories and you know import or blocking
- 44:25import dependencies uh that shouldn't be imported uh but and and it all enforces that and static
- 44:31analysis. So like uh there's CI checks, we have a lot of lints for bad patterns that we observe.
- 44:39uh compiler diagnostics uh there's also rules and bugbot which are um I think like three four five
- 44:46are more soft right these two actually make make CI red right so that you know there's a
- 44:53hard constraint where the agent can't just write crappy code for rules and skills and bogbot your
- 45:02agents can still forget right you can still or it may not always consistently apply them So,
- 45:08I like to layer them, but I don't I don't like to rely on them as the only source of
- 45:15enforcement because it's very very soft, right? And if you if you only have rules and bug bar and
- 45:20skills and a style guide for your code, you will it's only a matter of time before your
- 45:24codebase looks like complete trash. I'm sorry to say that, but uh I definitely recommend yeah like
- 45:30you know investing in you know things that can be hard and forced, right? And this is why you
- 45:37know maybe uh the choice of text stack that you use is also very important. Um like I think for
- 45:42example Rust is sort of making you know it's like getting super popular again. Uh because
- 45:48the compiler is so strict right the compiler enforces so many different things. you know,
- 45:53there's a borrow checker that you have to appease and if as long as you make sure your agents don't
- 45:58write unsafe code blocks, uh you can more or less feel somewhat confident that if the code compiles,
- 46:04it's probably works and it's good. Uh but you see, it gives you that level of trust and confidence
- 46:10that you as a human engineer no longer need to go and check it yourself. you know, you you rely on
- 46:18code and static analysis to actually make that uh a lot smoother. Um, and I I guess the worst part,
- 46:27the worst place to be in is if you are stuck in code review land where you actually enforce all of
- 46:33the constraints, the invariance in your codebase by literally the human person saying, you know,
- 46:39reading the code and like, okay, you should not do this, right? Every time you have to do that, you
- 46:44should consider that as a code smell, like a anti- pattern and you should say, "Okay, instead of me
- 46:49commenting on the PR, how do I turn this into a hard rule, right? How do I turn this into a lint
- 46:56rule? How do I turn this into a CI failure? Or how do I even categorically eliminate this problem uh
- 47:02entirely?" Uh I I can talk about another migration I've done, but I'll probably pause here. Sure.
- 47:09Yeah. I feel like that's that's where I am to be honest. is is what you're describing right now,
- 47:13which is that like I don't have all of these rules. So, I have some things to go do after this
- 47:17session in terms of being able to scale my agents. I'm I'm definitely on like the uh you know, maybe
- 47:23a couple of parallel ones locally stage. So, like two to three locally, and I'm sure most people
- 47:27here are on the same. So, uh yeah, I know we only couple minutes left. Lauren, was there anything
- 47:32else that you wanted to to highlight? Obviously, there's lots of questions, so I can grab more,
- 47:36but I want to give you a few minutes if there's anything else you want to talk Well, I think I've
- 47:39been yapping for quite a lot, so I'm maybe let's just do questions. Okay, cool. Uh, one question
- 47:44that had uh a couple of uh came up a couple times was just around like token usage. So, the the
- 47:50question is like is is what you're describing a realistic thing for people who are on, you know,
- 47:56uh a normal set of token usage. They don't have, you know, basically unlimited tokens uh to work
- 48:01with. I think that's a really good point. I mean like obviously you know I work at a AI lab where
- 48:07we have unlimited tokens. So uh I definitely cannot say that you know this is something
- 48:14everyone should do in the exact same way that I did it. I think it's possible to get to this point
- 48:19without you know breaking the bank. But you know if you're like an engineering leader or you know
- 48:25you're you have a startup that you lead um I think to me it's a question of ROI. Um, and it's like,
- 48:32uh, yes, you spend a lot of money on tokens in the upfront stage, you know, like refactoring
- 48:39a codebase is going to take a lot of tokens. Uh, adding all these things uh, is going to
- 48:44take a bunch of tokens. But if we're heading to a world where agents are writing all the code and,
- 48:50you know, you want to be very lean, right? You don't want to have to hire, you don't want to be,
- 48:55you don't want to become like meta, right? Like I mean like in terms of you don't want
- 48:58to become a 10,000 person engineering org because I mean that's a cool problem to have but also you
- 49:05have so much overhead. There's like planning you know like you it's it's a personally I I wouldn't
- 49:12uh it it's not super fun but um I think you want to stay very nimble right and you want to you want
- 49:18to be like agents are all about allowing you to do things that you couldn't do before. That's really
- 49:23to me like the value of agents, you know? It's not just storing tokens on every single little thing,
- 49:29but um to me like the thing I couldn't do before is like enforce this level of constraints in
- 49:36a codebase by myself, right? Like I'm just a single person, you know? Uh it would have taken
- 49:41me years to build this framework uh and do all the refactoring and test everything myself and verify,
- 49:50you know, like run imagine if there it was just me, right? you know in in pre- agent era just
- 49:55like running you know by it would take me so long right and my salary is pretty high right like so
- 50:01you know the the question I think an engineering leader might have is just then you know like what
- 50:07is there's a trade-off of do you hire someone to do this or do you spend the tokens to set up
- 50:13a code base so that even the the most naive right the dumbest agents can do a good job and when you
- 50:20actually get to this point like even agents that are not you know fable size do an excellent job
- 50:26of writing code and this pays a lot of dividends as well for me personally where I've empowered not
- 50:33just myself but again like PMS designers engineers who are not familiar with Grockbot to just
- 50:39contribute in a way that is sustainable so I think yeah it's definitely like a trade-off for sure you
- 50:47know like nothing is like free for sure uh and tokens are pretty expensive Uh but oh actually
- 50:53uh I I I don't know how many of you have seen this but we actually announced Grock 4.6 today. So very
- 50:59exciting finally out. Um so yeah, GRO 4.6 would be like a great it was very very smart. Uh it's
- 51:06really good on the on the benchmarks. Uh and it's the same the tokens uh well uh I hopefully I'm not
- 51:12saying this incorrectly, but uh I believe the cost per token is the same as 4.5. So you're actually
- 51:18getting more intelligence for the same cost. Uh I think this is an area that cursor tries to cursor
- 51:26and SpaceX AI try to really optimize for like that heredto frontier of you know cost versus
- 51:32intelligence. Uh you know we don't necessarily want to build the biggest model ever because that
- 51:37is extremely expensive to run. It's really about like how do you find that sweet spot right? you
- 51:42don't you don't need a giant model, but it's just super smart, right? And it's not very expensive
- 51:46for inference. Uh but um yeah, I think to kind of round it up, um I think it's like a it's it's
- 51:55there's a if you do your own analysis, I feel like it's pretty positive. It it'll be pretty positive
- 52:01that the ROI you get from investing in stuff like this uh just empowers not just yourself, but your
- 52:08whole team to be so much more productive, right? Like imagine if you have an army of engineers like
- 52:13me who are shipping so much improvements and and bug fixes uh you know every day, right? Like that
- 52:21is pretty exciting. Cool. Uh one last question before we wrap up. This one is for the people
- 52:26in product on the on the call. So let's say we do have an army of engineers who are shipping like
- 52:32Lauren. I'm just curious like how is the product team or other functions of your company keeping up
- 52:36given that like if you're shipping so quickly, have are they using AI more to do their jobs?
- 52:42Like as much as you can speak to that obviously you don't have like you're not in that role but
- 52:46just curious about how that works. Um I think this is where grathbot has been actually exceedingly
- 52:52powerful. uh where so before graphbot like you know uh obviously cursor only had cursor like we
- 52:59only had agents window we had a CLI we had an IDE and these are really like power user tools right
- 53:05like de they're designed for developers so it's very very developer centric you can do knowledge
- 53:10work in them but it like the UI is not really optimized for that so we actually didn't really
- 53:17have uh well I think like a lot of people like you know GTM product like they might have use cursor
- 53:23uh to do their work but it definitely wasn't like a delightful experience for them. Um I think now
- 53:29with grabbot uh it's become Grockbot is basically like the kusher moment for people who are not in
- 53:37tech in my opinion like it's like it's like a very very accessible way to use agents in
- 53:43a very comfortable very familiar interface. It looks like iMessage. Um, and it's very fun to,
- 53:49you know, you can give your agent a fun name. Uh, you can have you can kind of do orchestration with
- 53:54in a very like natural way where you can sort of, you know, each agent's like a person, right? Now,
- 53:59you got a team of agents like working on you have one one agent per account that you manage
- 54:02as an example. Or if you're a PM, you have, you know, you can have an agent that summarizes all
- 54:07the work that Lauren did last night and then now you know what I did, right? So I think our PMs
- 54:12are leveraging that a lot and they're shipping code too. Uh so you know like often times they
- 54:18will just say oh here's a bug I fixed it can you look at it and then I'll go review it and actually
- 54:22it's just perfect. I'm like okay stamp. Uh so uh that I think that shows that you know the the Dune
- 54:28architecture is holding up right the all the the really strict constraints allow people who are not
- 54:34experts in engineering to contribute at a high level. Uh so I'm I feel like I'm already seeing
- 54:40that pay off a lot where uh you know designers and PMs are just able to to to ship features directly.
- 54:48Um and that just makes the Grockbot team super fast, right? Where we can ship so quickly. Um
- 54:56and we have a lot planned. So, I'm very excited uh to, you know, uh to to ship more ship more
About this transcript
This page contains the full transcript of SpaceX AI engineer Lauren Tan part 2 by Ai insider, generated from the public captions YouTube serves with the video. The transcript has 10,284 words across 541 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.