SpaceX AI Engineer Lauren Tan: Part 1 — Transcript
Full transcript
- 0:00So, the first thing I want to talk
- 0:02about, it's pretty pretty serious, okay?
- 0:06I need all of you to take this
- 0:07seriously. Put your phones down, put
- 0:09your laptop down, put your water bottles
- 0:12down. We're done hydrating. Are you all
- 0:15ready?
- 0:20Very excited to announce.
- 0:23[applause]
- 0:25Uh, you can't end PM insult yet. We'll
- 0:27we'll release that later today. Okay. So
- 0:29don't don't take your laptops out. Um
- 0:32but yes very exciting milestone and
- 0:35after I do the release you can get
- 0:37started your applications by installing
- 0:39babel plugin react as well as the new
- 0:40eslint plugin. Uh okay so I assume you
- 0:43can see my screen.
- 0:44>> Yes. Yeah we're good.
- 0:46>> So yeah today yeah I think I think the
- 0:48big theme for me as I've been using
- 0:51agents to write code and I'm sure a lot
- 0:53of you have had the same experience as
- 0:55well is how do you trust it? You know,
- 0:57especially if you are an engineer that's
- 1:00been writing code for a very long time,
- 1:02you have a lot of opinions and lessons
- 1:05that you've learned about doing good
- 1:07engineering. And when you see agents
- 1:10just, you know, winging it and, you
- 1:12know, guessing, hallucinating,
- 1:14uh, you know, confidently stating that
- 1:16they found the smoking gun, uh, for the
- 1:19hundth time, uh, but it's actually not
- 1:21the real problem. You lose a lot of
- 1:23trust. And when you lose when you don't
- 1:26have much trust in your agents, I feel
- 1:29like you you really can't get the most
- 1:30out of them. And for me, the parallel is
- 1:33like with management. Uh so if I'm an a
- 1:36manager, an engineering manager of a
- 1:38team and I have a bunch of, you know, I
- 1:41have a team of engineers uh on my team
- 1:44and I don't trust them, then the mode of
- 1:46operation I'm going to be in is going to
- 1:48be like micromanagement, right? I'll
- 1:50have to spend a lot of time looking over
- 1:52my reports shoulders and checking that
- 1:55they're doing their work well, you know,
- 1:57like they're not shipping bugs to
- 1:59production.
- 2:01And so I drew this chart because uh it's
- 2:05not it's not a very scientific chart,
- 2:06but like this is how I imagine myself
- 2:10and my journey through using agents. So
- 2:13you know like fast forward or back
- 2:16forward uh or fast back uh fast
- 2:20backwards like a year or so when you
- 2:23know nobody was or not many people were
- 2:24using agents to code. Uh I think you
- 2:29uh you know get into this mode where you
- 2:31are
- 2:33in very heavily in the loop with one or
- 2:36several like a handful of agents and you
- 2:39find yourself just constantly figure uh
- 2:41you know trying to understand what your
- 2:42agents are doing uh and you're very very
- 2:45in loop. You're watching every single
- 2:47output. you are sitting there prompting
- 2:50um and you really can't parallelize
- 2:53beyond that because you don't again you
- 2:55don't have that trust right you can't go
- 2:57to a 100 agents uh like spawn 100 agents
- 3:00when you don't even trust the output of
- 3:02one agent
- 3:04so over the past 5 months I feel like
- 3:07I've really been able to uh like ascend
- 3:10this trust curve and now I'm at the
- 3:14point where uh I actually have this
- 3:17sounds kind kind of scary to say this
- 3:18and it makes me sound like a slop
- 3:20artist, but I I promise I'm not, but I
- 3:22actually have my agents now um
- 3:24automerging PRs for me. Uh which is like
- 3:27a wild thing to say, but um like I woke
- 3:30up today and there were like 20 PRs
- 3:33landed and I just reviewed them on main
- 3:35like they were already landed and they
- 3:37were good. Uh so how did I get to that
- 3:40point is basically what I wanted to talk
- 3:43about today.
- 3:45Uh and again like yeah feel free to jump
- 3:47in if you have questions Colin. Um but
- 3:51uh oh yeah of course I got to show this
- 3:53this chart. Uh where uh
- 3:59oh do do not trust to someone requested
- 4:02to control my computer. Uh probably
- 4:04won't do that. Uh but yeah, so this
- 4:07chart I think I I'm I'm sharing this
- 4:10chart not to kind of like flex but to
- 4:12kind of show like the journey like so
- 4:15you can see like the curve like it sort
- 4:16of like inversely matches the
- 4:19contributions I've been able to land at
- 4:20cursor. So I joined five months ago and
- 4:24five months ago like I you know my first
- 4:26month I was like not very productive
- 4:28because I was ob you know I was learning
- 4:29the codebase didn't know what the heck
- 4:31was going on and as I got more confident
- 4:35in in my agents uh I've really been able
- 4:37to kind of ramp up my productivity. Uh
- 4:41and again like yeah like last month I
- 4:43shipped a thousand PRs which is
- 4:46ridiculous. Uh, and then this month
- 4:48we're only on the 12th. I'm already at
- 4:51like almost 800 PRs landed. Uh, so the
- 4:56velocity is definitely high and you you
- 4:58I'm sure a lot of you will definitely be
- 5:00questioning like how how much of this
- 5:02code is actually good. Um, and I think
- 5:04yeah, like that's definitely fair to
- 5:06question.
- 5:08Um, but uh, yeah, I think I think if you
- 5:12set up your agents well, you can
- 5:15definitely get to a very similar level.
- 5:19Um, and so I'm going to talk about how
- 5:20we do that.
- 5:23Uh so for me I think I'm curious like I
- 5:27guess call in your experience as well
- 5:28but uh for me I think the most important
- 5:33skill that you should have in your
- 5:35toolbox when you work with agents is
- 5:38verification. Uh and by verification I
- 5:41mean the ability for an agent to
- 5:43actually run the code uh or take CPU
- 5:48traces or heap snapshots or uh you know
- 5:52open an iOS simulator whatever you know
- 5:55however your application is exposed to
- 5:57your users it can do the same thing and
- 6:02uh run it for real and actually test and
- 6:04verify it don't work because that's the
- 6:07thing that really closes the loop.
- 6:09uh it doesn't guarantee your agent
- 6:11writes good code uh but it allows them
- 6:14to at least write correct code uh which
- 6:17is a big a really big step forward for
- 6:19being able to trust your agents. Um
- 6:24I will I can share one example that we
- 6:26have uh within cursor
- 6:30uh oops
- 6:33where let me open this
- 6:43make this uh make me full screen. There
- 6:46you go. Uh so for the for cursor's agent
- 6:51window uh so this is actually an
- 6:54interesting story but uh when I joined
- 6:56cursor 5 months ago uh they're actually
- 7:00uh well I was supposed to join a
- 7:02different team I was supposed to join
- 7:03like the cloud agents team uh but then
- 7:05since I have a lot of experience working
- 7:07on react and agents window is a react
- 7:10application uh I was
- 7:14uh I was asked to basically help out
- 7:16with the agent window work. Uh but um
- 7:23there wasn't really a lot of like skills
- 7:25to help me. So I just found myself like
- 7:27okay uh agents window is going to launch
- 7:29in like a week, right? We have a really
- 7:31tight deadline and um there was uh you
- 7:35know I was just sitting there like okay
- 7:37I'm going to open up the performant the
- 7:38the Chrome dev tools and just like take
- 7:40a trace look at it myself and try to
- 7:43make sense of this flame graph. And keep
- 7:45in mind, I was just like in my first
- 7:47week, so I had no idea what I was
- 7:48looking at, no idea where, you know, I
- 7:50mean, I had some idea, but you know, the
- 7:52codebase was completely fresh to me. Uh,
- 7:55and I realized like my agent had no idea
- 7:58either, you know, like I would take a
- 8:00screenshot of the tree, I would download
- 8:01trace, I would send it to it, and it be
- 8:02like, yeah, it kind of looks like this,
- 8:04you know, uh, and it would like
- 8:06confidently state like it's this thing.
- 8:08And then I try to fix that and turns out
- 8:10that's not the actual thing.
- 8:13So this was a very very slow process and
- 8:15if you've ever done any like performance
- 8:17work yourself or you know just even
- 8:19development with an agent where you
- 8:21don't have a verification skill you are
- 8:24the verifier right you you're the
- 8:26bottleneck you you you tell your agent
- 8:28to do something and then it goes off and
- 8:30write some code then you open up your
- 8:32you know local dev build and then you
- 8:34start to say oh you know doesn't work
- 8:35then you got to copy paste screen uh you
- 8:37know screenshots or console errors or
- 8:40whatever uh and then your agent like
- 8:42slowly kind of like uh you know works
- 8:44with that and then tries to understand
- 8:46it and um fix the thing but then you
- 8:50were constantly just in the loop and and
- 8:52being a bottleneck. So there's really no
- 8:54way to parallelize. So the control glass
- 8:56skills are like one of the first skills
- 8:58I built uh for cursor uh and glass by
- 9:02the way is the code name for agents
- 9:03window that we use internally but it's
- 9:06just cursor I guess. Um, and so this
- 9:10skill uh is I guess the the the code
- 9:13itself is not super interesting. Your
- 9:15agent can very easily make one for you.
- 9:17uh where if if you're building an
- 9:19Electron app or a web app or even iOS uh
- 9:23applications uh you can teach your agent
- 9:26how to use like the ChromeDev tools
- 9:28protocol or through uh Apple has some
- 9:32utilities as well for running the
- 9:34simulator and taking traces and
- 9:36controlling programmatic control as
- 9:38well. Uh so that's really useful. Uh but
- 9:43one thing I want to talk about is uh the
- 9:48this thing
- 9:50uh where is the read me? Uh so this
- 9:53skill comes with this very unique
- 9:55feature called or not feature uh unique
- 9:58file called a feature map. And so the
- 10:02story then is like I built this skill
- 10:04and so now the agent was able to uh
- 10:06actually run the agent window and take
- 10:09traces and whatnot. Uh but it had no
- 10:12idea what what the agents window was. So
- 10:17um you know like someone would say like
- 10:18oh the the left sidebar is like laggy or
- 10:22something like that or you know the
- 10:24right side the the PR tab is not working
- 10:27and the agent would just be like kind of
- 10:28flailing around. It would spend a lot of
- 10:30time trying to like look up the code and
- 10:32you know where is this feature? How do I
- 10:33actually get to it on the UI which made
- 10:36it basically completely useless. Uh you
- 10:38know like we would I would run
- 10:41the skill locally and you know it would
- 10:43spawn a dev build uh but then it just be
- 10:47turnurning like I just try to click here
- 10:49it wouldn't know how to get to things um
- 10:52and it was just an awful experience uh
- 10:55so was putting arrows on my screen um so
- 11:00uh yeah this this feature map has been
- 11:03really useful uh because it teaches the
- 11:05agent how to get to all of the features
- 11:07that you have um and in PAC the plugin
- 11:11that I I've made uh if you search for
- 11:15PAC cursor on Google you you'll find it
- 11:18uh but there is a create verification
- 11:20skill in that plugin where it actually
- 11:22helps you set up something like this for
- 11:24yourself um including the feature map so
- 11:27it will actually explore the code and
- 11:29build up this initial feature map that
- 11:31tells your agent how to get to all of
- 11:33the different features that you have uh
- 11:36and this is extremely powerful because
- 11:38now like you have these user reports
- 11:40that come in uh you you can actually map
- 11:43even like a vague report or even a
- 11:45screenshot. So we have this uh
- 11:48internally at cursor where uh we have a
- 11:51slack channel with you know lots of
- 11:53people giving us feedback on the agents
- 11:56window and rockbot and whatnot. Uh, and
- 11:59often times the report is very bad like
- 12:02very low quality like someone will just
- 12:03put very often we get like a screenshot
- 12:06like and then someone just says question
- 12:07mark question mark question mark like
- 12:09what is this and you know like the
- 12:11without this your agent like I have no
- 12:13clue right but with a feature map like
- 12:16this it has a lot more context and
- 12:19understanding of how to actually
- 12:21navigate how to get to all the different
- 12:23features. Uh so like you know example
- 12:26like I guess like the sidebar like what
- 12:28is the sidebar uh you know like all the
- 12:30different sub features that are present
- 12:32in it um like from the user point of
- 12:35view here's where how do I get to it all
- 12:38the different keyboard shortcuts
- 12:40uh even like the the what do you call it
- 12:43the DOM elements or yeah like the
- 12:46attributes that you use for selecting
- 12:48things through the CDP uh are all there.
- 12:52So uh again yeah this is like really
- 12:54really powerful uh for for agents
- 12:58>> uh and pack ships uh that create
- 13:01verification skill but also a maintain
- 13:03verification skill uh so you can keep
- 13:06this up to date.
- 13:08>> Cool. Yeah, I was just going to ask how
- 13:09you created that. So do you mind sharing
- 13:10a little bit more about um that that
- 13:12process in the context of Pstack and
- 13:14maybe just what Pstack is for the folks
- 13:15who aren't familiar?
- 13:17Yeah. So, P stack is pretty interesting
- 13:19because uh well, first of all, the name
- 13:22is kind of goofy. like the P the P in P
- 13:25stack is like potato potato stack
- 13:28because I um so uh uh there is a pretty
- 13:34uh famous person Gary Tan who is the CEO
- 13:37of Y Combinator and he's come up with
- 13:40this plugin called Gstack uh Gary Stack
- 13:43and uh funnily enough we share the last
- 13:46name we have no relations uh but I
- 13:49thought it would be funny to kind of you
- 13:50know poke fun at Gary and make Pstack
- 13:53my version of of of of his plugin. Uh
- 13:57but kind of tailor it to my own set of p
- 14:00engineering practices.
- 14:03Uh but I honestly actually never set out
- 14:05to build PAC. Uh it just started with a
- 14:07bunch of skills, right? Like I started
- 14:09with that control glass skill and then I
- 14:11started with another skill like called
- 14:13how which I also noticed through like
- 14:16observing agents. Um, so like you know
- 14:19in the early days of me, you know,
- 14:21trying to climb this ladder, I was like
- 14:22super in the loop and I was basically
- 14:25nitpicking my agents to an extreme
- 14:27degree, I was like I would tell it um,
- 14:31you know, this feature has stopped
- 14:32working. Here's a bug report. Like why
- 14:35isn't it working?
- 14:37And very often the agent would just like
- 14:40confidently state like, "Oh, it has to
- 14:42be this, right? It has to be this
- 14:43thing." And I noticed like when I looked
- 14:46at the actual tool calls, I noticed it
- 14:48wasn't actually reading the code that I
- 14:51thought should be affected. And that
- 14:53made me just extremely suspicious. And
- 14:55at that point, I was like, I'm not going
- 14:56to I can't trust any this agent anymore
- 14:59cuz it's just it's just completely
- 15:00hallucinating. And I think
- 15:04I think it's very easy to just you know
- 15:06like build up that distrust and not and
- 15:08kind of feel helpless like you know you
- 15:11don't know how to help your agents
- 15:13succeed. But like again I think the the
- 15:16the management analogy is super helpful
- 15:18because like imagine if you were a
- 15:19manager of an engineering team and you
- 15:22had an engineer on your team who was a
- 15:24really good coder. No business context
- 15:26whatsoever. So you know they they just
- 15:28you just hired them and they they
- 15:29onboarded you know like 5 seconds ago.
- 15:33Uh and so how do you actually teach that
- 15:35person to be effective? So how you do
- 15:38that is through a skill. uh skill being
- 15:40just you know it's just markdown right
- 15:42but you know it encodes a lot of
- 15:44information instructions a lot of uh you
- 15:48can really draw out a lot of
- 15:51intelligence from an agent by well some
- 15:54people on Twitter call it like you know
- 15:55pull the agent to a different latent
- 15:57space which is kind of like a fancy way
- 15:59of just saying like since uh you know
- 16:01LLMs are sort of like they predict the
- 16:04next token uh when you give it some high
- 16:07quality tokens uh to begin with then you
- 16:09know it it can kind of pattern match on
- 16:12like a higher space that's you know
- 16:14smarter.
- 16:16Um so that's like a very interesting
- 16:19model there but yeah I built Pstack very
- 16:21very incrementally. Uh so uh started
- 16:25with just really observing how agents
- 16:27you know all the fail different failure
- 16:29modes of of that agents were having and
- 16:32every time I saw that I just okay I'm
- 16:33just going to make that a skill right
- 16:35like stop hallucinating actually go and
- 16:38search up look up the code use a lot of
- 16:40sub agents uh and yeah stop guessing
- 16:47>> yeah that makes sense one one kind of
- 16:48followup question here uh both from
- 16:50myself and from a bunch of people in the
- 16:51chat so
- 16:52>> I guess It's two two parts. So one is
- 16:54like how do you maintain these skills?
- 16:55So like the product changes over time.
- 16:57Obviously there's a lot of people who
- 16:58are shipping against the codebase. So
- 17:00how do these skills get maintained? Uh
- 17:02and then second to that is like how do
- 17:04you know when your verification is is
- 17:06good enough? Uh like and you know you
- 17:09can trust that the ver verification
- 17:10loops that you've built are going to I
- 17:12guess you trust that the outputs uh when
- 17:14they're done.
- 17:17>> Uh yeah maybe I'll talk about um I think
- 17:20somehat related. Maybe I'll start with
- 17:22this one first. So like how do I
- 17:25maintain these skills?
- 17:28So um if you're not familiar with this
- 17:31concept, [clears throat] an eval is
- 17:32essentially like a way to uh well I the
- 17:36mental model I have is like it's like a
- 17:37unit test for an agent. Um and uh you
- 17:41can actually make your own eval. You
- 17:44don't need like a special framework for
- 17:45them. You can build you can you can
- 17:47build one depending on like you know how
- 17:50scientific and how rigorous you want to
- 17:52be. Uh my screen is red.
- 17:55>> Yeah, there's a little button. Um sorry.
- 17:58Do
- 17:58>> you mind like disabling the drawing or
- 18:00something? I I can't see my screen.
- 18:02>> Yeah, sorry if you guys could not draw
- 18:04on the screen that'd be great. But um
- 18:05there's a little button in the
- 18:07>> I got a troll.
- 18:08>> Yeah, the little drop down.
- 18:11>> How do I clear?
- 18:13>> Yeah, you got it. Perfect.
- 18:16>> Um,
- 18:16>> good to know.
- 18:18>> Yeah. So, eval
- 18:21unit test your skills basically. And
- 18:24actually in Pstack, we ship uh under
- 18:26potato mode, there's a playbook if you
- 18:28search for it called Eval playbook. Um,
- 18:31and it's uh
- 18:33uh it's like not it's actually pretty
- 18:36pretty rigorous the way it's done. Uh,
- 18:38but um, essentially what I do is I spawn
- 18:41a lot of different sub agents. I have
- 18:43like my main coordinator agent uh come
- 18:46up with a rubric for uh what I want the
- 18:50skill to do. Um and then it spawns all
- 18:54these sub aents and it's it creates
- 18:56individual directories for them uh which
- 18:59are cleverly named to not let the sub
- 19:02aent know that it's being evaluated
- 19:05because uh agents can actually tell and
- 19:08when they do they change their behavior.
- 19:10Uh, but it does a bunch of stuff like
- 19:12that to um essentially yeah like test
- 19:16whether or not the skill I'm making or
- 19:19changing is actually doing what I think
- 19:21it does. Um, and one of the really nice
- 19:23things about cursor is that we are we we
- 19:26support so many different models. So you
- 19:28can actually eval your skill across all
- 19:30sorts of different models um and you
- 19:33know get a sense of how well it performs
- 19:36across that different matrix. um
- 19:39especially for the models that you use.
- 19:42Uh so I do this a lot. Every time I I
- 19:44modify a skill, I will run one of these
- 19:47uh like the EBAL playbook uh and make
- 19:49sure that you know it's actually leading
- 19:52to a result I want. Uh but I will say
- 19:55like
- 19:56maintaining skills is actually pretty
- 19:58hard. Uh it requires I think a lot of
- 20:01taste and observation. So you kind of
- 20:04need to be very good at being a backseat
- 20:07driver. You know what I mean? Like if
- 20:09you do pair if you ever done pair
- 20:10programming for example, uh and you
- 20:12watch a coworker code and you just like
- 20:15you you could probably do this better.
- 20:16You know, you could do you know like why
- 20:18did you not do this, right? You you ask
- 20:20a lot of questions to your coworker. And
- 20:22it's kind of a similar thing here. You
- 20:23like you don't want to just be a passive
- 20:25observer of your agent. You want to be
- 20:27very in the driver seat in the initial
- 20:29stages when you're building up your own
- 20:31set of skills. uh you know obviously you
- 20:33can use something like PAC but if you're
- 20:35building your own set of skills it's
- 20:37very I think you know opening up the all
- 20:40the tool calls and like reading the code
- 20:42and reading all the uh the agent
- 20:45behavior and their thinking blocks is a
- 20:48really great way to see where they they
- 20:50fail right like what what you know where
- 20:53are they being done and then you can go
- 20:55and build a skill for that and then with
- 20:58verification how you trust it is it's I
- 21:00think it's also very similar iteration
- 21:02loop uh where you know like I actually
- 21:05did the same process for verifying the
- 21:07verification skill where I actually get
- 21:11um so one thing that's interesting about
- 21:13eval is that you can sort of hill climb
- 21:16them meaning that uh your eval can
- 21:19produce a score right uh a score that
- 21:21you can get your coordinator to produce
- 21:24uh but also comp uh you can have a judge
- 21:26agent of a different model to uh kind of
- 21:30cross reference and make sure that the
- 21:33first model is not being biased, right?
- 21:35The model that's judging all of the sub
- 21:37aents that are running the thing. Uh but
- 21:39you can also like hill climb. So meaning
- 21:41that you can you can use like /loop in
- 21:44cursor and you can say okay keep looping
- 21:46on this eval right until everything is
- 21:5010 out of 10 as an example. Uh and I did
- 21:52the same the basically the same approach
- 21:54with the control skill. And so I kind of
- 21:56it was very it was very hands-off
- 21:58actually. Uh so you know I uh I kind of
- 22:01built I built that skill that way like
- 22:03the CLI and that skill. Um and over time
- 22:08it's gotten really good. Uh but yeah it
- 22:10was definitely not super smooth at the
- 22:12beginning. It required a lot of
- 22:14iteration and I think there's an analogy
- 22:17here for me which is um well I make this
- 22:20analogy later in a different slide on my
- 22:23drawing here. Uh but I think of it like
- 22:27uh you know as a as a engineer now
- 22:30you're sort of more like you uh like
- 22:33maybe a manager or the analogy I like is
- 22:35like you're like a a chef in a
- 22:38restaurant. Uh you you're the head chef.
- 22:40Uh you're not cooking all the food
- 22:42yourself anymore. You have a team of
- 22:44cooks, right? You have line cooks, you
- 22:46have a sue chef, you have, you know, all
- 22:47these different stations.
- 22:49Um and it's your job to really design
- 22:52the environment. you know, you you're in
- 22:54charge of setting up the kitchen. You're
- 22:55in charge of, you know, like giving
- 22:58tasks to different people. So,
- 23:02um yeah, it's a very interesting way of
- 23:05working. Uh but yeah, that's that's how
- 23:08I've basically built uh these
- 23:10verification skills.
- 23:11>> Yeah, just just one thought there on
- 23:13like to go try to go one layer deeper.
- 23:15So are you let's say we wanted to build
- 23:18um an eval or a skill for for something
- 23:22and we wanted to kind of get better on
- 23:24its own which is is what I think you're
- 23:25suggesting. Uh are you doing that in
- 23:28like a work tree kind of isolated with
- 23:30like the sub agents and and then the
- 23:32reviewer agent and and all that? Is it
- 23:34happening like in some type of cloud
- 23:35hosted environment? Like what's the the
- 23:37more the practical steps if I wanted to
- 23:39go do this uh and like set up a
- 23:41verification system for something? What
- 23:42would I what would I do or where would I
- 23:44start?
- 23:46Um I think that uh the best place to
- 23:49start is local because you can observe
- 23:53you can definitely observe what your
- 23:54agents are doing. So uh if you're
- 23:56building a verification skill for
- 23:58yourself uh I would definitely start
- 23:59local and just have your agent bring up
- 24:02the application whether it's like a CLI
- 24:04or a desktop app or whatever. And so you
- 24:08can actually observe right you can see
- 24:09how the agent is interacting with the
- 24:12the application. you can see it, you
- 24:15know, how it calls like the different
- 24:17APIs that that allow it to interact with
- 24:20the uh the application.
- 24:23Um, but uh for me personally uh I have
- 24:27basically been kind of all in mostly all
- 24:30in on cloud agents because they're
- 24:31extremely powerful. Uh and the really
- 24:35powerful thing about cursor is the the
- 24:37cloud agents actually where if you spend
- 24:39a little bit of time setting up your
- 24:40environment
- 24:42these control skills, these verification
- 24:44skills pay a huge amount of dividends
- 24:46because it's not just something that
- 24:49makes you as a single engineer better.
- 24:52It actually levels up your whole team uh
- 24:54and even your whole company because uh
- 24:57you can actually start thinking about
- 24:58cloud agents. start thinking about
- 25:00automations that automatically do things
- 25:03like uh I I get I I kind of talk about
- 25:06this a bit later, but I'll just kind of
- 25:09get into it. Uh where where you know,
- 25:11for example, like I talk a lot about
- 25:13this agent we have called Benny, right?
- 25:16who who uh you know takes all of the bug
- 25:19reports that we get and it automatically
- 25:22goes off in the cloud, opens up a cloud
- 25:24uh it's you know its desktop. It runs
- 25:27cursor in its own computer and it uses
- 25:30the same control skills to interact with
- 25:33the application and try to reproduce the
- 25:34bug uh or the user report, right? And
- 25:37this is so so powerful because at once I
- 25:40can immediately I I get so much
- 25:42information from this automatically like
- 25:44here in this example you can see that uh
- 25:46the Benny actually reproduced the bug uh
- 25:49but it's already fixed on main. So it
- 25:53actually confirms that we fixed this
- 25:54problem already and all I need to do is
- 25:56just release another build of of cursor.
- 26:00Uh so that's like huge information there
- 26:02that I didn't have to go off and sit
- 26:04with an agent, you know, and spend an
- 26:05hour trying to figure like is this
- 26:06fixed? this is not fixed. So you you you
- 26:09gain back so much time. Uh but you know
- 26:12everybody on my team benefits from this.
- 26:14Everybody at the company benefits from
- 26:15this. Uh so definitely think that uh you
- 26:20know keeping these uh using cloud agents
- 26:22is super powerful. Uh but yeah it's like
- 26:25a journey. You have to trust it first
- 26:28right before you you get to this point.
- 26:30And that's it goes back to what I was
- 26:32saying here where you know it's very
- 26:34hard. It's almost impossible. And I
- 26:36would definitely encourage you not to
- 26:37try to jump from, you know, like if
- 26:40you're still in this zone, you don't
- 26:42want to jump to like I'm going to spawn
- 26:44hundred of thousand or thousands of
- 26:46cloud agents right now because you're
- 26:48just going to waste a lot of tokens. Um,
- 26:51and it's going to be extremely
- 26:52expensive.
- 26:53>> Yeah. So just to kind of recap so far,
- 26:55basically the if we wanted to go on the
- 26:57journey that you've kind of gone on, it
- 26:59would be to start with verification,
- 27:02building some some skills and some some
- 27:04ways of determining that the agents are
- 27:05producing at least like correct code.
- 27:08Whether like you said whether it's good
- 27:09code or not is maybe a separate
- 27:10question, but like it's it's technically
- 27:11solving the problem by looking at you
- 27:13know stack traces, looking at you know
- 27:15the the actual behavior in the app and
- 27:17so on. Um, and then once we trust it
- 27:19locally, then we can start to think
- 27:21about scaling into the cloud and running
- 27:22more agents that are picking up signals,
- 27:25I guess, on their own, right? So whether
- 27:26that's like a bug report that comes in
- 27:28or something, they can go and pick it up
- 27:29and solve the problem and and give us
- 27:31back a PR. And then maybe the last step
- 27:34is like automerging the PRs, which uh is
- 27:36where you're at, maybe not where we want
- 27:38to
- 27:39>> um and then and reviewing the one main,
- 27:40but um is that is that about right?
- 27:43>> Yeah, exactly. I think yeah, that's why
- 27:44I drew this this uh this this curve,
- 27:46right? because that this this basically
- 27:49describes my journey of you know when I
- 27:51started barely could use a couple agents
- 27:53and I was just observing every single
- 27:55thing. I think there's really no
- 27:57shortcut for going from here to there
- 27:59because this is really about your
- 28:01personal level of trust in agents,
- 28:03right? Um obviously, you know, as a as
- 28:06engineer, you don't want to just slop
- 28:07code into production. So, how do you
- 28:09actually build up that trust? Takes um a
- 28:12lot of uh I guess taste and judgment. Um
- 28:16but uh you know, like I think plugins
- 28:19like Pstack definitely kind of help you
- 28:22uh get up to speed much quicker. Uh and
- 28:25so I guess it's like if you trust me and
- 28:27you trust PAC then in by extension you
- 28:30can maybe trust your agents but if you
- 28:33don't trust me and I I definitely would
- 28:35not encourage people to blindly trust me
- 28:38uh uh you know if you build up your own
- 28:41set of skills that you can obviously you
- 28:43know take a look at PAC and kind of fork
- 28:45it make it your own improve the skills
- 28:47definitely encourage that uh but for me
- 28:50it's really all about it just keeps
- 28:52coming back to trust you Every one of us
- 28:55here in this chat have a different
- 28:56standard for engineering. Uh and there
- 28:59are different things that are important
- 29:00for us in our codebase and uh when you
- 29:05are able to encode all of that into
- 29:06skills and you can verify that your
- 29:08agent is actually doing them that allows
- 29:10you to really kind of ascend this curve
- 29:12and um uh you know start automating
- 29:16things. Uh there's another piece I
- 29:18wanted to talk about. Um if there's more
- 29:22>> Yeah, go for it. I'll I'll pick up more
- 29:23questions as I go. But
- 29:24>> yeah, I think there's a third part to
- 29:25this which I haven't talked about yet,
- 29:27which is kind of an interesting one
- 29:29which is like refactoring and rewriting.
- 29:32Like one of the uh I guess most
- 29:34controversial one of the most
- 29:36controversial topics in the industry I
- 29:38think is like should you rewrite your
- 29:40app or not? Um because I think engineers
- 29:44are very prone to this where especially
- 29:46when you join a company you come in and
- 29:48you see like the codebase and you're
- 29:50like oh man this is like who wrote
- 29:52this code you know it's terrible I want
- 29:54to rewrite the whole thing there is a
- 29:56very common inclination and I think a
- 29:59lot of you know before agents um and I
- 30:02guess arguably even now people will
- 30:04definitely discourage you from re
- 30:06rewriting stuff but I'm actually here to
- 30:08make a case for why you might want to
- 30:10consider
- 30:12Um because
- 30:15I think it really depends. Uh you know
- 30:17uh brownfield applications I think are
- 30:19actually in a pretty good spot
- 30:21especially if they're set up well
- 30:23already. Uh and like recently I've been
- 30:26talking to some people but uh you know I
- 30:28I was just observing I I just noticed
- 30:31this parallel which is that a lot of big
- 30:35tech company problems are now
- 30:37everybody's problems. Um, and the big
- 30:39tech company problem, you know, like
- 30:41when I was working at Meta, like we had
- 30:42this giant monor repo, we had like, I
- 30:45don't know, tens of thousands of
- 30:46engineers just, you know, like banging
- 30:49on their keyboards and and shipping code
- 30:51and
- 30:53a lot of really great engineers at Meta.
- 30:55Uh, but, uh, I'll say like, you know,
- 30:58you'll be surprised that the code
- 30:59quality is actually not that good.
- 31:01[laughter]
- 31:02Um, and so I often joke that like, you
- 31:04know, before AI slop, we had human
- 31:06sloth. Um and so uh you know I think a
- 31:09lot of big tech infra like uh like what
- 31:12Meta has or Google you know you know
- 31:14really big tech companies are actually
- 31:17designed for that where you you're sort
- 31:18of like you're catering to the the you
- 31:21know like uh this sounds so bad to say
- 31:23but like the the least capable engineer
- 31:25on your team right you build you build
- 31:28frameworks you build conventions you
- 31:30build guard rails you know you restrict
- 31:32credentials so that you know your intern
- 31:34doesn't wipe your production database.
- 31:37Um
- 31:39there's uh you know if you have that
- 31:41level of infra already I think your
- 31:43agents can actually already do a very
- 31:45solid job right because they have the
- 31:48the guard rails are already in place for
- 31:51agents to not cause havoc or not cause
- 31:53too much havoc uh in your codebase um
- 31:56and you can always add more you know
- 31:58guard rails. Uh but I think like green
- 32:01field applications especially are you
- 32:02know like the brand new applications are
- 32:05like the biggest risk in my opinion. Uh
- 32:08and also the greatest opportunity
- 32:10because you know if you vibe code a
- 32:12project uh a prototype um like we did
- 32:15for Grockbot you know Grockbot was spun
- 32:17up very very very quickly. Um and if you
- 32:20if you haven't heard of of Grockbot it's
- 32:22like a a new application we just
- 32:24launched yesterday. Uh it's it's really
- 32:26cool. uh lets you orchestrate your
- 32:29create like individual agents that have
- 32:31their own identity and you can kind of
- 32:33orchestrate them. It's super cool.
- 32:34Definitely check it out. Um but yeah,
- 32:36that was it's like a very it was a very
- 32:38green field application like most
- 32:40prototypes are so like vibe coded very
- 32:43quickly. Humans were not reading the
- 32:45code at all. And uh I had this tweet
- 32:49recently uh where I said something about
- 32:52organic architecture. Um
- 32:56maybe I'll find it. Uh but the idea is
- 32:59that
- 33:01uh when you have a completely vibe coded
- 33:03application, you essentially have no
- 33:05guardrails whatsoever. So uh your agents
- 33:10when you give them a task, they will
- 33:12just solve it in whatever method is the
- 33:14most convenient. And over time you get
- 33:16into this uh situation where you have a
- 33:20codebase that is spiraling out of
- 33:22control because you don't understand it.
- 33:24Uh your agents understand it I guess in
- 33:27a way but like they've built something
- 33:29that is you know optimized for short for
- 33:31shortcuts. Uh and uh you know it will
- 33:35you will suffer you'll have a lot of of
- 33:37issues with that application.
- 33:40Uh so I think starting your codebase
- 33:43with uh like very strong constraints is
- 33:47very much needed uh because like when
- 33:50you have a codebase that you can trust,
- 33:52right? when you have guardrails that
- 33:54actually help you uh uh help your agents
- 33:58write good code, you can get into the,
- 34:00you know, like into this part of the
- 34:02curve where I I where like I I said, you
- 34:05know, I woke up today and I had like 20
- 34:06PRs merged u by my agents and that's
- 34:10because I invested a lot a lot of time
- 34:13uh over 600 PRs I I I calculated
- 34:16yesterday uh when I refactored all of
- 34:19Grockbot to this new architecture that
- 34:21I've been thing. Um,
- 34:24and yeah, I've gotten to a point where I
- 34:28I don't really look I really don't look
- 34:29at the code anymore. And um, I say that
- 34:32not just, you know, to sell you tokens,
- 34:34but because I, you know, it it it took a
- 34:38lot of work to get to that point. I
- 34:39spent a lot of tokens to get the
- 34:41codebase to this point where I no longer
- 34:43have to look at it. Uh, but I'm very
- 34:46excited because, you know, of the
- 34:48potential where, you know, it's not just
- 34:50this doesn't just benefit me. It
- 34:51benefits everyone contributing to
- 34:53Grockbot and it also empowers you know
- 34:56designers and product managers and you
- 34:59know pe uh even GTM people to add
- 35:02features to grabbot and I don't have to
- 35:04worry you know I don't have to to wake
- 35:06up at night in in the middle of the
- 35:07night and worry like oh someone's
- 35:09just merged a perf regression right I
- 35:11have a ton of constraints and CI it's
- 35:14like it's actually very annoying to
- 35:16write code in in graph web but like
- 35:18agents absorb all of that annoyance
- 35:21Um but yeah, I'm happy to talk about
- 35:23what exactly that is. Um
- 35:26>> yeah, I think one question um
- 35:29>> before we get into the this part here is
- 35:31just around that element of like what
- 35:33your your your CI looks like or maybe
- 35:35some of the constraints and then also
- 35:36like the average PR size. I saw a
- 35:38question about that earlier just to give
- 35:40people you know kind of a a glance. It
- 35:42doesn't have to be like mathematically
- 35:43average but just a you know like what
- 35:46generally the size of the a PR is. um if
- 35:49it's only a couple lines of code or you
- 35:50know um yeah
- 35:52>> um
- 35:54I think it depends uh let me
- 35:58>> I'm trying to do this in a way where I'm
- 36:00not going to like
- 36:01>> you yeah you don't have to share the
- 36:02actual number like an actual average
- 36:04>> I think this is fine
- 36:05>> benchmark
- 36:06>> but like we have so okay this is not
- 36:09that interesting but uh well fun fact is
- 36:12that virtualization in grogbot and in uh
- 36:16cursor is actually powered by uh pretext
- 36:20uh which is a sort of new library that
- 36:23someone's built. Um that's really
- 36:26interesting. You should you should check
- 36:27it out. But that's not really that
- 36:29important. Uh I think the average PR
- 36:31size I actually don't know. I I don't
- 36:34know if I want to click on these. Uh I
- 36:36probably can, but I would say
- 36:38[clears throat] like they can range
- 36:39anywhere from a few hundred lines or 50
- 36:41lines to like a thousand depending on
- 36:44what the thing is doing. Uh, so like
- 36:47here I'm actually like deleting a bunch
- 36:48of files. So I expect that it's just
- 36:50this like mostly deletion. Uh, but yeah,
- 36:53it kind of varies.
- 36:55There's no like Yeah,
- 36:57>> there's no like hard cap or hard limit.
- 36:58Are there all like 50 line PR?
- 36:59>> There's no hard cap. Yeah, there's
- 37:00definitely no hard cap, but I I do
- 37:01encourage my agents to split up their
- 37:03work into multiple PRs. Uh I do that
- 37:06mostly because uh I like I like the idea
- 37:11of the I guess maybe this is much harder
- 37:13to do now as in the world of agents and
- 37:15you have like so many commits but I like
- 37:18the idea that you know the git history
- 37:19is a very rich source of context. Uh,
- 37:22and I like the I like each PR to sort of
- 37:25atomically describe what that small
- 37:29piece of thing is doing, which also
- 37:31makes it easier for me to revert changes
- 37:32and like figure out, you know, oh, I
- 37:34shipped a bug and it's just it's here,
- 37:36right? It's not in this 40,000 line PR
- 37:39where you who knows what landed in
- 37:41there.
- 37:43Uh, but I don't have a hard cap on PR
- 37:46size.
- 37:47>> Cool. And then um yeah, also quick
- 37:50question on like CI. So again, you don't
- 37:51have to go into like uh the screen share
- 37:53of like your CI does, but just generally
- 37:56would you describe what the CI kind of
- 37:57looks like uh or how strict it is?
- 38:01>> Uh yeah. So uh well specifically for
- 38:04Grockbot. So Dune is the is the sort of
- 38:07cheeky code code name for the
- 38:09architecture that we've built for
- 38:12Grockbot. Um the CI looks pretty
- 38:16annoying because there's checks for
- 38:18everything. So like literally I have um
- 38:22uh well if you've written any react for
- 38:23example you know you know that one of
- 38:25the biggest foot guns in react is use
- 38:27effect. Uh so in
- 38:30uh Dune and in graphbot we've banned use
- 38:33effect. So Dune is just you can the the
- 38:36mental model of what Dune is uh you can
- 38:38kind of think of it as like Nex.js JS
- 38:40for uh electron apps and it's designed
- 38:43for agents to write uh and it's like
- 38:46custom for you know our agent powered
- 38:48applications. Um so the CI checks are
- 38:52very like specific to that like you know
- 38:53don't use use effect it's it's it's
- 38:56banned like CI will fail uh and yell at
- 38:58you. We have like some of the more
- 39:01interesting ones that people might raise
- 39:02eyebrows is like I actually ban code
- 39:05comments as well uh which is very
- 39:07interesting. Uh, but I've noticed that
- 39:1199% of the time agents just write code
- 39:14comments that kind of describe some
- 39:16historical thing that is actually
- 39:18totally irrelevant to the code. Um, like
- 39:21it will often say like, you know, oh,
- 39:22Lauren said you should never do this and
- 39:24it's now in in a code comment. Like
- 39:26what? Like why what that was? I didn't
- 39:28say that as like a durable, you know,
- 39:30global rule. I just meant like your this
- 39:33PR sucks and you should change that
- 39:35part.
- 39:36But agents don't really understand us
- 39:39that well surprisingly uh and or they
- 39:41kind of assume too much and they kind of
- 39:44do things in like very stupid ways. So
- 39:46like yeah we just ban everything.
- 39:48Everything you can imagine like the
- 39:50agents are bad at we ban. Uh so one
- 39:54example that we actually suffer a lot in
- 39:56the agents window is we have uh you know
- 40:00you if you've used the agents window
- 40:01you've definitely seen performance
- 40:02issues and you know we're constantly
- 40:04trying to fix them. Uh but it's like a
- 40:07it's a never- ending struggle because
- 40:09there's so many pull requests that get
- 40:11merged. Every any one of them could just
- 40:13regress performance or stability or
- 40:15reliability. Uh you know the agents
- 40:18window doesn't have this architecture
- 40:19yet. I plan to do bring this learning
- 40:21back there and kind of refactor
- 40:24everything there. uh but uh it just
- 40:27regresses super often uh because uh
- 40:30there's just one example is like we have
- 40:33very poor um isolation between
- 40:35processes. So like on you know on on
- 40:38Electron you have a renderer thread that
- 40:40renders your UI but you also have like a
- 40:42main thread that you can run other code
- 40:44that you know doesn't need to block the
- 40:46renderer.
- 40:48Um, but we do a poor job of separating
- 40:50those things. And so oftentimes you just
- 40:52accidentally have code that gets pulled
- 40:54into running on the renderer thread and
- 40:57then all of a sudden you're competing
- 40:58with the the renderer that you know that
- 41:01has a very if you want like 60 fps you
- 41:04have to every frame that gets drawn has
- 41:06to be done in 16 milliseconds. So very
- 41:09very small you know deadline per frame
- 41:12uh if you want you know a very smooth
- 41:14product. Uh and when you start building
- 41:16bringing in accidentally bringing in you
- 41:18know things that are like very
- 41:20computationally heavy or they have a lot
- 41:22of IO uh then you just get into like a
- 41:25lot of jank right your your FPS really
- 41:27drops you start uh you know losing
- 41:30frames you get long tasks that take more
- 41:33than 16 milliseconds and you just get
- 41:34this really choppy experience.
- 41:37So all of those patterns that we've
- 41:38learned basically building electron apps
- 41:40we've encoded into this framework and it
- 41:43becomes like a hard failure. So I
- 41:45literally in in grabbot we literally
- 41:47have a directory called electron main
- 41:49electron renderer and we have uh import
- 41:54uh CI I guess where we actually check
- 41:57the dependency graph to make sure you're
- 41:59not accidentally importing code from one
- 42:01directory to another. Uh so that's
- 42:04enforced by CI um as well as bug bots uh
- 42:09which is our which cursors um like code
- 42:12review tool that runs on CI uh you know
- 42:15in our agents MD it's everywhere like so
- 42:17I I I I um I have this thing here where
- 42:21I I talk about like um you know like
- 42:24there are multiple layers I think for
- 42:26building a good codebase. Uh obviously
- 42:29the codebase is one where uh if you have
- 42:31an architecture like this where it's
- 42:34extremely strict uh you know the the the
- 42:37way to build features is very
- 42:38conventional that's like the strongest
- 42:40strongest level of enforcement because
- 42:43agents just love to copy existing
- 42:45patterns. So, uh, one example of this in
- 42:48Rockbot is like we have this these
- 42:50concepts called like a feature and we
- 42:52have entry points and transcript cards
- 42:54like oh you know the cards that you see
- 42:56in the chat these are all like like
- 42:59nouns I guess in in the framework and so
- 43:02there's a very conventional way of
- 43:04creating them and so like a feature is
- 43:07all in in a single directory as an
- 43:09example and so all of the code that
- 43:11contributes to that feature lives in one
- 43:13directory so it's all coll-located in
- 43:15one case makes it super easy. You know,
- 43:17agents don't have to like uh grap around
- 43:19and try to figure out like where all the
- 43:21things are. It just looks at the feature
- 43:23and like, oh, okay, I'm working on the
- 43:25onboarding feature in Grockbot. Uh I'm
- 43:29just going to work in this directory.
- 43:30And for 80% of the work, it's mostly
- 43:32just very encapsulated there.
- 43:34[clears throat]
- 43:35But uh like it's like very it's like
- 43:38designed again for you know like the
- 43:40dumbest agent like you don't have to
- 43:42think, right? the the the one of the key
- 43:45principles I have for this framework is
- 43:47like the shortest the shortest path is
- 43:50the best path.
- 43:52So uh because that plays exactly to how
- 43:55agents love to write code is like they
- 43:56like to take shortcuts really you know
- 43:59they they'll find the quickest way to
- 44:01solve the problem. So why not make that
- 44:04the best way to solve the problem? Uh so
- 44:07I I I probably won't get into all the
- 44:09specific details. Um and uh the this
- 44:13framework is really more of a collection
- 44:14of ideas and principles rather than
- 44:16something that will open source. Uh you
- 44:19can you can you know screenshot this I
- 44:20guess if you want and uh tell your agent
- 44:23to uh do some build build something like
- 44:26this for you too.
- 44:28Um yeah, but it's really all about the
- 44:30layers uh you know like the the codebase
- 44:33is one part with features uh and
- 44:35directories and you know import or
- 44:38blocking import dependencies uh that
- 44:40shouldn't be imported uh but and and it
- 44:43all enforces that and static analysis.
- 44:45So like uh there's CI checks. We have a
- 44:48lot of lints for bad patterns that we
- 44:51observe. Uh compiler diagnostics. Uh
- 44:55there's also rules and bugbot which are
- 44:58um I think like three, four, five are
- 45:00more soft, right? These two actually
- 45:03make make CI red, right? So that you
- 45:07know there's a hard constraint where the
- 45:09agent can't just write crappy code
- 45:12for rules and skills and bogbot. Your
- 45:16agents can still forget, right? You can
- 45:18still or it may not always consistently
- 45:21apply them. So I like to layer them, but
- 45:25I don't I don't like to rely on them as
- 45:28the only source of enforcement because
- 45:31it's very very soft, right? And if you
- 45:32if you only have rules and bug bot and
- 45:34skills and a style guide for your code,
- 45:37you will it's only a matter of time
- 45:39before your codebase looks like complete
- 45:41trash. I'm sorry to say that but uh I
- 45:44definitely recommend yeah like you know
- 45:45investing in you know things that can be
- 45:48hard and forced right and this is why
- 45:51you know maybe the choice of tech stack
- 45:54that you use is also very important. Um
- 45:57like I think for example Rust is sort of
- 45:59making you know it's like getting super
- 46:01popular again uh because the compiler is
- 46:05so strict right the compiler enforces so
- 46:07many different things you know there's a
- 46:09borrow checker that you have to appease
- 46:11and if as long as you make sure your
- 46:12agents don't write unsafe code blocks uh
- 46:15you can more or less feel somewhat
- 46:17confident that if the code compiles it
- 46:20probably works and it's good. Uh but you
- 46:22see it gives you that level of trust and
- 46:25confidence that you as a human engineer
- 46:28no longer need to go and check it
- 46:31yourself. You know you you rely on code
- 46:34and static analysis to actually make
- 46:37that uh a lot smoother. Um, and I I
- 46:42guess the worst part, the worst place to
- 46:44be in is if you are stuck in code review
- 46:47land where you actually enforce all of
- 46:49the constraints, the invariance in your
- 46:51codebase by literally the human person
- 46:55saying, you know, reading the code and
- 46:56like, okay, you should not do this,
- 46:58right?
- 46:59Every time you have to do that, you
- 47:00should consider that as a code smell,
- 47:02like a anti- pattern. And you should
- 47:05say, okay, instead of me commenting on
- 47:06the PR, how do I turn this into a hard
- 47:10rule, right? How do I turn this into a
- 47:12lint rule, how do I turn this into a CI
- 47:14failure, or how do I even categorically
- 47:17eliminate this problem uh entirely? Uh I
- 47:22I can talk about another migration I've
- 47:23done, but I'll probably pause here.
- 47:25>> Sure.
- 47:26>> Yeah. I feel like that's that's where I
- 47:28am to be honest is is what you're
- 47:29describing right now which is that like
- 47:31I don't have all of these rules. So I
- 47:33have some things to go do after this
- 47:34session in terms of being able to scale
- 47:36my agents. I'm I'm definitely on like
- 47:38the uh you know maybe a couple of
- 47:40parallel ones locally stage. So like two
- 47:42to three locally and I'm sure most
- 47:44people here are on the same. So uh yeah
- 47:47I know we only have a couple minutes
- 47:48left. Lauren, was there anything else
- 47:49that you wanted to to highlight? I
- 47:51obviously there's lots of questions so I
- 47:52can grab more but I want to give you a
- 47:54few minutes if there's anything else you
- 47:55want to talk about.
- 47:56>> I think I've been yapping for quite a
- 47:57lot so I'm maybe let's just do
- 47:59questions.
- 48:00>> Okay cool. Uh one question that had uh a
- 48:03couple of uh came up a couple times was
- 48:04just around like token usage.
- 48:07>> So the the question is like is is what
- 48:09you're describing a realistic thing for
- 48:11people who are on you know uh a normal
- 48:15set of token usage. They don't have you
- 48:17know basically unlimited tokens uh to
- 48:19work with.
- 48:21I think that's a really good point. I
- 48:22mean, like obviously, you know, I work
- 48:24at a AI lab where we have unlimited
- 48:26tokens. So, uh I definitely cannot
- 48:31say that, you know, this is something
- 48:32everyone should do in the exact same way
- 48:35that I did it. I think it's possible to
- 48:37get to this point without, you know,
- 48:38breaking the bank.
- 48:41But, you know, if you're like an
- 48:42engineering leader or, you know, you're
- 48:44you you have a startup that you lead, um
- 48:47I think to me it's a question of ROI. Um
- 48:50and it's like uh yes you spend a lot of
- 48:54money on tokens in the upfront stage you
- 48:57know like refactoring a codebase is
- 48:59going to take a lot of tokens uh adding
- 49:01all these things uh is going to take a
- 49:02bunch of tokens but if we're heading to
- 49:05a world where agents are writing all the
- 49:07code and you know you want to be very
- 49:11lean right you don't want to have to
- 49:13hire you don't want to be you don't want
- 49:14to become like meta right like I mean
- 49:16like in terms of you don't want to
- 49:17become a 10,000 person engineering org
- 49:20because I mean that's a cool problem to
- 49:23have but also you you have so much
- 49:25overhead there's like planning you know
- 49:27like you it's it's a personally I I
- 49:30wouldn't uh it it's not super fun but um
- 49:34I think you want to stay very nimble
- 49:36right and you want to you want to be
- 49:38like agents are all about allowing you
- 49:40to do things that you couldn't do before
- 49:42that's really to me like the value of
- 49:44agents you know it's not just storing
- 49:47tokens on every single little thing but
- 49:49um to me like the thing I couldn't do
- 49:51before is like enforce this level of
- 49:54constraints in a codebase by myself
- 49:57right like I'm just a single person you
- 50:00know uh it would have taken me years to
- 50:02build this framework uh and do all the
- 50:05refactoring and test everything myself
- 50:09and verify you know like run imagine if
- 50:11there it was just me right now in in
- 50:13pre- agent era just like running you
- 50:16know by it would take me so long, right?
- 50:18And my salary is pretty high, right?
- 50:20Like, so, you know, the the question I
- 50:23think an engineering leader might have
- 50:25is just then, you know, like what is
- 50:27there's a trade-off of do you hire
- 50:30someone to do this or do you spend the
- 50:32tokens to set up a code base so that
- 50:34even the the most naive, right, the
- 50:38dumbest agents can do a good job. And
- 50:41when you actually get to this point like
- 50:42even agents that are not you know fable
- 50:45size do an excellent job of writing code
- 50:48and this pays a lot of dividends as well
- 50:50for me personally where I've empowered
- 50:53not just myself but again like PMs
- 50:57designers engineers who are not familiar
- 50:59with Grockbot to just contribute in a
- 51:02way that is sustainable.
- 51:05So, I think yeah, it's definitely like a
- 51:06trade-off for sure. You know, like
- 51:08nothing is like free for sure. Uh, and
- 51:11tokens are pretty expensive. Uh, but oh,
- 51:13actually, uh, I I I don't know how many
- 51:16of you have seen this, but we actually
- 51:18announced Grock 4.6 today. So, very
- 51:20exciting. Finally out. Um, so yeah,
- 51:23Groth 4.6 would be like a great it was
- 51:25very very smart. Uh, it's really good on
- 51:28the on the benchmarks. Uh, and it's the
- 51:30same the tokens. Uh well uh I hopefully
- 51:33I'm not saying this incorrectly, but uh
- 51:35I believe the cost per token is the same
- 51:38as 4.5. So you're actually getting more
- 51:41intelligence for the same cost. Uh I
- 51:45think this is an area that cursor tries
- 51:47to cursor and SpaceX AI try to really
- 51:50optimize for like that heredto frontier
- 51:53of you know cost versus intelligence. uh
- 51:56you know we don't necessarily want to
- 51:57build the biggest model ever because
- 51:59that is extremely expensive to run. It's
- 52:02really about like how do you find that
- 52:04sweet spot right? You don't you don't
- 52:05need a giant model but it's just super
- 52:07smart right and it's not very expensive
- 52:09for inference.
- 52:11Uh but um yeah, I think to kind of round
- 52:15it up, um I think it's like a it's it's
- 52:18there's a if you do your own analysis, I
- 52:20feel like it's pretty positive. It it'll
- 52:23be pretty positive that the ROI you get
- 52:25from investing in stuff like this, uh
- 52:28just empowers not just yourself, but
- 52:30your whole team to be so much more
- 52:32productive, right? Like imagine if you
- 52:34have an army of engineers like me who
- 52:36are shipping so much improvements and
- 52:39and bug fixes. uh you know every day,
- 52:42right? Like that is pretty exciting.
- 52:46>> Cool. Uh one last question before we
- 52:48wrap up. This one is for the people in
- 52:50product on the on the call.
- 52:51>> So let's say we do have an army of
- 52:53engineers who are shipping like Lauren.
- 52:55I'm just curious like how is the product
- 52:58team or other functions of your company
- 52:59keeping up given that like if you're
- 53:02shipping so quickly, have are they using
- 53:04AI more to do their jobs? like as much
- 53:06as you can speak to that obviously you
- 53:08don't have like you're not in that role
- 53:09but just curious about how that works.
- 53:12Um I think this is where Grothbot has
- 53:14been actually exceedingly powerful. Uh
- 53:17where so before grabbot like you know uh
- 53:20obviously cursor only had cursor like we
- 53:22only had agents window we had a CLI we
- 53:25had an IDE and these are really like
- 53:28power user tools right like devel
- 53:30they're designed for developers so it's
- 53:32very very developerentric you can do
- 53:34knowledge work in them but it like the
- 53:36UI is not really optimized for that. So
- 53:40we actually didn't really have uh well I
- 53:42think like a lot of people like you know
- 53:44GTM product like they might have used
- 53:46cursor uh to do their work but it
- 53:49definitely wasn't like a delightful
- 53:50experience for them. Um I think now with
- 53:54Grogbot
- 53:55uh it's become Grogbot is basically like
- 53:59their kusher moment for people who are
- 54:01not in tech in my opinion like it's like
- 54:03it's like a very very accessible way to
- 54:06use agents in a very comfortable very
- 54:09familiar interface. It looks like
- 54:10iMessage. Um and it's very fun to you
- 54:14know you can give your agent a fun name.
- 54:16uh you can have you can kind of do
- 54:18orchestration with in a very like
- 54:20natural way where you can sort of you
- 54:22know each agent is like a person right
- 54:23now you got a team of agents like
- 54:25working on you have one one agent per
- 54:27account that you manage as an example or
- 54:29if you're a PM you have you know you can
- 54:31have an agent that summarizes all the
- 54:32work that Lauren did last night and then
- 54:34now you know what I did right so I think
- 54:37our PMs are leveraging that a lot and
- 54:40they're shipping code too uh so you know
- 54:42like often times they will just say oh
- 54:44here's a bug I fixed it can you look at
- 54:45it and then I'll go review it and
- 54:47actually it's just perfect. I'm like
- 54:49okay stamp. Uh so uh that I think that
- 54:52shows that you know the the Dune
- 54:53architecture is holding up right the all
- 54:56the the really strict constraints allow
- 54:59people who are not experts in
- 55:01engineering to contribute at a high
- 55:03level. Uh so I'm I feel like I'm already
- 55:05seeing that pay off a lot where uh you
- 55:08know designers and PMs are just able to
- 55:10to to ship features directly. Um, and
- 55:14that just makes the Grockbot team super
- 55:18fast, right? Where we can ship so
- 55:20quickly. Um, and we have a lot planned.
- 55:24So, I'm very excited uh to, you know, uh
- 55:27to to ship more ship more
About this transcript
This page contains the full transcript of SpaceX AI Engineer Lauren Tan: Part 1 by Ai insider, generated from the public captions YouTube serves with the video. The transcript has 10,172 words across 1,389 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.