YouTube2Text

SpaceX AI Engineer Lauren Tan: Part 1 — Transcript

by Ai insider · 10,172 words · 1,389 segments · language en · Watch on YouTube

Full transcript

  1. 0:00So, the first thing I want to talk
  2. 0:02about, it's pretty pretty serious, okay?
  3. 0:06I need all of you to take this
  4. 0:07seriously. Put your phones down, put
  5. 0:09your laptop down, put your water bottles
  6. 0:12down. We're done hydrating. Are you all
  7. 0:15ready?
  8. 0:20Very excited to announce.
  9. 0:23[applause]
  10. 0:25Uh, you can't end PM insult yet. We'll
  11. 0:27we'll release that later today. Okay. So
  12. 0:29don't don't take your laptops out. Um
  13. 0:32but yes very exciting milestone and
  14. 0:35after I do the release you can get
  15. 0:37started your applications by installing
  16. 0:39babel plugin react as well as the new
  17. 0:40eslint plugin. Uh okay so I assume you
  18. 0:43can see my screen.
  19. 0:44>> Yes. Yeah we're good.
  20. 0:46>> So yeah today yeah I think I think the
  21. 0:48big theme for me as I've been using
  22. 0:51agents to write code and I'm sure a lot
  23. 0:53of you have had the same experience as
  24. 0:55well is how do you trust it? You know,
  25. 0:57especially if you are an engineer that's
  26. 1:00been writing code for a very long time,
  27. 1:02you have a lot of opinions and lessons
  28. 1:05that you've learned about doing good
  29. 1:07engineering. And when you see agents
  30. 1:10just, you know, winging it and, you
  31. 1:12know, guessing, hallucinating,
  32. 1:14uh, you know, confidently stating that
  33. 1:16they found the smoking gun, uh, for the
  34. 1:19hundth time, uh, but it's actually not
  35. 1:21the real problem. You lose a lot of
  36. 1:23trust. And when you lose when you don't
  37. 1:26have much trust in your agents, I feel
  38. 1:29like you you really can't get the most
  39. 1:30out of them. And for me, the parallel is
  40. 1:33like with management. Uh so if I'm an a
  41. 1:36manager, an engineering manager of a
  42. 1:38team and I have a bunch of, you know, I
  43. 1:41have a team of engineers uh on my team
  44. 1:44and I don't trust them, then the mode of
  45. 1:46operation I'm going to be in is going to
  46. 1:48be like micromanagement, right? I'll
  47. 1:50have to spend a lot of time looking over
  48. 1:52my reports shoulders and checking that
  49. 1:55they're doing their work well, you know,
  50. 1:57like they're not shipping bugs to
  51. 1:59production.
  52. 2:01And so I drew this chart because uh it's
  53. 2:05not it's not a very scientific chart,
  54. 2:06but like this is how I imagine myself
  55. 2:10and my journey through using agents. So
  56. 2:13you know like fast forward or back
  57. 2:16forward uh or fast back uh fast
  58. 2:20backwards like a year or so when you
  59. 2:23know nobody was or not many people were
  60. 2:24using agents to code. Uh I think you
  61. 2:29uh you know get into this mode where you
  62. 2:31are
  63. 2:33in very heavily in the loop with one or
  64. 2:36several like a handful of agents and you
  65. 2:39find yourself just constantly figure uh
  66. 2:41you know trying to understand what your
  67. 2:42agents are doing uh and you're very very
  68. 2:45in loop. You're watching every single
  69. 2:47output. you are sitting there prompting
  70. 2:50um and you really can't parallelize
  71. 2:53beyond that because you don't again you
  72. 2:55don't have that trust right you can't go
  73. 2:57to a 100 agents uh like spawn 100 agents
  74. 3:00when you don't even trust the output of
  75. 3:02one agent
  76. 3:04so over the past 5 months I feel like
  77. 3:07I've really been able to uh like ascend
  78. 3:10this trust curve and now I'm at the
  79. 3:14point where uh I actually have this
  80. 3:17sounds kind kind of scary to say this
  81. 3:18and it makes me sound like a slop
  82. 3:20artist, but I I promise I'm not, but I
  83. 3:22actually have my agents now um
  84. 3:24automerging PRs for me. Uh which is like
  85. 3:27a wild thing to say, but um like I woke
  86. 3:30up today and there were like 20 PRs
  87. 3:33landed and I just reviewed them on main
  88. 3:35like they were already landed and they
  89. 3:37were good. Uh so how did I get to that
  90. 3:40point is basically what I wanted to talk
  91. 3:43about today.
  92. 3:45Uh and again like yeah feel free to jump
  93. 3:47in if you have questions Colin. Um but
  94. 3:51uh oh yeah of course I got to show this
  95. 3:53this chart. Uh where uh
  96. 3:59oh do do not trust to someone requested
  97. 4:02to control my computer. Uh probably
  98. 4:04won't do that. Uh but yeah, so this
  99. 4:07chart I think I I'm I'm sharing this
  100. 4:10chart not to kind of like flex but to
  101. 4:12kind of show like the journey like so
  102. 4:15you can see like the curve like it sort
  103. 4:16of like inversely matches the
  104. 4:19contributions I've been able to land at
  105. 4:20cursor. So I joined five months ago and
  106. 4:24five months ago like I you know my first
  107. 4:26month I was like not very productive
  108. 4:28because I was ob you know I was learning
  109. 4:29the codebase didn't know what the heck
  110. 4:31was going on and as I got more confident
  111. 4:35in in my agents uh I've really been able
  112. 4:37to kind of ramp up my productivity. Uh
  113. 4:41and again like yeah like last month I
  114. 4:43shipped a thousand PRs which is
  115. 4:46ridiculous. Uh, and then this month
  116. 4:48we're only on the 12th. I'm already at
  117. 4:51like almost 800 PRs landed. Uh, so the
  118. 4:56velocity is definitely high and you you
  119. 4:58I'm sure a lot of you will definitely be
  120. 5:00questioning like how how much of this
  121. 5:02code is actually good. Um, and I think
  122. 5:04yeah, like that's definitely fair to
  123. 5:06question.
  124. 5:08Um, but uh, yeah, I think I think if you
  125. 5:12set up your agents well, you can
  126. 5:15definitely get to a very similar level.
  127. 5:19Um, and so I'm going to talk about how
  128. 5:20we do that.
  129. 5:23Uh so for me I think I'm curious like I
  130. 5:27guess call in your experience as well
  131. 5:28but uh for me I think the most important
  132. 5:33skill that you should have in your
  133. 5:35toolbox when you work with agents is
  134. 5:38verification. Uh and by verification I
  135. 5:41mean the ability for an agent to
  136. 5:43actually run the code uh or take CPU
  137. 5:48traces or heap snapshots or uh you know
  138. 5:52open an iOS simulator whatever you know
  139. 5:55however your application is exposed to
  140. 5:57your users it can do the same thing and
  141. 6:02uh run it for real and actually test and
  142. 6:04verify it don't work because that's the
  143. 6:07thing that really closes the loop.
  144. 6:09uh it doesn't guarantee your agent
  145. 6:11writes good code uh but it allows them
  146. 6:14to at least write correct code uh which
  147. 6:17is a big a really big step forward for
  148. 6:19being able to trust your agents. Um
  149. 6:24I will I can share one example that we
  150. 6:26have uh within cursor
  151. 6:30uh oops
  152. 6:33where let me open this
  153. 6:43make this uh make me full screen. There
  154. 6:46you go. Uh so for the for cursor's agent
  155. 6:51window uh so this is actually an
  156. 6:54interesting story but uh when I joined
  157. 6:56cursor 5 months ago uh they're actually
  158. 7:00uh well I was supposed to join a
  159. 7:02different team I was supposed to join
  160. 7:03like the cloud agents team uh but then
  161. 7:05since I have a lot of experience working
  162. 7:07on react and agents window is a react
  163. 7:10application uh I was
  164. 7:14uh I was asked to basically help out
  165. 7:16with the agent window work. Uh but um
  166. 7:23there wasn't really a lot of like skills
  167. 7:25to help me. So I just found myself like
  168. 7:27okay uh agents window is going to launch
  169. 7:29in like a week, right? We have a really
  170. 7:31tight deadline and um there was uh you
  171. 7:35know I was just sitting there like okay
  172. 7:37I'm going to open up the performant the
  173. 7:38the Chrome dev tools and just like take
  174. 7:40a trace look at it myself and try to
  175. 7:43make sense of this flame graph. And keep
  176. 7:45in mind, I was just like in my first
  177. 7:47week, so I had no idea what I was
  178. 7:48looking at, no idea where, you know, I
  179. 7:50mean, I had some idea, but you know, the
  180. 7:52codebase was completely fresh to me. Uh,
  181. 7:55and I realized like my agent had no idea
  182. 7:58either, you know, like I would take a
  183. 8:00screenshot of the tree, I would download
  184. 8:01trace, I would send it to it, and it be
  185. 8:02like, yeah, it kind of looks like this,
  186. 8:04you know, uh, and it would like
  187. 8:06confidently state like it's this thing.
  188. 8:08And then I try to fix that and turns out
  189. 8:10that's not the actual thing.
  190. 8:13So this was a very very slow process and
  191. 8:15if you've ever done any like performance
  192. 8:17work yourself or you know just even
  193. 8:19development with an agent where you
  194. 8:21don't have a verification skill you are
  195. 8:24the verifier right you you're the
  196. 8:26bottleneck you you you tell your agent
  197. 8:28to do something and then it goes off and
  198. 8:30write some code then you open up your
  199. 8:32you know local dev build and then you
  200. 8:34start to say oh you know doesn't work
  201. 8:35then you got to copy paste screen uh you
  202. 8:37know screenshots or console errors or
  203. 8:40whatever uh and then your agent like
  204. 8:42slowly kind of like uh you know works
  205. 8:44with that and then tries to understand
  206. 8:46it and um fix the thing but then you
  207. 8:50were constantly just in the loop and and
  208. 8:52being a bottleneck. So there's really no
  209. 8:54way to parallelize. So the control glass
  210. 8:56skills are like one of the first skills
  211. 8:58I built uh for cursor uh and glass by
  212. 9:02the way is the code name for agents
  213. 9:03window that we use internally but it's
  214. 9:06just cursor I guess. Um, and so this
  215. 9:10skill uh is I guess the the the code
  216. 9:13itself is not super interesting. Your
  217. 9:15agent can very easily make one for you.
  218. 9:17uh where if if you're building an
  219. 9:19Electron app or a web app or even iOS uh
  220. 9:23applications uh you can teach your agent
  221. 9:26how to use like the ChromeDev tools
  222. 9:28protocol or through uh Apple has some
  223. 9:32utilities as well for running the
  224. 9:34simulator and taking traces and
  225. 9:36controlling programmatic control as
  226. 9:38well. Uh so that's really useful. Uh but
  227. 9:43one thing I want to talk about is uh the
  228. 9:48this thing
  229. 9:50uh where is the read me? Uh so this
  230. 9:53skill comes with this very unique
  231. 9:55feature called or not feature uh unique
  232. 9:58file called a feature map. And so the
  233. 10:02story then is like I built this skill
  234. 10:04and so now the agent was able to uh
  235. 10:06actually run the agent window and take
  236. 10:09traces and whatnot. Uh but it had no
  237. 10:12idea what what the agents window was. So
  238. 10:17um you know like someone would say like
  239. 10:18oh the the left sidebar is like laggy or
  240. 10:22something like that or you know the
  241. 10:24right side the the PR tab is not working
  242. 10:27and the agent would just be like kind of
  243. 10:28flailing around. It would spend a lot of
  244. 10:30time trying to like look up the code and
  245. 10:32you know where is this feature? How do I
  246. 10:33actually get to it on the UI which made
  247. 10:36it basically completely useless. Uh you
  248. 10:38know like we would I would run
  249. 10:41the skill locally and you know it would
  250. 10:43spawn a dev build uh but then it just be
  251. 10:47turnurning like I just try to click here
  252. 10:49it wouldn't know how to get to things um
  253. 10:52and it was just an awful experience uh
  254. 10:55so was putting arrows on my screen um so
  255. 11:00uh yeah this this feature map has been
  256. 11:03really useful uh because it teaches the
  257. 11:05agent how to get to all of the features
  258. 11:07that you have um and in PAC the plugin
  259. 11:11that I I've made uh if you search for
  260. 11:15PAC cursor on Google you you'll find it
  261. 11:18uh but there is a create verification
  262. 11:20skill in that plugin where it actually
  263. 11:22helps you set up something like this for
  264. 11:24yourself um including the feature map so
  265. 11:27it will actually explore the code and
  266. 11:29build up this initial feature map that
  267. 11:31tells your agent how to get to all of
  268. 11:33the different features that you have uh
  269. 11:36and this is extremely powerful because
  270. 11:38now like you have these user reports
  271. 11:40that come in uh you you can actually map
  272. 11:43even like a vague report or even a
  273. 11:45screenshot. So we have this uh
  274. 11:48internally at cursor where uh we have a
  275. 11:51slack channel with you know lots of
  276. 11:53people giving us feedback on the agents
  277. 11:56window and rockbot and whatnot. Uh, and
  278. 11:59often times the report is very bad like
  279. 12:02very low quality like someone will just
  280. 12:03put very often we get like a screenshot
  281. 12:06like and then someone just says question
  282. 12:07mark question mark question mark like
  283. 12:09what is this and you know like the
  284. 12:11without this your agent like I have no
  285. 12:13clue right but with a feature map like
  286. 12:16this it has a lot more context and
  287. 12:19understanding of how to actually
  288. 12:21navigate how to get to all the different
  289. 12:23features. Uh so like you know example
  290. 12:26like I guess like the sidebar like what
  291. 12:28is the sidebar uh you know like all the
  292. 12:30different sub features that are present
  293. 12:32in it um like from the user point of
  294. 12:35view here's where how do I get to it all
  295. 12:38the different keyboard shortcuts
  296. 12:40uh even like the the what do you call it
  297. 12:43the DOM elements or yeah like the
  298. 12:46attributes that you use for selecting
  299. 12:48things through the CDP uh are all there.
  300. 12:52So uh again yeah this is like really
  301. 12:54really powerful uh for for agents
  302. 12:58>> uh and pack ships uh that create
  303. 13:01verification skill but also a maintain
  304. 13:03verification skill uh so you can keep
  305. 13:06this up to date.
  306. 13:08>> Cool. Yeah, I was just going to ask how
  307. 13:09you created that. So do you mind sharing
  308. 13:10a little bit more about um that that
  309. 13:12process in the context of Pstack and
  310. 13:14maybe just what Pstack is for the folks
  311. 13:15who aren't familiar?
  312. 13:17Yeah. So, P stack is pretty interesting
  313. 13:19because uh well, first of all, the name
  314. 13:22is kind of goofy. like the P the P in P
  315. 13:25stack is like potato potato stack
  316. 13:28because I um so uh uh there is a pretty
  317. 13:34uh famous person Gary Tan who is the CEO
  318. 13:37of Y Combinator and he's come up with
  319. 13:40this plugin called Gstack uh Gary Stack
  320. 13:43and uh funnily enough we share the last
  321. 13:46name we have no relations uh but I
  322. 13:49thought it would be funny to kind of you
  323. 13:50know poke fun at Gary and make Pstack
  324. 13:53my version of of of of his plugin. Uh
  325. 13:57but kind of tailor it to my own set of p
  326. 14:00engineering practices.
  327. 14:03Uh but I honestly actually never set out
  328. 14:05to build PAC. Uh it just started with a
  329. 14:07bunch of skills, right? Like I started
  330. 14:09with that control glass skill and then I
  331. 14:11started with another skill like called
  332. 14:13how which I also noticed through like
  333. 14:16observing agents. Um, so like you know
  334. 14:19in the early days of me, you know,
  335. 14:21trying to climb this ladder, I was like
  336. 14:22super in the loop and I was basically
  337. 14:25nitpicking my agents to an extreme
  338. 14:27degree, I was like I would tell it um,
  339. 14:31you know, this feature has stopped
  340. 14:32working. Here's a bug report. Like why
  341. 14:35isn't it working?
  342. 14:37And very often the agent would just like
  343. 14:40confidently state like, "Oh, it has to
  344. 14:42be this, right? It has to be this
  345. 14:43thing." And I noticed like when I looked
  346. 14:46at the actual tool calls, I noticed it
  347. 14:48wasn't actually reading the code that I
  348. 14:51thought should be affected. And that
  349. 14:53made me just extremely suspicious. And
  350. 14:55at that point, I was like, I'm not going
  351. 14:56to I can't trust any this agent anymore
  352. 14:59cuz it's just it's just completely
  353. 15:00hallucinating. And I think
  354. 15:04I think it's very easy to just you know
  355. 15:06like build up that distrust and not and
  356. 15:08kind of feel helpless like you know you
  357. 15:11don't know how to help your agents
  358. 15:13succeed. But like again I think the the
  359. 15:16the management analogy is super helpful
  360. 15:18because like imagine if you were a
  361. 15:19manager of an engineering team and you
  362. 15:22had an engineer on your team who was a
  363. 15:24really good coder. No business context
  364. 15:26whatsoever. So you know they they just
  365. 15:28you just hired them and they they
  366. 15:29onboarded you know like 5 seconds ago.
  367. 15:33Uh and so how do you actually teach that
  368. 15:35person to be effective? So how you do
  369. 15:38that is through a skill. uh skill being
  370. 15:40just you know it's just markdown right
  371. 15:42but you know it encodes a lot of
  372. 15:44information instructions a lot of uh you
  373. 15:48can really draw out a lot of
  374. 15:51intelligence from an agent by well some
  375. 15:54people on Twitter call it like you know
  376. 15:55pull the agent to a different latent
  377. 15:57space which is kind of like a fancy way
  378. 15:59of just saying like since uh you know
  379. 16:01LLMs are sort of like they predict the
  380. 16:04next token uh when you give it some high
  381. 16:07quality tokens uh to begin with then you
  382. 16:09know it it can kind of pattern match on
  383. 16:12like a higher space that's you know
  384. 16:14smarter.
  385. 16:16Um so that's like a very interesting
  386. 16:19model there but yeah I built Pstack very
  387. 16:21very incrementally. Uh so uh started
  388. 16:25with just really observing how agents
  389. 16:27you know all the fail different failure
  390. 16:29modes of of that agents were having and
  391. 16:32every time I saw that I just okay I'm
  392. 16:33just going to make that a skill right
  393. 16:35like stop hallucinating actually go and
  394. 16:38search up look up the code use a lot of
  395. 16:40sub agents uh and yeah stop guessing
  396. 16:47>> yeah that makes sense one one kind of
  397. 16:48followup question here uh both from
  398. 16:50myself and from a bunch of people in the
  399. 16:51chat so
  400. 16:52>> I guess It's two two parts. So one is
  401. 16:54like how do you maintain these skills?
  402. 16:55So like the product changes over time.
  403. 16:57Obviously there's a lot of people who
  404. 16:58are shipping against the codebase. So
  405. 17:00how do these skills get maintained? Uh
  406. 17:02and then second to that is like how do
  407. 17:04you know when your verification is is
  408. 17:06good enough? Uh like and you know you
  409. 17:09can trust that the ver verification
  410. 17:10loops that you've built are going to I
  411. 17:12guess you trust that the outputs uh when
  412. 17:14they're done.
  413. 17:17>> Uh yeah maybe I'll talk about um I think
  414. 17:20somehat related. Maybe I'll start with
  415. 17:22this one first. So like how do I
  416. 17:25maintain these skills?
  417. 17:28So um if you're not familiar with this
  418. 17:31concept, [clears throat] an eval is
  419. 17:32essentially like a way to uh well I the
  420. 17:36mental model I have is like it's like a
  421. 17:37unit test for an agent. Um and uh you
  422. 17:41can actually make your own eval. You
  423. 17:44don't need like a special framework for
  424. 17:45them. You can build you can you can
  425. 17:47build one depending on like you know how
  426. 17:50scientific and how rigorous you want to
  427. 17:52be. Uh my screen is red.
  428. 17:55>> Yeah, there's a little button. Um sorry.
  429. 17:58Do
  430. 17:58>> you mind like disabling the drawing or
  431. 18:00something? I I can't see my screen.
  432. 18:02>> Yeah, sorry if you guys could not draw
  433. 18:04on the screen that'd be great. But um
  434. 18:05there's a little button in the
  435. 18:07>> I got a troll.
  436. 18:08>> Yeah, the little drop down.
  437. 18:11>> How do I clear?
  438. 18:13>> Yeah, you got it. Perfect.
  439. 18:16>> Um,
  440. 18:16>> good to know.
  441. 18:18>> Yeah. So, eval
  442. 18:21unit test your skills basically. And
  443. 18:24actually in Pstack, we ship uh under
  444. 18:26potato mode, there's a playbook if you
  445. 18:28search for it called Eval playbook. Um,
  446. 18:31and it's uh
  447. 18:33uh it's like not it's actually pretty
  448. 18:36pretty rigorous the way it's done. Uh,
  449. 18:38but um, essentially what I do is I spawn
  450. 18:41a lot of different sub agents. I have
  451. 18:43like my main coordinator agent uh come
  452. 18:46up with a rubric for uh what I want the
  453. 18:50skill to do. Um and then it spawns all
  454. 18:54these sub aents and it's it creates
  455. 18:56individual directories for them uh which
  456. 18:59are cleverly named to not let the sub
  457. 19:02aent know that it's being evaluated
  458. 19:05because uh agents can actually tell and
  459. 19:08when they do they change their behavior.
  460. 19:10Uh, but it does a bunch of stuff like
  461. 19:12that to um essentially yeah like test
  462. 19:16whether or not the skill I'm making or
  463. 19:19changing is actually doing what I think
  464. 19:21it does. Um, and one of the really nice
  465. 19:23things about cursor is that we are we we
  466. 19:26support so many different models. So you
  467. 19:28can actually eval your skill across all
  468. 19:30sorts of different models um and you
  469. 19:33know get a sense of how well it performs
  470. 19:36across that different matrix. um
  471. 19:39especially for the models that you use.
  472. 19:42Uh so I do this a lot. Every time I I
  473. 19:44modify a skill, I will run one of these
  474. 19:47uh like the EBAL playbook uh and make
  475. 19:49sure that you know it's actually leading
  476. 19:52to a result I want. Uh but I will say
  477. 19:55like
  478. 19:56maintaining skills is actually pretty
  479. 19:58hard. Uh it requires I think a lot of
  480. 20:01taste and observation. So you kind of
  481. 20:04need to be very good at being a backseat
  482. 20:07driver. You know what I mean? Like if
  483. 20:09you do pair if you ever done pair
  484. 20:10programming for example, uh and you
  485. 20:12watch a coworker code and you just like
  486. 20:15you you could probably do this better.
  487. 20:16You know, you could do you know like why
  488. 20:18did you not do this, right? You you ask
  489. 20:20a lot of questions to your coworker. And
  490. 20:22it's kind of a similar thing here. You
  491. 20:23like you don't want to just be a passive
  492. 20:25observer of your agent. You want to be
  493. 20:27very in the driver seat in the initial
  494. 20:29stages when you're building up your own
  495. 20:31set of skills. uh you know obviously you
  496. 20:33can use something like PAC but if you're
  497. 20:35building your own set of skills it's
  498. 20:37very I think you know opening up the all
  499. 20:40the tool calls and like reading the code
  500. 20:42and reading all the uh the agent
  501. 20:45behavior and their thinking blocks is a
  502. 20:48really great way to see where they they
  503. 20:50fail right like what what you know where
  504. 20:53are they being done and then you can go
  505. 20:55and build a skill for that and then with
  506. 20:58verification how you trust it is it's I
  507. 21:00think it's also very similar iteration
  508. 21:02loop uh where you know like I actually
  509. 21:05did the same process for verifying the
  510. 21:07verification skill where I actually get
  511. 21:11um so one thing that's interesting about
  512. 21:13eval is that you can sort of hill climb
  513. 21:16them meaning that uh your eval can
  514. 21:19produce a score right uh a score that
  515. 21:21you can get your coordinator to produce
  516. 21:24uh but also comp uh you can have a judge
  517. 21:26agent of a different model to uh kind of
  518. 21:30cross reference and make sure that the
  519. 21:33first model is not being biased, right?
  520. 21:35The model that's judging all of the sub
  521. 21:37aents that are running the thing. Uh but
  522. 21:39you can also like hill climb. So meaning
  523. 21:41that you can you can use like /loop in
  524. 21:44cursor and you can say okay keep looping
  525. 21:46on this eval right until everything is
  526. 21:5010 out of 10 as an example. Uh and I did
  527. 21:52the same the basically the same approach
  528. 21:54with the control skill. And so I kind of
  529. 21:56it was very it was very hands-off
  530. 21:58actually. Uh so you know I uh I kind of
  531. 22:01built I built that skill that way like
  532. 22:03the CLI and that skill. Um and over time
  533. 22:08it's gotten really good. Uh but yeah it
  534. 22:10was definitely not super smooth at the
  535. 22:12beginning. It required a lot of
  536. 22:14iteration and I think there's an analogy
  537. 22:17here for me which is um well I make this
  538. 22:20analogy later in a different slide on my
  539. 22:23drawing here. Uh but I think of it like
  540. 22:27uh you know as a as a engineer now
  541. 22:30you're sort of more like you uh like
  542. 22:33maybe a manager or the analogy I like is
  543. 22:35like you're like a a chef in a
  544. 22:38restaurant. Uh you you're the head chef.
  545. 22:40Uh you're not cooking all the food
  546. 22:42yourself anymore. You have a team of
  547. 22:44cooks, right? You have line cooks, you
  548. 22:46have a sue chef, you have, you know, all
  549. 22:47these different stations.
  550. 22:49Um and it's your job to really design
  551. 22:52the environment. you know, you you're in
  552. 22:54charge of setting up the kitchen. You're
  553. 22:55in charge of, you know, like giving
  554. 22:58tasks to different people. So,
  555. 23:02um yeah, it's a very interesting way of
  556. 23:05working. Uh but yeah, that's that's how
  557. 23:08I've basically built uh these
  558. 23:10verification skills.
  559. 23:11>> Yeah, just just one thought there on
  560. 23:13like to go try to go one layer deeper.
  561. 23:15So are you let's say we wanted to build
  562. 23:18um an eval or a skill for for something
  563. 23:22and we wanted to kind of get better on
  564. 23:24its own which is is what I think you're
  565. 23:25suggesting. Uh are you doing that in
  566. 23:28like a work tree kind of isolated with
  567. 23:30like the sub agents and and then the
  568. 23:32reviewer agent and and all that? Is it
  569. 23:34happening like in some type of cloud
  570. 23:35hosted environment? Like what's the the
  571. 23:37more the practical steps if I wanted to
  572. 23:39go do this uh and like set up a
  573. 23:41verification system for something? What
  574. 23:42would I what would I do or where would I
  575. 23:44start?
  576. 23:46Um I think that uh the best place to
  577. 23:49start is local because you can observe
  578. 23:53you can definitely observe what your
  579. 23:54agents are doing. So uh if you're
  580. 23:56building a verification skill for
  581. 23:58yourself uh I would definitely start
  582. 23:59local and just have your agent bring up
  583. 24:02the application whether it's like a CLI
  584. 24:04or a desktop app or whatever. And so you
  585. 24:08can actually observe right you can see
  586. 24:09how the agent is interacting with the
  587. 24:12the application. you can see it, you
  588. 24:15know, how it calls like the different
  589. 24:17APIs that that allow it to interact with
  590. 24:20the uh the application.
  591. 24:23Um, but uh for me personally uh I have
  592. 24:27basically been kind of all in mostly all
  593. 24:30in on cloud agents because they're
  594. 24:31extremely powerful. Uh and the really
  595. 24:35powerful thing about cursor is the the
  596. 24:37cloud agents actually where if you spend
  597. 24:39a little bit of time setting up your
  598. 24:40environment
  599. 24:42these control skills, these verification
  600. 24:44skills pay a huge amount of dividends
  601. 24:46because it's not just something that
  602. 24:49makes you as a single engineer better.
  603. 24:52It actually levels up your whole team uh
  604. 24:54and even your whole company because uh
  605. 24:57you can actually start thinking about
  606. 24:58cloud agents. start thinking about
  607. 25:00automations that automatically do things
  608. 25:03like uh I I get I I kind of talk about
  609. 25:06this a bit later, but I'll just kind of
  610. 25:09get into it. Uh where where you know,
  611. 25:11for example, like I talk a lot about
  612. 25:13this agent we have called Benny, right?
  613. 25:16who who uh you know takes all of the bug
  614. 25:19reports that we get and it automatically
  615. 25:22goes off in the cloud, opens up a cloud
  616. 25:24uh it's you know its desktop. It runs
  617. 25:27cursor in its own computer and it uses
  618. 25:30the same control skills to interact with
  619. 25:33the application and try to reproduce the
  620. 25:34bug uh or the user report, right? And
  621. 25:37this is so so powerful because at once I
  622. 25:40can immediately I I get so much
  623. 25:42information from this automatically like
  624. 25:44here in this example you can see that uh
  625. 25:46the Benny actually reproduced the bug uh
  626. 25:49but it's already fixed on main. So it
  627. 25:53actually confirms that we fixed this
  628. 25:54problem already and all I need to do is
  629. 25:56just release another build of of cursor.
  630. 26:00Uh so that's like huge information there
  631. 26:02that I didn't have to go off and sit
  632. 26:04with an agent, you know, and spend an
  633. 26:05hour trying to figure like is this
  634. 26:06fixed? this is not fixed. So you you you
  635. 26:09gain back so much time. Uh but you know
  636. 26:12everybody on my team benefits from this.
  637. 26:14Everybody at the company benefits from
  638. 26:15this. Uh so definitely think that uh you
  639. 26:20know keeping these uh using cloud agents
  640. 26:22is super powerful. Uh but yeah it's like
  641. 26:25a journey. You have to trust it first
  642. 26:28right before you you get to this point.
  643. 26:30And that's it goes back to what I was
  644. 26:32saying here where you know it's very
  645. 26:34hard. It's almost impossible. And I
  646. 26:36would definitely encourage you not to
  647. 26:37try to jump from, you know, like if
  648. 26:40you're still in this zone, you don't
  649. 26:42want to jump to like I'm going to spawn
  650. 26:44hundred of thousand or thousands of
  651. 26:46cloud agents right now because you're
  652. 26:48just going to waste a lot of tokens. Um,
  653. 26:51and it's going to be extremely
  654. 26:52expensive.
  655. 26:53>> Yeah. So just to kind of recap so far,
  656. 26:55basically the if we wanted to go on the
  657. 26:57journey that you've kind of gone on, it
  658. 26:59would be to start with verification,
  659. 27:02building some some skills and some some
  660. 27:04ways of determining that the agents are
  661. 27:05producing at least like correct code.
  662. 27:08Whether like you said whether it's good
  663. 27:09code or not is maybe a separate
  664. 27:10question, but like it's it's technically
  665. 27:11solving the problem by looking at you
  666. 27:13know stack traces, looking at you know
  667. 27:15the the actual behavior in the app and
  668. 27:17so on. Um, and then once we trust it
  669. 27:19locally, then we can start to think
  670. 27:21about scaling into the cloud and running
  671. 27:22more agents that are picking up signals,
  672. 27:25I guess, on their own, right? So whether
  673. 27:26that's like a bug report that comes in
  674. 27:28or something, they can go and pick it up
  675. 27:29and solve the problem and and give us
  676. 27:31back a PR. And then maybe the last step
  677. 27:34is like automerging the PRs, which uh is
  678. 27:36where you're at, maybe not where we want
  679. 27:38to
  680. 27:39>> um and then and reviewing the one main,
  681. 27:40but um is that is that about right?
  682. 27:43>> Yeah, exactly. I think yeah, that's why
  683. 27:44I drew this this uh this this curve,
  684. 27:46right? because that this this basically
  685. 27:49describes my journey of you know when I
  686. 27:51started barely could use a couple agents
  687. 27:53and I was just observing every single
  688. 27:55thing. I think there's really no
  689. 27:57shortcut for going from here to there
  690. 27:59because this is really about your
  691. 28:01personal level of trust in agents,
  692. 28:03right? Um obviously, you know, as a as
  693. 28:06engineer, you don't want to just slop
  694. 28:07code into production. So, how do you
  695. 28:09actually build up that trust? Takes um a
  696. 28:12lot of uh I guess taste and judgment. Um
  697. 28:16but uh you know, like I think plugins
  698. 28:19like Pstack definitely kind of help you
  699. 28:22uh get up to speed much quicker. Uh and
  700. 28:25so I guess it's like if you trust me and
  701. 28:27you trust PAC then in by extension you
  702. 28:30can maybe trust your agents but if you
  703. 28:33don't trust me and I I definitely would
  704. 28:35not encourage people to blindly trust me
  705. 28:38uh uh you know if you build up your own
  706. 28:41set of skills that you can obviously you
  707. 28:43know take a look at PAC and kind of fork
  708. 28:45it make it your own improve the skills
  709. 28:47definitely encourage that uh but for me
  710. 28:50it's really all about it just keeps
  711. 28:52coming back to trust you Every one of us
  712. 28:55here in this chat have a different
  713. 28:56standard for engineering. Uh and there
  714. 28:59are different things that are important
  715. 29:00for us in our codebase and uh when you
  716. 29:05are able to encode all of that into
  717. 29:06skills and you can verify that your
  718. 29:08agent is actually doing them that allows
  719. 29:10you to really kind of ascend this curve
  720. 29:12and um uh you know start automating
  721. 29:16things. Uh there's another piece I
  722. 29:18wanted to talk about. Um if there's more
  723. 29:22>> Yeah, go for it. I'll I'll pick up more
  724. 29:23questions as I go. But
  725. 29:24>> yeah, I think there's a third part to
  726. 29:25this which I haven't talked about yet,
  727. 29:27which is kind of an interesting one
  728. 29:29which is like refactoring and rewriting.
  729. 29:32Like one of the uh I guess most
  730. 29:34controversial one of the most
  731. 29:36controversial topics in the industry I
  732. 29:38think is like should you rewrite your
  733. 29:40app or not? Um because I think engineers
  734. 29:44are very prone to this where especially
  735. 29:46when you join a company you come in and
  736. 29:48you see like the codebase and you're
  737. 29:50like oh man this is like who wrote
  738. 29:52this code you know it's terrible I want
  739. 29:54to rewrite the whole thing there is a
  740. 29:56very common inclination and I think a
  741. 29:59lot of you know before agents um and I
  742. 30:02guess arguably even now people will
  743. 30:04definitely discourage you from re
  744. 30:06rewriting stuff but I'm actually here to
  745. 30:08make a case for why you might want to
  746. 30:10consider
  747. 30:12Um because
  748. 30:15I think it really depends. Uh you know
  749. 30:17uh brownfield applications I think are
  750. 30:19actually in a pretty good spot
  751. 30:21especially if they're set up well
  752. 30:23already. Uh and like recently I've been
  753. 30:26talking to some people but uh you know I
  754. 30:28I was just observing I I just noticed
  755. 30:31this parallel which is that a lot of big
  756. 30:35tech company problems are now
  757. 30:37everybody's problems. Um, and the big
  758. 30:39tech company problem, you know, like
  759. 30:41when I was working at Meta, like we had
  760. 30:42this giant monor repo, we had like, I
  761. 30:45don't know, tens of thousands of
  762. 30:46engineers just, you know, like banging
  763. 30:49on their keyboards and and shipping code
  764. 30:51and
  765. 30:53a lot of really great engineers at Meta.
  766. 30:55Uh, but, uh, I'll say like, you know,
  767. 30:58you'll be surprised that the code
  768. 30:59quality is actually not that good.
  769. 31:01[laughter]
  770. 31:02Um, and so I often joke that like, you
  771. 31:04know, before AI slop, we had human
  772. 31:06sloth. Um and so uh you know I think a
  773. 31:09lot of big tech infra like uh like what
  774. 31:12Meta has or Google you know you know
  775. 31:14really big tech companies are actually
  776. 31:17designed for that where you you're sort
  777. 31:18of like you're catering to the the you
  778. 31:21know like uh this sounds so bad to say
  779. 31:23but like the the least capable engineer
  780. 31:25on your team right you build you build
  781. 31:28frameworks you build conventions you
  782. 31:30build guard rails you know you restrict
  783. 31:32credentials so that you know your intern
  784. 31:34doesn't wipe your production database.
  785. 31:37Um
  786. 31:39there's uh you know if you have that
  787. 31:41level of infra already I think your
  788. 31:43agents can actually already do a very
  789. 31:45solid job right because they have the
  790. 31:48the guard rails are already in place for
  791. 31:51agents to not cause havoc or not cause
  792. 31:53too much havoc uh in your codebase um
  793. 31:56and you can always add more you know
  794. 31:58guard rails. Uh but I think like green
  795. 32:01field applications especially are you
  796. 32:02know like the brand new applications are
  797. 32:05like the biggest risk in my opinion. Uh
  798. 32:08and also the greatest opportunity
  799. 32:10because you know if you vibe code a
  800. 32:12project uh a prototype um like we did
  801. 32:15for Grockbot you know Grockbot was spun
  802. 32:17up very very very quickly. Um and if you
  803. 32:20if you haven't heard of of Grockbot it's
  804. 32:22like a a new application we just
  805. 32:24launched yesterday. Uh it's it's really
  806. 32:26cool. uh lets you orchestrate your
  807. 32:29create like individual agents that have
  808. 32:31their own identity and you can kind of
  809. 32:33orchestrate them. It's super cool.
  810. 32:34Definitely check it out. Um but yeah,
  811. 32:36that was it's like a very it was a very
  812. 32:38green field application like most
  813. 32:40prototypes are so like vibe coded very
  814. 32:43quickly. Humans were not reading the
  815. 32:45code at all. And uh I had this tweet
  816. 32:49recently uh where I said something about
  817. 32:52organic architecture. Um
  818. 32:56maybe I'll find it. Uh but the idea is
  819. 32:59that
  820. 33:01uh when you have a completely vibe coded
  821. 33:03application, you essentially have no
  822. 33:05guardrails whatsoever. So uh your agents
  823. 33:10when you give them a task, they will
  824. 33:12just solve it in whatever method is the
  825. 33:14most convenient. And over time you get
  826. 33:16into this uh situation where you have a
  827. 33:20codebase that is spiraling out of
  828. 33:22control because you don't understand it.
  829. 33:24Uh your agents understand it I guess in
  830. 33:27a way but like they've built something
  831. 33:29that is you know optimized for short for
  832. 33:31shortcuts. Uh and uh you know it will
  833. 33:35you will suffer you'll have a lot of of
  834. 33:37issues with that application.
  835. 33:40Uh so I think starting your codebase
  836. 33:43with uh like very strong constraints is
  837. 33:47very much needed uh because like when
  838. 33:50you have a codebase that you can trust,
  839. 33:52right? when you have guardrails that
  840. 33:54actually help you uh uh help your agents
  841. 33:58write good code, you can get into the,
  842. 34:00you know, like into this part of the
  843. 34:02curve where I I where like I I said, you
  844. 34:05know, I woke up today and I had like 20
  845. 34:06PRs merged u by my agents and that's
  846. 34:10because I invested a lot a lot of time
  847. 34:13uh over 600 PRs I I I calculated
  848. 34:16yesterday uh when I refactored all of
  849. 34:19Grockbot to this new architecture that
  850. 34:21I've been thing. Um,
  851. 34:24and yeah, I've gotten to a point where I
  852. 34:28I don't really look I really don't look
  853. 34:29at the code anymore. And um, I say that
  854. 34:32not just, you know, to sell you tokens,
  855. 34:34but because I, you know, it it it took a
  856. 34:38lot of work to get to that point. I
  857. 34:39spent a lot of tokens to get the
  858. 34:41codebase to this point where I no longer
  859. 34:43have to look at it. Uh, but I'm very
  860. 34:46excited because, you know, of the
  861. 34:48potential where, you know, it's not just
  862. 34:50this doesn't just benefit me. It
  863. 34:51benefits everyone contributing to
  864. 34:53Grockbot and it also empowers you know
  865. 34:56designers and product managers and you
  866. 34:59know pe uh even GTM people to add
  867. 35:02features to grabbot and I don't have to
  868. 35:04worry you know I don't have to to wake
  869. 35:06up at night in in the middle of the
  870. 35:07night and worry like oh someone's
  871. 35:09just merged a perf regression right I
  872. 35:11have a ton of constraints and CI it's
  873. 35:14like it's actually very annoying to
  874. 35:16write code in in graph web but like
  875. 35:18agents absorb all of that annoyance
  876. 35:21Um but yeah, I'm happy to talk about
  877. 35:23what exactly that is. Um
  878. 35:26>> yeah, I think one question um
  879. 35:29>> before we get into the this part here is
  880. 35:31just around that element of like what
  881. 35:33your your your CI looks like or maybe
  882. 35:35some of the constraints and then also
  883. 35:36like the average PR size. I saw a
  884. 35:38question about that earlier just to give
  885. 35:40people you know kind of a a glance. It
  886. 35:42doesn't have to be like mathematically
  887. 35:43average but just a you know like what
  888. 35:46generally the size of the a PR is. um if
  889. 35:49it's only a couple lines of code or you
  890. 35:50know um yeah
  891. 35:52>> um
  892. 35:54I think it depends uh let me
  893. 35:58>> I'm trying to do this in a way where I'm
  894. 36:00not going to like
  895. 36:01>> you yeah you don't have to share the
  896. 36:02actual number like an actual average
  897. 36:04>> I think this is fine
  898. 36:05>> benchmark
  899. 36:06>> but like we have so okay this is not
  900. 36:09that interesting but uh well fun fact is
  901. 36:12that virtualization in grogbot and in uh
  902. 36:16cursor is actually powered by uh pretext
  903. 36:20uh which is a sort of new library that
  904. 36:23someone's built. Um that's really
  905. 36:26interesting. You should you should check
  906. 36:27it out. But that's not really that
  907. 36:29important. Uh I think the average PR
  908. 36:31size I actually don't know. I I don't
  909. 36:34know if I want to click on these. Uh I
  910. 36:36probably can, but I would say
  911. 36:38[clears throat] like they can range
  912. 36:39anywhere from a few hundred lines or 50
  913. 36:41lines to like a thousand depending on
  914. 36:44what the thing is doing. Uh, so like
  915. 36:47here I'm actually like deleting a bunch
  916. 36:48of files. So I expect that it's just
  917. 36:50this like mostly deletion. Uh, but yeah,
  918. 36:53it kind of varies.
  919. 36:55There's no like Yeah,
  920. 36:57>> there's no like hard cap or hard limit.
  921. 36:58Are there all like 50 line PR?
  922. 36:59>> There's no hard cap. Yeah, there's
  923. 37:00definitely no hard cap, but I I do
  924. 37:01encourage my agents to split up their
  925. 37:03work into multiple PRs. Uh I do that
  926. 37:06mostly because uh I like I like the idea
  927. 37:11of the I guess maybe this is much harder
  928. 37:13to do now as in the world of agents and
  929. 37:15you have like so many commits but I like
  930. 37:18the idea that you know the git history
  931. 37:19is a very rich source of context. Uh,
  932. 37:22and I like the I like each PR to sort of
  933. 37:25atomically describe what that small
  934. 37:29piece of thing is doing, which also
  935. 37:31makes it easier for me to revert changes
  936. 37:32and like figure out, you know, oh, I
  937. 37:34shipped a bug and it's just it's here,
  938. 37:36right? It's not in this 40,000 line PR
  939. 37:39where you who knows what landed in
  940. 37:41there.
  941. 37:43Uh, but I don't have a hard cap on PR
  942. 37:46size.
  943. 37:47>> Cool. And then um yeah, also quick
  944. 37:50question on like CI. So again, you don't
  945. 37:51have to go into like uh the screen share
  946. 37:53of like your CI does, but just generally
  947. 37:56would you describe what the CI kind of
  948. 37:57looks like uh or how strict it is?
  949. 38:01>> Uh yeah. So uh well specifically for
  950. 38:04Grockbot. So Dune is the is the sort of
  951. 38:07cheeky code code name for the
  952. 38:09architecture that we've built for
  953. 38:12Grockbot. Um the CI looks pretty
  954. 38:16annoying because there's checks for
  955. 38:18everything. So like literally I have um
  956. 38:22uh well if you've written any react for
  957. 38:23example you know you know that one of
  958. 38:25the biggest foot guns in react is use
  959. 38:27effect. Uh so in
  960. 38:30uh Dune and in graphbot we've banned use
  961. 38:33effect. So Dune is just you can the the
  962. 38:36mental model of what Dune is uh you can
  963. 38:38kind of think of it as like Nex.js JS
  964. 38:40for uh electron apps and it's designed
  965. 38:43for agents to write uh and it's like
  966. 38:46custom for you know our agent powered
  967. 38:48applications. Um so the CI checks are
  968. 38:52very like specific to that like you know
  969. 38:53don't use use effect it's it's it's
  970. 38:56banned like CI will fail uh and yell at
  971. 38:58you. We have like some of the more
  972. 39:01interesting ones that people might raise
  973. 39:02eyebrows is like I actually ban code
  974. 39:05comments as well uh which is very
  975. 39:07interesting. Uh, but I've noticed that
  976. 39:1199% of the time agents just write code
  977. 39:14comments that kind of describe some
  978. 39:16historical thing that is actually
  979. 39:18totally irrelevant to the code. Um, like
  980. 39:21it will often say like, you know, oh,
  981. 39:22Lauren said you should never do this and
  982. 39:24it's now in in a code comment. Like
  983. 39:26what? Like why what that was? I didn't
  984. 39:28say that as like a durable, you know,
  985. 39:30global rule. I just meant like your this
  986. 39:33PR sucks and you should change that
  987. 39:35part.
  988. 39:36But agents don't really understand us
  989. 39:39that well surprisingly uh and or they
  990. 39:41kind of assume too much and they kind of
  991. 39:44do things in like very stupid ways. So
  992. 39:46like yeah we just ban everything.
  993. 39:48Everything you can imagine like the
  994. 39:50agents are bad at we ban. Uh so one
  995. 39:54example that we actually suffer a lot in
  996. 39:56the agents window is we have uh you know
  997. 40:00you if you've used the agents window
  998. 40:01you've definitely seen performance
  999. 40:02issues and you know we're constantly
  1000. 40:04trying to fix them. Uh but it's like a
  1001. 40:07it's a never- ending struggle because
  1002. 40:09there's so many pull requests that get
  1003. 40:11merged. Every any one of them could just
  1004. 40:13regress performance or stability or
  1005. 40:15reliability. Uh you know the agents
  1006. 40:18window doesn't have this architecture
  1007. 40:19yet. I plan to do bring this learning
  1008. 40:21back there and kind of refactor
  1009. 40:24everything there. uh but uh it just
  1010. 40:27regresses super often uh because uh
  1011. 40:30there's just one example is like we have
  1012. 40:33very poor um isolation between
  1013. 40:35processes. So like on you know on on
  1014. 40:38Electron you have a renderer thread that
  1015. 40:40renders your UI but you also have like a
  1016. 40:42main thread that you can run other code
  1017. 40:44that you know doesn't need to block the
  1018. 40:46renderer.
  1019. 40:48Um, but we do a poor job of separating
  1020. 40:50those things. And so oftentimes you just
  1021. 40:52accidentally have code that gets pulled
  1022. 40:54into running on the renderer thread and
  1023. 40:57then all of a sudden you're competing
  1024. 40:58with the the renderer that you know that
  1025. 41:01has a very if you want like 60 fps you
  1026. 41:04have to every frame that gets drawn has
  1027. 41:06to be done in 16 milliseconds. So very
  1028. 41:09very small you know deadline per frame
  1029. 41:12uh if you want you know a very smooth
  1030. 41:14product. Uh and when you start building
  1031. 41:16bringing in accidentally bringing in you
  1032. 41:18know things that are like very
  1033. 41:20computationally heavy or they have a lot
  1034. 41:22of IO uh then you just get into like a
  1035. 41:25lot of jank right your your FPS really
  1036. 41:27drops you start uh you know losing
  1037. 41:30frames you get long tasks that take more
  1038. 41:33than 16 milliseconds and you just get
  1039. 41:34this really choppy experience.
  1040. 41:37So all of those patterns that we've
  1041. 41:38learned basically building electron apps
  1042. 41:40we've encoded into this framework and it
  1043. 41:43becomes like a hard failure. So I
  1044. 41:45literally in in grabbot we literally
  1045. 41:47have a directory called electron main
  1046. 41:49electron renderer and we have uh import
  1047. 41:54uh CI I guess where we actually check
  1048. 41:57the dependency graph to make sure you're
  1049. 41:59not accidentally importing code from one
  1050. 42:01directory to another. Uh so that's
  1051. 42:04enforced by CI um as well as bug bots uh
  1052. 42:09which is our which cursors um like code
  1053. 42:12review tool that runs on CI uh you know
  1054. 42:15in our agents MD it's everywhere like so
  1055. 42:17I I I I um I have this thing here where
  1056. 42:21I I talk about like um you know like
  1057. 42:24there are multiple layers I think for
  1058. 42:26building a good codebase. Uh obviously
  1059. 42:29the codebase is one where uh if you have
  1060. 42:31an architecture like this where it's
  1061. 42:34extremely strict uh you know the the the
  1062. 42:37way to build features is very
  1063. 42:38conventional that's like the strongest
  1064. 42:40strongest level of enforcement because
  1065. 42:43agents just love to copy existing
  1066. 42:45patterns. So, uh, one example of this in
  1067. 42:48Rockbot is like we have this these
  1068. 42:50concepts called like a feature and we
  1069. 42:52have entry points and transcript cards
  1070. 42:54like oh you know the cards that you see
  1071. 42:56in the chat these are all like like
  1072. 42:59nouns I guess in in the framework and so
  1073. 43:02there's a very conventional way of
  1074. 43:04creating them and so like a feature is
  1075. 43:07all in in a single directory as an
  1076. 43:09example and so all of the code that
  1077. 43:11contributes to that feature lives in one
  1078. 43:13directory so it's all coll-located in
  1079. 43:15one case makes it super easy. You know,
  1080. 43:17agents don't have to like uh grap around
  1081. 43:19and try to figure out like where all the
  1082. 43:21things are. It just looks at the feature
  1083. 43:23and like, oh, okay, I'm working on the
  1084. 43:25onboarding feature in Grockbot. Uh I'm
  1085. 43:29just going to work in this directory.
  1086. 43:30And for 80% of the work, it's mostly
  1087. 43:32just very encapsulated there.
  1088. 43:34[clears throat]
  1089. 43:35But uh like it's like very it's like
  1090. 43:38designed again for you know like the
  1091. 43:40dumbest agent like you don't have to
  1092. 43:42think, right? the the the one of the key
  1093. 43:45principles I have for this framework is
  1094. 43:47like the shortest the shortest path is
  1095. 43:50the best path.
  1096. 43:52So uh because that plays exactly to how
  1097. 43:55agents love to write code is like they
  1098. 43:56like to take shortcuts really you know
  1099. 43:59they they'll find the quickest way to
  1100. 44:01solve the problem. So why not make that
  1101. 44:04the best way to solve the problem? Uh so
  1102. 44:07I I I probably won't get into all the
  1103. 44:09specific details. Um and uh the this
  1104. 44:13framework is really more of a collection
  1105. 44:14of ideas and principles rather than
  1106. 44:16something that will open source. Uh you
  1107. 44:19can you can you know screenshot this I
  1108. 44:20guess if you want and uh tell your agent
  1109. 44:23to uh do some build build something like
  1110. 44:26this for you too.
  1111. 44:28Um yeah, but it's really all about the
  1112. 44:30layers uh you know like the the codebase
  1113. 44:33is one part with features uh and
  1114. 44:35directories and you know import or
  1115. 44:38blocking import dependencies uh that
  1116. 44:40shouldn't be imported uh but and and it
  1117. 44:43all enforces that and static analysis.
  1118. 44:45So like uh there's CI checks. We have a
  1119. 44:48lot of lints for bad patterns that we
  1120. 44:51observe. Uh compiler diagnostics. Uh
  1121. 44:55there's also rules and bugbot which are
  1122. 44:58um I think like three, four, five are
  1123. 45:00more soft, right? These two actually
  1124. 45:03make make CI red, right? So that you
  1125. 45:07know there's a hard constraint where the
  1126. 45:09agent can't just write crappy code
  1127. 45:12for rules and skills and bogbot. Your
  1128. 45:16agents can still forget, right? You can
  1129. 45:18still or it may not always consistently
  1130. 45:21apply them. So I like to layer them, but
  1131. 45:25I don't I don't like to rely on them as
  1132. 45:28the only source of enforcement because
  1133. 45:31it's very very soft, right? And if you
  1134. 45:32if you only have rules and bug bot and
  1135. 45:34skills and a style guide for your code,
  1136. 45:37you will it's only a matter of time
  1137. 45:39before your codebase looks like complete
  1138. 45:41trash. I'm sorry to say that but uh I
  1139. 45:44definitely recommend yeah like you know
  1140. 45:45investing in you know things that can be
  1141. 45:48hard and forced right and this is why
  1142. 45:51you know maybe the choice of tech stack
  1143. 45:54that you use is also very important. Um
  1144. 45:57like I think for example Rust is sort of
  1145. 45:59making you know it's like getting super
  1146. 46:01popular again uh because the compiler is
  1147. 46:05so strict right the compiler enforces so
  1148. 46:07many different things you know there's a
  1149. 46:09borrow checker that you have to appease
  1150. 46:11and if as long as you make sure your
  1151. 46:12agents don't write unsafe code blocks uh
  1152. 46:15you can more or less feel somewhat
  1153. 46:17confident that if the code compiles it
  1154. 46:20probably works and it's good. Uh but you
  1155. 46:22see it gives you that level of trust and
  1156. 46:25confidence that you as a human engineer
  1157. 46:28no longer need to go and check it
  1158. 46:31yourself. You know you you rely on code
  1159. 46:34and static analysis to actually make
  1160. 46:37that uh a lot smoother. Um, and I I
  1161. 46:42guess the worst part, the worst place to
  1162. 46:44be in is if you are stuck in code review
  1163. 46:47land where you actually enforce all of
  1164. 46:49the constraints, the invariance in your
  1165. 46:51codebase by literally the human person
  1166. 46:55saying, you know, reading the code and
  1167. 46:56like, okay, you should not do this,
  1168. 46:58right?
  1169. 46:59Every time you have to do that, you
  1170. 47:00should consider that as a code smell,
  1171. 47:02like a anti- pattern. And you should
  1172. 47:05say, okay, instead of me commenting on
  1173. 47:06the PR, how do I turn this into a hard
  1174. 47:10rule, right? How do I turn this into a
  1175. 47:12lint rule, how do I turn this into a CI
  1176. 47:14failure, or how do I even categorically
  1177. 47:17eliminate this problem uh entirely? Uh I
  1178. 47:22I can talk about another migration I've
  1179. 47:23done, but I'll probably pause here.
  1180. 47:25>> Sure.
  1181. 47:26>> Yeah. I feel like that's that's where I
  1182. 47:28am to be honest is is what you're
  1183. 47:29describing right now which is that like
  1184. 47:31I don't have all of these rules. So I
  1185. 47:33have some things to go do after this
  1186. 47:34session in terms of being able to scale
  1187. 47:36my agents. I'm I'm definitely on like
  1188. 47:38the uh you know maybe a couple of
  1189. 47:40parallel ones locally stage. So like two
  1190. 47:42to three locally and I'm sure most
  1191. 47:44people here are on the same. So uh yeah
  1192. 47:47I know we only have a couple minutes
  1193. 47:48left. Lauren, was there anything else
  1194. 47:49that you wanted to to highlight? I
  1195. 47:51obviously there's lots of questions so I
  1196. 47:52can grab more but I want to give you a
  1197. 47:54few minutes if there's anything else you
  1198. 47:55want to talk about.
  1199. 47:56>> I think I've been yapping for quite a
  1200. 47:57lot so I'm maybe let's just do
  1201. 47:59questions.
  1202. 48:00>> Okay cool. Uh one question that had uh a
  1203. 48:03couple of uh came up a couple times was
  1204. 48:04just around like token usage.
  1205. 48:07>> So the the question is like is is what
  1206. 48:09you're describing a realistic thing for
  1207. 48:11people who are on you know uh a normal
  1208. 48:15set of token usage. They don't have you
  1209. 48:17know basically unlimited tokens uh to
  1210. 48:19work with.
  1211. 48:21I think that's a really good point. I
  1212. 48:22mean, like obviously, you know, I work
  1213. 48:24at a AI lab where we have unlimited
  1214. 48:26tokens. So, uh I definitely cannot
  1215. 48:31say that, you know, this is something
  1216. 48:32everyone should do in the exact same way
  1217. 48:35that I did it. I think it's possible to
  1218. 48:37get to this point without, you know,
  1219. 48:38breaking the bank.
  1220. 48:41But, you know, if you're like an
  1221. 48:42engineering leader or, you know, you're
  1222. 48:44you you have a startup that you lead, um
  1223. 48:47I think to me it's a question of ROI. Um
  1224. 48:50and it's like uh yes you spend a lot of
  1225. 48:54money on tokens in the upfront stage you
  1226. 48:57know like refactoring a codebase is
  1227. 48:59going to take a lot of tokens uh adding
  1228. 49:01all these things uh is going to take a
  1229. 49:02bunch of tokens but if we're heading to
  1230. 49:05a world where agents are writing all the
  1231. 49:07code and you know you want to be very
  1232. 49:11lean right you don't want to have to
  1233. 49:13hire you don't want to be you don't want
  1234. 49:14to become like meta right like I mean
  1235. 49:16like in terms of you don't want to
  1236. 49:17become a 10,000 person engineering org
  1237. 49:20because I mean that's a cool problem to
  1238. 49:23have but also you you have so much
  1239. 49:25overhead there's like planning you know
  1240. 49:27like you it's it's a personally I I
  1241. 49:30wouldn't uh it it's not super fun but um
  1242. 49:34I think you want to stay very nimble
  1243. 49:36right and you want to you want to be
  1244. 49:38like agents are all about allowing you
  1245. 49:40to do things that you couldn't do before
  1246. 49:42that's really to me like the value of
  1247. 49:44agents you know it's not just storing
  1248. 49:47tokens on every single little thing but
  1249. 49:49um to me like the thing I couldn't do
  1250. 49:51before is like enforce this level of
  1251. 49:54constraints in a codebase by myself
  1252. 49:57right like I'm just a single person you
  1253. 50:00know uh it would have taken me years to
  1254. 50:02build this framework uh and do all the
  1255. 50:05refactoring and test everything myself
  1256. 50:09and verify you know like run imagine if
  1257. 50:11there it was just me right now in in
  1258. 50:13pre- agent era just like running you
  1259. 50:16know by it would take me so long, right?
  1260. 50:18And my salary is pretty high, right?
  1261. 50:20Like, so, you know, the the question I
  1262. 50:23think an engineering leader might have
  1263. 50:25is just then, you know, like what is
  1264. 50:27there's a trade-off of do you hire
  1265. 50:30someone to do this or do you spend the
  1266. 50:32tokens to set up a code base so that
  1267. 50:34even the the most naive, right, the
  1268. 50:38dumbest agents can do a good job. And
  1269. 50:41when you actually get to this point like
  1270. 50:42even agents that are not you know fable
  1271. 50:45size do an excellent job of writing code
  1272. 50:48and this pays a lot of dividends as well
  1273. 50:50for me personally where I've empowered
  1274. 50:53not just myself but again like PMs
  1275. 50:57designers engineers who are not familiar
  1276. 50:59with Grockbot to just contribute in a
  1277. 51:02way that is sustainable.
  1278. 51:05So, I think yeah, it's definitely like a
  1279. 51:06trade-off for sure. You know, like
  1280. 51:08nothing is like free for sure. Uh, and
  1281. 51:11tokens are pretty expensive. Uh, but oh,
  1282. 51:13actually, uh, I I I don't know how many
  1283. 51:16of you have seen this, but we actually
  1284. 51:18announced Grock 4.6 today. So, very
  1285. 51:20exciting. Finally out. Um, so yeah,
  1286. 51:23Groth 4.6 would be like a great it was
  1287. 51:25very very smart. Uh, it's really good on
  1288. 51:28the on the benchmarks. Uh, and it's the
  1289. 51:30same the tokens. Uh well uh I hopefully
  1290. 51:33I'm not saying this incorrectly, but uh
  1291. 51:35I believe the cost per token is the same
  1292. 51:38as 4.5. So you're actually getting more
  1293. 51:41intelligence for the same cost. Uh I
  1294. 51:45think this is an area that cursor tries
  1295. 51:47to cursor and SpaceX AI try to really
  1296. 51:50optimize for like that heredto frontier
  1297. 51:53of you know cost versus intelligence. uh
  1298. 51:56you know we don't necessarily want to
  1299. 51:57build the biggest model ever because
  1300. 51:59that is extremely expensive to run. It's
  1301. 52:02really about like how do you find that
  1302. 52:04sweet spot right? You don't you don't
  1303. 52:05need a giant model but it's just super
  1304. 52:07smart right and it's not very expensive
  1305. 52:09for inference.
  1306. 52:11Uh but um yeah, I think to kind of round
  1307. 52:15it up, um I think it's like a it's it's
  1308. 52:18there's a if you do your own analysis, I
  1309. 52:20feel like it's pretty positive. It it'll
  1310. 52:23be pretty positive that the ROI you get
  1311. 52:25from investing in stuff like this, uh
  1312. 52:28just empowers not just yourself, but
  1313. 52:30your whole team to be so much more
  1314. 52:32productive, right? Like imagine if you
  1315. 52:34have an army of engineers like me who
  1316. 52:36are shipping so much improvements and
  1317. 52:39and bug fixes. uh you know every day,
  1318. 52:42right? Like that is pretty exciting.
  1319. 52:46>> Cool. Uh one last question before we
  1320. 52:48wrap up. This one is for the people in
  1321. 52:50product on the on the call.
  1322. 52:51>> So let's say we do have an army of
  1323. 52:53engineers who are shipping like Lauren.
  1324. 52:55I'm just curious like how is the product
  1325. 52:58team or other functions of your company
  1326. 52:59keeping up given that like if you're
  1327. 53:02shipping so quickly, have are they using
  1328. 53:04AI more to do their jobs? like as much
  1329. 53:06as you can speak to that obviously you
  1330. 53:08don't have like you're not in that role
  1331. 53:09but just curious about how that works.
  1332. 53:12Um I think this is where Grothbot has
  1333. 53:14been actually exceedingly powerful. Uh
  1334. 53:17where so before grabbot like you know uh
  1335. 53:20obviously cursor only had cursor like we
  1336. 53:22only had agents window we had a CLI we
  1337. 53:25had an IDE and these are really like
  1338. 53:28power user tools right like devel
  1339. 53:30they're designed for developers so it's
  1340. 53:32very very developerentric you can do
  1341. 53:34knowledge work in them but it like the
  1342. 53:36UI is not really optimized for that. So
  1343. 53:40we actually didn't really have uh well I
  1344. 53:42think like a lot of people like you know
  1345. 53:44GTM product like they might have used
  1346. 53:46cursor uh to do their work but it
  1347. 53:49definitely wasn't like a delightful
  1348. 53:50experience for them. Um I think now with
  1349. 53:54Grogbot
  1350. 53:55uh it's become Grogbot is basically like
  1351. 53:59their kusher moment for people who are
  1352. 54:01not in tech in my opinion like it's like
  1353. 54:03it's like a very very accessible way to
  1354. 54:06use agents in a very comfortable very
  1355. 54:09familiar interface. It looks like
  1356. 54:10iMessage. Um and it's very fun to you
  1357. 54:14know you can give your agent a fun name.
  1358. 54:16uh you can have you can kind of do
  1359. 54:18orchestration with in a very like
  1360. 54:20natural way where you can sort of you
  1361. 54:22know each agent is like a person right
  1362. 54:23now you got a team of agents like
  1363. 54:25working on you have one one agent per
  1364. 54:27account that you manage as an example or
  1365. 54:29if you're a PM you have you know you can
  1366. 54:31have an agent that summarizes all the
  1367. 54:32work that Lauren did last night and then
  1368. 54:34now you know what I did right so I think
  1369. 54:37our PMs are leveraging that a lot and
  1370. 54:40they're shipping code too uh so you know
  1371. 54:42like often times they will just say oh
  1372. 54:44here's a bug I fixed it can you look at
  1373. 54:45it and then I'll go review it and
  1374. 54:47actually it's just perfect. I'm like
  1375. 54:49okay stamp. Uh so uh that I think that
  1376. 54:52shows that you know the the Dune
  1377. 54:53architecture is holding up right the all
  1378. 54:56the the really strict constraints allow
  1379. 54:59people who are not experts in
  1380. 55:01engineering to contribute at a high
  1381. 55:03level. Uh so I'm I feel like I'm already
  1382. 55:05seeing that pay off a lot where uh you
  1383. 55:08know designers and PMs are just able to
  1384. 55:10to to ship features directly. Um, and
  1385. 55:14that just makes the Grockbot team super
  1386. 55:18fast, right? Where we can ship so
  1387. 55:20quickly. Um, and we have a lot planned.
  1388. 55:24So, I'm very excited uh to, you know, uh
  1389. 55:27to to ship more ship more

About this transcript

This page contains the full transcript of SpaceX AI Engineer Lauren Tan: Part 1 by Ai insider, generated from the public captions YouTube serves with the video. The transcript has 10,172 words across 1,389 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.