Rethinking Claude Code Best Practices — Lydia Hallie, Anthropic — Transcript

Full transcript

1. 0:00I had some technical difficulties during
2. 0:03my talk this morning, so I ran back home
3. 0:05and I'm just going to record it from my
4. 0:07computer instead. So, hi everyone. I'm
5. 0:09Lydia. I work on the Cloud Code team at
6. 0:11Anthropic. And the models have gotten
7. 0:14really good lately. So we have Opus 55,
8. 0:16which is my favorite daily driver. And
9. 0:19earlier this week, we also launched
10. 0:20Fable 52. And these models are extremely
11. 0:23good at long-running tasks and
12. 0:25understanding your codebase. And
13. 0:27especially Fable is also really good at
14. 0:29verifying its own output. But as these
15. 0:31models get better, the way that we work
16. 0:33with them in Cloud Code also changes.
17. 0:36So earlier this year, I don't know if
18. 0:38you remember, but sessions kind of used
19. 0:40to look like this. So we'd use a model
20. 0:41like Opus 45, which honestly for its
21. 0:44time was really good. I really enjoyed
22. 0:45working with Opus 45. But compared to
23. 0:48today, it would just stop a lot earlier.
24. 0:50So it would just run for shorter amounts
25. 0:52of time.
26. 0:53And we'd also keep correcting it. We'd
27. 0:55keep reminding it to do things, add
28. 0:57things like important, exclamation mark,
29. 0:59all these things to our Cloud MD file.
30. 1:01And also features like auto mode didn't
31. 1:03exist yet. So we were either approving
32. 1:05everything manually or taking the
33. 1:08riskier approach with dangerously skip
34. 1:10permission. So we were still very much
35. 1:13in the loop as humans.
36. 1:15But as these models got better at taking
37. 1:17on bigger tasks, our sessions also
38. 1:19changed. So nowadays, at least at
39. 1:21Anthropic, our sessions look more like
40. 1:23this. So we're running dozens of
41. 1:24sessions at the same time, running for
42. 1:26hours, verifying their own results along
43. 1:29the way, and we even have sessions
44. 1:31orchestrate other sessions. So whereas
45. 1:33we used to be in the loop for almost
46. 1:35every step and running one or just a
47. 1:37couple sessions at a time, we've now
48. 1:39essentially become managers for dozens
49. 1:42of sessions.
50. 1:43We also got a lot more done though. Now
51. 1:45you may have seen our blog post in
52. 1:46October about how Anthropic went from
53. 1:48about 2,500 changes a day in May to more
54. 1:51than 30,000 a day in September. But this
55. 1:55also means that we as developers just
56. 1:57have to rethink on what we're focused on
57. 1:59instead.
58. 2:00So, if you think about our best
59. 2:01practices from earlier this year, it was
60. 2:03mainly about steering sessions,
61. 2:05approving commands, saying no to certain
62. 2:08things or restricting what Claude could
63. 2:09do,
64. 2:11the output, and also starting the
65. 2:12sessions. So, everywhere that we now
66. 2:15step out of that loop, something else
67. 2:17has to take over. And this is either the
68. 2:19model, the harness, or the environment
69. 2:22that it's running in. So, if we are not
70. 2:24in the loop to approve every command and
71. 2:26we're running so many sessions at the
72. 2:27same time, how can we make sure that the
73. 2:30code it's writing is still high quality?
74. 2:32And most importantly, if these sessions
75. 2:34are running for hours without our
76. 2:36supervision, how can we make sure that
77. 2:38they're running safely? So, let's go
78. 2:40through that by taking this old session
79. 2:42and seeing what we have to change to get
80. 2:44to that end result that I showed
81. 2:45earlier.
82. 2:46Well, first of all, let's use the actual
83. 2:47defaults we start sessions in nowadays.
84. 2:49So, that's not really something you
85. 2:51should think about. So, let's update the
86. 2:52model here from Opus 4 5 to Opus 5 5 and
87. 2:55change the manual permission to auto
88. 2:57mode. And Opus 5 5 is a lot better at
89. 3:00understanding your intent and
90. 3:01understanding your code base. So, we
91. 3:03really don't have to steer it as much
92. 3:04anymore mid session. And you also no
93. 3:06longer have to approve every command
94. 3:08because we started making auto mode the
95. 3:10default permission mode back in August.
96. 3:12Now, being better at understanding your
97. 3:14intent and your code base also means
98. 3:16that we have to rethink the instructions
99. 3:18we have in our code base. So,
100. 3:20if you followed our best practices from
101. 3:21earlier this year, you may still have
102. 3:23things like important or never do this,
103. 3:25exclamation mark, all these things in
104. 3:27your cloud MD file. But with today's
105. 3:29models, you really don't need that and
106. 3:31it can actually make the model go off
107. 3:32track. So, you end up with more tool
108. 3:34calls and overall a worse results.
109. 3:37So, nowadays your cloud MD file should
110. 3:39only include the things the Claude
111. 3:40cannot see from your code. So, for
112. 3:42example, maybe you have a service that's
113. 3:44still using some legacy source
114. 3:46somewhere, but it's not directly
115. 3:47referenced in the code. That's still
116. 3:49good to keep in there. But everything
117. 3:51else, like what stack you're on or where
118. 3:53to look, you really don't need that
119. 3:54anymore. And we actually have a command
120. 3:56for this called doctor prompt audit,
121. 3:58which makes it really easy to optimize
122. 4:00your environment for a specific model.
123. 4:02So, if on Opus 55 here and you run this
124. 4:05command, it will automatically check
125. 4:06your cloud MD and your skills to see
126. 4:08what the new model no longer needs.
127. 4:11Now, auto mode is a permission mode that
128. 4:13runs a separate classifier on pretty
129. 4:15much everything that isn't just reading
130. 4:17or editing files in your project.
131. 4:20So, it automatically allows commands
132. 4:22that it knows are safe, like running
133. 4:23tests or installing dependencies, but it
134. 4:26still blocks commands where it's unsure
135. 4:28whether they're safe.
136. 4:29And normally that's pretty good. Uh
137. 4:31normally that's kind of, you know, what
138. 4:32we want. But for example here, if we ask
139. 4:35it to ship something to staging, we
140. 4:37would have just, you know, approved the
141. 4:39push to our deploy repo. But by default,
142. 4:42auto mode only trusts this folder and
143. 4:44this repo, so it would block that
144. 4:45command. So, we're back in the loop.
145. 4:48Now, the defaults should be good in most
146. 4:50cases, but if you still notice that you
147. 4:51have to approve a lot of uh commands
148. 4:54even in auto mode, the first thing to
149. 4:56look at is settings here because you can
150. 4:58configure auto mode in your own settings
151. 5:00just in plain English, and this is what
152. 5:01gets sent to the classifier, which then
153. 5:03decides whether to reject or allow
154. 5:05certain commands.
155. 5:07So, here we can just write which repos,
156. 5:09buckets, domains we trust um and what it
157. 5:11should always allow or never do. Okay,
158. 5:13so honestly this was pretty easy by just
159. 5:15using today's defaults. We've already
160. 5:17gotten rid of most of our in-the-loop
161. 5:19stuff
162. 5:20um because the model took over the
163. 5:21steering and auto mode took over all the
164. 5:23approving, so getting out of the loop is
165. 5:25easy. But the harder question now is how
166. 5:28do we make sure it's okay that we're not
167. 5:29there anymore? So, first is keeping it
168. 5:32safe and then also verification.
169. 5:34So, let's start with keeping it safe
170. 5:35because everything so far, so upgrading
171. 5:37the model and using auto mode is just
172. 5:40working with a model.
173. 5:42And in most cases it does what you tell
174. 5:44it to do, but that's not a guarantee.
175. 5:46So, in a study we published back in
176. 5:48August, we tested auto mode versus real
177. 5:50people in the loop, and only about 14%
178. 5:53of people caught a clearly dangerous
179. 5:55command. Auto mode blocked 89% of them,
180. 5:58so it is a lot better at it than we
181. 6:00were.
182. 6:01But about one in 10 of those commands
183. 6:03still went through. So, if do want the
184. 6:05guarantee, you can only enforce that in
185. 6:08the harness. And this is where some of
186. 6:10our older best practices still matters,
187. 6:11and that's permissions. So, for example,
188. 6:13if you have a deny rule for a specific
189. 6:16tool called in your project, Cloud Code
190. 6:18looks at this rule before the automo
191. 6:20classifier even looks at it. In fact, if
192. 6:22a tool called matches a deny rule, the
193. 6:24classifier never even sees it. So,
194. 6:26having deny rules is still the best way
195. 6:28to make sure Cloud actually never does
196. 6:31something.
197. 6:32But then how do you know what needs to
198. 6:34be a rule in your project versus what
199. 6:35the model can just know because they've
200. 6:37gotten so good at it? Um I think it kind
201. 6:39of just comes down to if Cloud gets this
202. 6:41wrong once, can I take it back? So, if
203. 6:44it writes the wrong code, you can still
204. 6:46revert that. Um so, you can still leave
205. 6:48that to the model. But if it throws away
206. 6:50our uncommitted changes or it reads a
207. 6:52key that's now in your transcript and
208. 6:53you don't don't want that, that should
209. 6:55still be a rule. So, for example, let's
210. 6:58say that we just never want Cloud
211. 7:00reading our .env file. Now, by default,
212. 7:02automo is fine with this, so it would
213. 7:03have just approved it.
214. 7:05So so, if you want to make sure that
215. 7:07Cloud can never read this file, you
216. 7:09should just add a deny rule. And then
217. 7:11commands like cat cannot open this file
218. 7:12either. But it's not always that
219. 7:14straightforward cuz sometimes certain
220. 7:16commands can have side effects. So, for
221. 7:19example, let's say that npm test here, a
222. 7:21very normal command, it's just a test,
223. 7:23so automo runs it, that's fine.
224. 7:25But when the script is running, it runs
225. 7:27code that you didn't write, like your
226. 7:29dependencies. Or maybe Cloud runs a
227. 7:31script real quick that it wrote just to
228. 7:33quickly check something.
229. 7:35So, if one of those scripts is
230. 7:36compromised, it could just open that
231. 7:38same .env file or your SSH keys and send
232. 7:41those to a server you've never heard of,
233. 7:43right? And even if we were still there
234. 7:46as humans approving everything,
235. 7:48honestly, we would have just clicked
236. 7:49allow cuz there's nothing to say no to.
237. 7:50It's just part of the script that in
238. 7:53this example case npm test runs. And
239. 7:56permissions only decide if a tool call
240. 7:59runs, but they cannot limit what it can
241. 8:01reach once it's running. So, we want to
242. 8:03make sure that these kinds of scripts
243. 8:05can still run because this is just part
244. 8:07of, you know, how Cloud can check its
245. 8:08own work running tests.
246. 8:10So we cannot just block npm tests, but
247. 8:12we want to limit its side effects. So
248. 8:14really, we just want to limit the
249. 8:16environment that it's running in.
250. 8:18So if you're running it locally or on
251. 8:20your own machine, you can run commands
252. 8:22like that in a sandbox. So then you have
253. 8:24control over which files in a command it
254. 8:26can read, which folders it can write to,
255. 8:29which hosts it can reach. And the
256. 8:31operating system enforces that.
257. 8:34Also, this also applies to everything
258. 8:35that command then starts, so any child
259. 8:37processes.
260. 8:39So on its own, that deny rule gets
261. 8:41checked by Cloud before the command
262. 8:42runs, but all it sees there is npm
263. 8:45tests. So it doesn't see which files
264. 8:47that script ends up opening. But with
265. 8:49the sandbox on, that same rule gets
266. 8:51passed on to the operating system. So
267. 8:53now when when that dependency tries to
268. 8:56open or end file, the operating system
269. 8:57just doesn't let it.
270. 8:59And by default, it can still read most
271. 9:01of your machine, so you'd want to add
272. 9:02another rule for your SSH keys if you
273. 9:04care about that.
274. 9:06And it also cannot reach a host that
275. 9:08wasn't approved.
276. 9:10But the sandbox only covers the shell
277. 9:12commands that Cloud runs and whatever
278. 9:14those start. But Cloud's own files, your
279. 9:16MCP servers, and all of your hooks still
280. 9:18run outside of it. So the rest of your
281. 9:21session still runs on your own machine
282. 9:22and everything that's on there. So
283. 9:24ideally, you want this whole thing to
284. 9:26run somewhere where none of that is even
285. 9:28there in the first place. And that is a
286. 9:31Cloud session because there it gets its
287. 9:33own machine with a fresh copy of a repo.
288. 9:36And this runs even when we close our
289. 9:37laptop, which is definitely something
290. 9:39you want for these longer running
291. 9:40sessions that run for hours.
292. 9:43So in this case, this automatically gets
293. 9:45rid of the end file issue because it's
294. 9:47not in Git, and neither are my SSH keys
295. 9:49or our GitHub token.
296. 9:51And if it still needs an API key, you
297. 9:54can add that as a network secret where
298. 9:55it will it will tell you like, hey, I'm
299. 9:57missing an API key mid-session,
300. 10:00so it can use it without actually ever
301. 10:01sending it to the model. And by default,
302. 10:04also in the Cloud session, network
303. 10:06access is limited to things like package
304. 10:08registries in GitHub. So you have to
305. 10:10explicitly add those domains to your
306. 10:12environment for Cloud to to reach other
307. 10:14hosts.
308. 10:16Okay, so with a Cloud session we can run
309. 10:18these long-running sessions even with
310. 10:20our laptop closed and it keeps us safe
311. 10:23because it runs in its own environment.
312. 10:26Now it's still good to remember that
313. 10:28this machine is disposable, but the
314. 10:29things that leave it aren't. So a commit
315. 10:32pushed or a message it sent through a
316. 10:34connector. So we still run it in auto
317. 10:36mode with our deny rules just commit it
318. 10:38into the repo so they come along. Uh so
319. 10:41it it could still catch things like a
320. 10:42forced push or a secret in a commit, all
321. 10:44that kind of stuff.
322. 10:47And beyond just safety um we also need
323. 10:50to make sure that Cloud is actually
324. 10:51working on the right thing. So next is
325. 10:54verification. And the nice thing about
326. 10:56being in a loop uh like we kind of knew
327. 10:58the work was good because we'd already
328. 11:00kind of seen every change, we'd approved
329. 11:02everything.
330. 11:03But now if Cloud works for hours in the
331. 11:05background, we don't have that anymore.
332. 11:07Um and we're not really going to read
333. 11:09every file at the end either.
334. 11:12So you know, say that we ask you to move
335. 11:14our auth to a new API. Well Cloud is
336. 11:17very good at understanding what we want,
337. 11:19but there are still a lot of small
338. 11:20decisions that it has to make along the
339. 11:22way. So with the exact same prompt, you
340. 11:25know, one run might move it over but
341. 11:26leave the old API in there or one might
342. 11:29move it but only for new logins and
343. 11:31another one might move it but log
344. 11:33everyone out. And these all look pretty
345. 11:35reasonable, but it really helps to set
346. 11:38up these verification points along the
347. 11:40way. So Cloud keeps verifying its own
348. 11:42results. And those can be anything that
349. 11:44comes back as a pass or fail. So tests,
350. 11:47a build, a screenshot, whatever you
351. 11:48have.
352. 11:49So now we just want to make sure that we
353. 11:51review these verification points. Uh
354. 11:54there are a lot shorter to review than
355. 11:55if we just had to you know, review the
356. 11:57whole change at the end. So so that's
357. 11:58pretty reasonable.
358. 12:00So if one of these runs would still log
359. 12:01everyone out, it doesn't get past that
360. 12:03first check and then it just has to find
361. 12:05another way. And there's a couple ways
362. 12:07to set this up. Um the easiest one is of
363. 12:09course just in your prompt. So you know,
364. 12:11you say what done looks like and ask it
365. 12:13to split that into tasks so they have
366. 12:15their own tests. Um, and I also like to
367. 12:17ask it to show the actual output because
368. 12:19you weren't there to see it. But again,
369. 12:21prompting is not a guarantee that a
370. 12:23model will do it. So for the ones that
371. 12:25have to run
372. 12:27guaranteed run, I would use hooks, um,
373. 12:29which are just, you know, scripts that
374. 12:31Cloud Code runs at a specific point.
375. 12:33And
376. 12:34those give you a lot of control because
377. 12:36you decide when they run and what they
378. 12:38run and what they check.
379. 12:39So for example, every time it edits a
380. 12:41file, we can run our linter. Every time
381. 12:43it wants to mark a task as complete, we
382. 12:45can run the tests for what it changed.
383. 12:48Um, so you know, if it touched our auth
384. 12:49code or something, it has to pass the
385. 12:51auth auth tests.
386. 12:53And when it wants to finish the whole
387. 12:54session, we can just run the entire test
388. 12:55suite.
389. 12:57And if one of those fails, it goes, oh,
390. 12:59that's still broken. It goes back until
391. 13:01it passes, so it fixes it. Now, just one
392. 13:04thing,
393. 13:05um, when a test fails, by default, most
394. 13:07tools that exit with a one exit code,
395. 13:10um, but with a one, that task by Cloud
396. 13:12Code just gets marked as completed
397. 13:14anyway. So make sure that your scripts
398. 13:17exit with a two exit code because that's
399. 13:19what actually sends Cloud back to fix
400. 13:21it.
401. 13:22So
402. 13:24okay, now we have one session that I'm
403. 13:25pretty comfortable with leaving alone.
404. 13:28So now we can start with a few more of
405. 13:30those because they've all got their own
406. 13:32machines, so they're, you know, not all
407. 13:33working in the same folder and we don't
408. 13:35really have to think about work trees
409. 13:37too much ourselves anymore.
410. 13:39But then we still start every session
411. 13:41ourselves and we review every one of
412. 13:43them. And honestly, this is already a
413. 13:45pretty good place to be. We are still
414. 13:47fully in control over what gets started
415. 13:50and when we call a task done.
416. 13:53But even there, um, you know, we're
417. 13:55still the ones kicking off every session
418. 13:57and we're still in the loop for all of
419. 13:59them. So I think the most important
420. 14:01thing to have in place here now is your
421. 14:02CI because
422. 14:04all of these sessions eventually open a
423. 14:06PR and your CI runs the same checks on
424. 14:08every one of them, no matter how that
425. 14:10session was set up or, you know, what
426. 14:11Cloud decided to do along the way. And
427. 14:13And you turn on auto fix, we have like
428. 14:15CI auto-fix,
429. 14:16Claude can also see when one of those
430. 14:18checks fails and then go back to fix it.
431. 14:20So you're not the one spending your
432. 14:22review on things that, you know, a check
433. 14:24could have already caught.
434. 14:27And for some of them, you can even turn
435. 14:28on merge when ready. So then GitHub
436. 14:30merges the PR once all of the checks
437. 14:32have passed. So you're not in the loop
438. 14:34to review. Um but again, that that's not
439. 14:37the default and that really depends on
440. 14:38how risky the change is. So for example,
441. 14:40on our team, every PR gets reviewed by
442. 14:43Claude first. So not by the session that
443. 14:45wrote it. So it comes with a fresh
444. 14:48context and also a separate group of
445. 14:50agents to just double check every bug it
446. 14:52finds.
447. 14:53And then how much we review ourselves
448. 14:55just depends on the risk. So smaller
449. 14:57changes like docs or copy, they don't
450. 14:59need a person. So once the automated
451. 15:01checks are green, we're pretty good to
452. 15:03go. Um maybe ordinary product changes
453. 15:05just need one reviewer and then risky
454. 15:07changes might need two.
455. 15:10Now ultimately, kind of where we see
456. 15:12things going um and we're already kind
457. 15:14of doing more of that at Anthropic is
458. 15:16that you're not really starting all of
459. 15:18those sessions yourself either. Because
460. 15:19if you're anything like me, I'm terrible
461. 15:21at context switching. So having to
462. 15:23manage so many sessions at the at the
463. 15:25same time doesn't always go well.
464. 15:27So instead, you just have one main
465. 15:29session that's your orchestrator and you
466. 15:31can already see this in things like
467. 15:33Claude Code projects, which is still in
468. 15:35beta, but
469. 15:37you just tell it what it needs to be
470. 15:39done and the orchestrator splits that up
471. 15:41and then starts other sessions for each
472. 15:43task and it keeps track of what comes
473. 15:45back. So it automatically starts and
474. 15:47verifies these threads which are just
475. 15:49new sessions. So instead of talking to
476. 15:52eight sessions, you're just talking to
477. 15:53one.
478. 15:55And the main session doesn't write the
479. 15:56code itself. So there you want a model
480. 15:58that's really good at deciding what to
481. 16:00do and judging what comes back, a model
482. 16:02like Fable. And if something comes back
483. 16:05that isn't actually done, it just, you
484. 16:06know, sends it back so until it's
485. 16:08actually done and passes everything.
486. 16:10And the threads need a model that's good
487. 16:12at actually doing the work like Opus 55.
488. 16:14And you can set those separately. So,
489. 16:17you know, one model for the main session
490. 16:18and then another one for the threads.
491. 16:20And another benefit is that with this
492. 16:22setup with CloakCode projects
493. 16:24specifically, we don't manage context
494. 16:26anymore. That's all done automatically
495. 16:28for us.
496. 16:29no longer have to think about, you know,
497. 16:31when to compact or clear all of that
498. 16:33stuff.
499. 16:34Okay, so if you look back at those old
500. 16:36best practices, they're all still there,
501. 16:38but they've just moved.
502. 16:40So, the model steers it now, um auto
503. 16:42mode approves your deny rules and the
504. 16:44environment restricts what's possible,
505. 16:46um your tests, your hooks, and your CI
506. 16:48checks it. And then ultimately, Cloud
507. 16:50can just start them as well. So, almost
508. 16:52everything we used to do ourselves is
509. 16:54now either the model, the harness, or
510. 16:57the environment. And we're really just
511. 16:59here to start that first one and to just
512. 17:00review what comes out.
513. 17:02So, if you want to try this in your own
514. 17:04setup, I would say just take one thing
515. 17:06that's way bigger than you normally give
516. 17:07Cloud. Make sure your CI actually
517. 17:10catches the things you care about, and
518. 17:12then spend a bit of time just
519. 17:13configuring your environment, like its
520. 17:14network access, its secret, and its, you
521. 17:16know, setup script.
522. 17:18And then start in the cloud and see
523. 17:20where your sessions get stuck and where
524. 17:21you're still in the loop because that's
525. 17:23the next thing to hand over to Cloud. I
526. 17:26also highly recommend checking out our
527. 17:27best practices in the docs as well.
528. 17:30Thank you so much.