TOP 1% Agentic Engineering: Claude Code Verifiers — Transcript

Full transcript

1. 0:00Okay, now you may have seen this graph
2. 0:01recently of the number of changes that
3. 0:03Enthropic are merging in every single
4. 0:05day and it's become increasingly clear
5. 0:06that almost nobody at Anthropic or
6. 0:09OpenAI are reading every single line of
7. 0:11code anymore. So, it kind of begs a
8. 0:13question, what exactly are the engineers
9. 0:15doing all day? Now, they've kind of been
10. 0:16hinting at this in different blog posts,
11. 0:18podcast episodes, and tweets, but I
12. 0:20think this interview is one of the
13. 0:21clearest examples that I found, and it
14. 0:23shows that what they care about the most
15. 0:24is the plan and the verification instead
16. 0:27of reading every single line of code.
17. 0:29Now, in this video, I want to talk about
18. 0:30what exactly that looks like, but I will
19. 0:32be going to many of these ideas in much
20. 0:34more detail inside of my upcoming cohort
21. 0:36that will be linked down below. So,
22. 0:37inside of this podcast episode, they
23. 0:39talked about how they rewrote the
24. 0:40messages API for Claude from Python to
25. 0:43Rust. And apparently, when they started,
26. 0:45nobody on the team was even a Rust
27. 0:47developer. Now, Claude wrote the code,
28. 0:49but before they had Claude write any of
29. 0:50it, they basically stopped and asked
30. 0:52themselves the question, okay, what
31. 0:54exactly is a verifier here? How will we
32. 0:56know that the new version is correct?
33. 0:58and the answer was shadow traffic. So
34. 1:00the new version in Rust and the old API
35. 1:03in Python would get the exact same real
36. 1:05traffic. And if the new version is
37. 1:07correct, then it should give the exact
38. 1:09same answers as the old one. And I
39. 1:11imagine what they did is that anytime
40. 1:12the Rust version gave a different
41. 1:14answer, that diff went back into Claude.
42. 1:17Claude worked out what was wrong, fixed
43. 1:19it, and then they ran the traffic again.
44. 1:20And once the two matched, they slowly
45. 1:22moved the real traffic over. Now, here's
46. 1:24how they put it in the podcast episode.
47. 1:26The two things that mattered the most
48. 1:27were the plan to do the thing and then
49. 1:29the verification of the thing because
50. 1:31actual implementation here was Claude
51. 1:33writing all the code. So Claude wrote
52. 1:35the code whilst humans used cloud to
53. 1:38write the plan, design the system and
54. 1:40make sure that the verifier was correct.
55. 1:42And basically they did this because of
56. 1:43the compute constraint. So they managed
57. 1:45to drop the CPU usage on the API by 40
58. 1:48to 50%. Now, this is the same mental
59. 1:50shift that I've been teaching many
60. 1:51engineers at pretty big companies over
61. 1:53the last few months when doing
62. 1:54consulting and training workshops
63. 1:56whereby instead of people thinking, how
64. 1:58do I get my agents to write better
65. 1:59tests? They think, okay, what is a
66. 2:01verify for this when I'm about to do a
67. 2:03big refactor or implement a big feature,
68. 2:05how will I know that's correct? What
69. 2:08verification do I need here? So, the
70. 2:09agent can hill climb towards the right
71. 2:11answer. And I think that this is
72. 2:12important because nowadays when you get
73. 2:13agents to write tests, especially unit
74. 2:16and integration tests, they're basically
75. 2:17restating what the code already does. So
76. 2:19all the tests can be green, but when you
77. 2:21open up the page, the button still does
78. 2:23nothing. And that's where a lot of
79. 2:24engineers are kind of facing a losing
80. 2:26battle where they keep adding more rules
81. 2:28to their cloud MD files saying like
82. 2:29always write proper tests, never weaken
83. 2:31an assertion, hoping that the agent will
84. 2:33suddenly write better tests. But if we
85. 2:35look at what the engineers at Entropic
86. 2:37are doing, they're not talking about
87. 2:38getting the agents to write better
88. 2:39tests. They're talking about creating
89. 2:41verification systems, tooling, and
90. 2:43environments. So, I kind of want to zoom
91. 2:44out here and make sure that we're all on
92. 2:45the same page by kind of defining what a
93. 2:48verifier is. So, it's basically a check
94. 2:50that your agent can't argue against. So,
95. 2:53every good verifier has three parts. It
96. 2:55has a surface, and this is the same
97. 2:57surface that a real user or another
98. 2:59system would be meeting the change that
99. 3:01your coding agent is making. So, this
100. 3:02could be a web page, a API, AMCP server,
101. 3:05a CLI, mobile app or desktop
102. 3:07application. And if you're making, I
103. 3:09don't know, some kind of voice agent,
104. 3:10then it could even be over the phone as
105. 3:12well. Now, next up, we have what I'm
106. 3:14calling the answer key. And that is what
107. 3:16writes looks like for your system. Now,
108. 3:18the most important thing here is that
109. 3:19the agent didn't write it. You kind of
110. 3:21define it up front with the help of your
111. 3:23agent. And finally, we have the
112. 3:25comparison itself. So, does what
113. 3:27happened match the answer key or not?
114. 3:29And that's kind of the big difference
115. 3:30from getting your agents to write tests.
116. 3:32So, when your agent writes a test, it
117. 3:33writes the answer key, too. So, of
118. 3:35course, it will end up passing. But you
119. 3:37got to kind of define the answer key up
120. 3:38front. Now, how good a verifier is comes
121. 3:40down to its answer key. And I find that
122. 3:42the best answer key is an old system
123. 3:44that already exists and does the same
124. 3:46job. So in the case of Entropic's
125. 3:48earlier example of the messages API, the
126. 3:50answer key was the old Python system.
127. 3:53Another answer key that we could have is
128. 3:54a snapshot that I will go into later. It
129. 3:56could also be a independent test suite
130. 3:59that doesn't depend on the code itself.
131. 4:00So this could be something like
132. 4:01playright end to end tests. And at the
133. 4:03bottom, we have a spec that could also
134. 4:05be an answer key. Now, in my own specs,
135. 4:08my answer key looks something kind of
136. 4:09like a table at the very bottom of the
137. 4:11spec that defines what a user should be
138. 4:13able to do and the expected result. So,
139. 4:15for example, if I had a really simple
140. 4:17note-taking application and I made a
141. 4:19spec to allow for tagging to be added,
142. 4:21then essentially I would have a table
143. 4:23kind of like a user should be able to
144. 4:25add a tag to a note and then the
145. 4:26expected result would be a tag shows on
146. 4:29the note at once. Another user action
147. 4:31would be a user can filter notes by tag
148. 4:34and the expected result would be only
149. 4:36notes with that tag show or for example
150. 4:38another user action would be deleting a
151. 4:40tag completely and then the expected
152. 4:42result would be that the tag goes from
153. 4:44every note and from the tag list. So I
154. 4:46could basically define like 10 or 15 of
155. 4:48these depending on how complicated of a
156. 4:50tagging system that I want and this
157. 4:52would form my answer key. Now, of
158. 4:53course, this one is slightly weaker
159. 4:55because we are depending on an agent to
160. 4:56make the final call. And ideally, you
161. 4:58should have another agent deciding
162. 5:00whether the expected result has been
163. 5:02met. And a pretty nice open-source
164. 5:04framework that I like for this is called
165. 5:06E2E. So, you can kind of see that you
166. 5:08can define end to end tests kind of like
167. 5:10this. So, for example, a member would
168. 5:12upgrade to pro and then we would open up
169. 5:14the billing page. We would then have an
170. 5:16agent using browser use basically act
171. 5:19and then upgrade the workspace to pro
172. 5:21and then we have another agent with a
173. 5:23different model maybe and this could be
174. 5:24a fast cheap system one model kind of
175. 5:26like Jeb deciding okay the invoice
176. 5:28preview shows a prorated amount and then
177. 5:30we can pass in some more determinism.
178. 5:32So, I really like this framework because
179. 5:33we can combine deterministic and agentic
180. 5:36testing together. And you can see from
181. 5:38these two lines over here, they agree
182. 5:40with the idea whereby the agent that is
183. 5:42deciding whether the expected result has
184. 5:43been met is different from the agent
185. 5:45that is actually taking out the actions.
186. 5:47Now, this is pretty cool because they
187. 5:48have a bunch of stuff like caching to
188. 5:50make sure you're only paying for tokens
189. 5:51when a test fails. And for some of my
190. 5:53projects, I basically deleted every
191. 5:55single test. So, you can see in this PR,
192. 5:57I deleted 110,000 lines. In this other
193. 6:00PR, I delayed 295,000 lines and then
194. 6:02replace them only with E2E tests
195. 6:04instead. And I generally found that with
196. 6:06this, I'm catching even more bugs per
197. 6:08PR. But essentially, what is happening
198. 6:10with me is that anytime the spec has
199. 6:12been implemented, any user behavior that
200. 6:14was defined at the bottom of the spec
201. 6:15gets turned into an E2 test instead to
202. 6:18basically make sure that user behavior
203. 6:20is conserved going forwards. And this is
204. 6:22basically acting as my verifier whereby
205. 6:24my answer key is taken from the spec and
206. 6:27then defined in the code kind of like
207. 6:29this. And finally, the worst situation
208. 6:31is basically having no answer key at all
209. 6:33because the agent will basically decide
210. 6:34for itself what exactly verify looks
211. 6:37like. And that could be basically having
212. 6:38green unit tests but still having broken
213. 6:41buttons. Now, I'll also be going into
214. 6:43this in much more detail inside my
215. 6:44upcoming cohort in the office hours and
216. 6:46the lessons that will be linked down
217. 6:48below. It will be starting in just over
218. 6:49a week. If you're interested and if you
219. 6:50sign up for the next few days, then
220. 6:52you'll benefit from the early bird
221. 6:54discount. So now, as we've seen so far,
222. 6:56some of our verifiers can be completely
223. 6:58deterministic and depend solely on
224. 7:00scripts, kind of like the API response
225. 7:02diff that we talked about at the very
226. 7:03beginning, and some of them can depend
227. 7:05on your agents judgment, such as
228. 7:07implementing a spec over here. And
229. 7:09finally, some of them can be a bit of
230. 7:10both. So a big split that I personally
231. 7:12have is that is this a refactor in which
232. 7:14case we can use a old system or a
233. 7:17snapshot as a verifier or is this a new
234. 7:19feature in which case we may want to use
235. 7:21a aentic end to end test. So nowadays
236. 7:23just like many of the entropic and open
237. 7:25AAI engineers building and maintaining
238. 7:27good verifiers is where most of my day
239. 7:29goes these days and it basically comes
240. 7:31down to free jobs. So we have building
241. 7:33good verifiers, maintaining them so they
242. 7:35don't go stale and then and then also
243. 7:37auditing them to make sure that they're
244. 7:39actually catching problems. Now I did
245. 7:41package all of this up into some free
246. 7:43skills that I have for cloud coding
247. 7:44codeex that will be linked down below.
248. 7:46So I have create prove it then I have
249. 7:48maintain prove it and then finally audit
250. 7:50prove it. So in the case of create prove
251. 7:52it it's making another skill which will
252. 7:54be a prove it skill. So the order in
253. 7:56which it works is basically it maps all
254. 7:58the surfaces. It then checks whether you
255. 8:00can actually reach the surfaces. It then
256. 8:02finds any things that can be used as
257. 8:04answer keys and brainstorms with you.
258. 8:06And then it generates a skill and then
259. 8:08after you've actually gone ahead with
260. 8:09implementation, you can run it for real.
261. 8:11So I'll go for a really quick example.
262. 8:13So let's say that I want to add a MCP
263. 8:15server to my sandboxing service,
264. 8:16Impell.AI. By the way, this is one of
265. 8:18the cheapest sandboxing service
266. 8:20providers that you will find online. So
267. 8:21if you are spending hundreds or even
268. 8:23thousands of dollars in sandboxes every
269. 8:25single month, you may want to consider
270. 8:27moving to this. Essentially, if I want
271. 8:28to add a MCP server, I can start it by
272. 8:31installing the skill. So, I'm going to
273. 8:33quickly add the marketplace and then
274. 8:34copy over the over command to basically
275. 8:36go ahead and install the skill. Install
276. 8:38it for all users. And then I can do
277. 8:41/create pro it. And this will be my
278. 8:44skill that is always used for verifying
279. 8:46and proving any changes to the MCP
280. 8:49server that ends up happening. So, I can
281. 8:51say, okay, I want to make a brand new
282. 8:53MCP server for this repo. Can you
283. 8:55basically help me brainstorm like how
284. 8:57exactly we're going to like verify this,
285. 8:58prove it once it's completed and then
286. 9:00basically press enter. So what you're
287. 9:02basically doing is that you're
288. 9:03triggering the skill and then you're
289. 9:04making a prove it skill for any of these
290. 9:07surfaces in your application. But you
291. 9:09can also use it for any big refactors as
292. 9:11well. So if I said something like /crate
293. 9:13prove it and then said hey so I
294. 9:15basically want to migrate the backend
295. 9:17API from go to rust instead and then it
296. 9:20will basically go ahead and find ways in
297. 9:22which I can prove that it's done. Now
298. 9:23the MCP response came back and it
299. 9:25basically said the surface would be the
300. 9:27MCP server. So this is planned. Our
301. 9:30existing surfaces are the public API,
302. 9:32the dashboard UI, the stored state and
303. 9:35also the logs as well. And then it
304. 9:36basically gives me a bunch of ways that
305. 9:38we can go ahead and prove it. And it
306. 9:40shows me the method that we could use,
307. 9:42the answer key, and then also the cost
308. 9:44as well. And it's like, okay, let's
309. 9:45combine a bunch of these together. So
310. 9:47we'll combine 1 2 3 and six together.
311. 9:49And now I can turn this into a skill
312. 9:51that I can basically use once the MCP
313. 9:53server has been implemented to verify
314. 9:55that everything works. If something
315. 9:56failed for whatever reason, then that
316. 9:58would go back to the builder. The
317. 10:00builder agent would make changes and
318. 10:01then the verifier would try to reverify
319. 10:03the flow. And now afterwards, I would
320. 10:05have a skill which is kind of like prove
321. 10:08MCP and anytime a change happens to MCP7
322. 10:11in the future, then it will
323. 10:12automatically use a skill to basically
324. 10:14verify that the changes have been made
325. 10:16correctly. Likewise for the migration
326. 10:18from go to Rust if I were to do that it
327. 10:20basically shows me a bunch of methods
328. 10:21that I could take what I would catch
329. 10:23what the setup cost would be as well. So
330. 10:25for example for the read only shadow a
331. 10:27shadow API that has a pretty high setup
332. 10:30cost because we're basically running
333. 10:31continuously but it also has the best
334. 10:34performance in the sense it would be
335. 10:36using traffic shapes that no one thought
336. 10:38to actually go ahead and script. So in
337. 10:40the case of a big refactor, the first
338. 10:42thing you want to figure out is what is
339. 10:44your answer key? And in the case of the
340. 10:45infropic example, the answer key was the
341. 10:48old API. So you basically keep your old
342. 10:50system running and then copy any
343. 10:52requests over to a new system and then
344. 10:54compare every response and then pass any
345. 10:57diffs or any things that disagree back
346. 10:59over to Claude. It can find the recalls,
347. 11:01fix it, and then you continue until you
348. 11:03hit zero differences. And they kind of
349. 11:05kind of brainstormed this by using the
350. 11:07/create private skill that I made. Now
351. 11:08bun did something very similar when they
352. 11:10rewrote their runtime from a zig to rust
353. 11:12but the answer key was basically their
354. 11:14existing test suite because if you go to
355. 11:16the article that they showed it in then
356. 11:18scrolling down I can basically see over
357. 11:20here that they said fortunately bun's
358. 11:22own test suite is written in typescript
359. 11:24which means that it doesn't depend on
360. 11:26the runtime's programming language. So
361. 11:28in that case after they got claw to use
362. 11:30a whole bunch of sub aents in a dynamic
363. 11:32workflow to basically do the migration
364. 11:34they hill climbed to 100% of the test
365. 11:37passing again with zero test being
366. 11:39skipped or deleted. Now another
367. 11:40interesting verify that we have is bite
368. 11:43for bite equality. So, there's been an
369. 11:45interesting trend of people using agents
370. 11:47to do matching decompilations of old
371. 11:49games, which basically means that you
372. 11:51have the agent rebuild the source code
373. 11:53of the game from the binary and then you
374. 11:55check it to make sure it compiles back
375. 11:57into exact same machine code as the
376. 11:59original game. So, for example, over
377. 12:01here, someone had agents running for a
378. 12:03few hundred hours to basically do a
379. 12:05matching decompilation of Crimson Land
380. 12:08that they then posted on GitHub over
381. 12:09here. And I can actually go over to link
382. 12:11inside of my browser and then play it if
383. 12:12I wanted to. But they're basically using
384. 12:14the same idea of having verifiers as
385. 12:16well. So in this case, the answer key is
386. 12:18the original game. You compile the
387. 12:20agents code with the same odd compiler
388. 12:22and the comparison is a bytes. So do
389. 12:24they match exactly or not? And another
390. 12:26good example of bite for byite equality
391. 12:28is this one by Mitchell Hashimoto who
392. 12:30made Ghosty the terminal. And he
393. 12:32basically wanted Ghosty to show images
394. 12:34the same way that Kitty does. So Kitty,
395. 12:36which is a terminal that I like using,
396. 12:38has a way of showing images inside of a
397. 12:40terminal and many programs now use it.
398. 12:42and he basically wanted the same
399. 12:44programs to work in ghosty as well. So
400. 12:45in his case the answer key was basically
401. 12:47kitty. Now another related example that
402. 12:49you could consider is pixel diffs where
403. 12:51basically you can have a screenshot as
404. 12:53an answer key. So if I take this website
405. 12:55which is fingerprint.com I really like
406. 12:57the design of this website. I can
407. 12:59basically give it over to an agent and
408. 13:00then say use a builder verifier loop to
409. 13:03copy the design of the website using a
410. 13:05pixel diff as your verifier. Now I can
411. 13:08kind of see it working over here. So it
412. 13:09has original on the left hand side, my
413. 13:11version on the right hand side, and then
414. 13:12it shows me the pixel diff. And if I
415. 13:14would scroll down, then it basically
416. 13:16says that it went from having 19% diff
417. 13:19to 1.8% to 0.54% to 0.52% and then it
418. 13:23stopped. And then it says if I remove
419. 13:25any logos, then it would actually be
420. 13:270.07%.
421. 13:28Now, of course, this is more of a easy
422. 13:30to understand example, but some people
423. 13:32have been doing the same thing when it
424. 13:33comes to making any font files smaller
425. 13:36on their website to make their website
426. 13:37load faster. They basically use a pixel
427. 13:40diff as their verifier. Now everything
428. 13:41that I mentioned so far depends on there
429. 13:43being some kind of answer key. But often
430. 13:45when it comes to a brand new feature,
431. 13:47there is no old system that you can
432. 13:49depend on. So in this case, the answer
433. 13:50key becomes your own specification. So
434. 13:53when you're writing a spec with the help
435. 13:54of your agent, you kind of want to
436. 13:56define what is expected behavior for a
437. 13:58user. So going back to early example, we
438. 14:00want to make sure like okay, if we're
439. 14:02adding tags to notes, the tags actually
440. 14:04appear. And the main thing here is you
441. 14:06want to make sure that all the
442. 14:07behaviors, the potential paths that your
443. 14:09user could take for your application are
444. 14:11defined during the spec writing phase.
445. 14:13And as I said with the E2 library, after
446. 14:15the spec has been implemented, you can
447. 14:17then turn this into end to end test.
448. 14:19Now, to show a real example for a real
449. 14:21application here, I'm going to go ahead
450. 14:22and open up Slack. And despite most my
451. 14:24videos being inside of cloud code in the
452. 14:26terminal, like 97% of my coding happens
453. 14:29in cloud sandboxes via Slack instead.
454. 14:32Because as I do talk about inside my
455. 14:33cohorts as well, I think that Slack is a
456. 14:36best surface in which you should be
457. 14:37building with your agents, especially
458. 14:38with the brand new code channels
459. 14:40feature. So if I go over then I can
460. 14:42basically see all the code that had been
461. 14:44written by the agent. And then I can see
462. 14:46video recordings of everything that I
463. 14:48went ahead and verified. So if I go to
464. 14:50this recording over here, I can
465. 14:51basically see that's like verifying and
466. 14:53testing the Mac OS application to make
467. 14:55sure that this brand new like a mic
468. 14:57picker functionality that I added is
469. 14:59actually working. And I can see that in
470. 15:01this particular case, it basically
471. 15:02verified a whole bunch of user flows.
472. 15:04And then it said at the very bottom that
473. 15:06on a real Windows machine, it passed
474. 15:08four of the five flows. And then I think
475. 15:10it went back to the builder to basically
476. 15:12fix what it was broken. And it kind of
477. 15:13like went back and forth with this like
478. 15:15builder verifier loop to make sure that
479. 15:17everything had been implemented
480. 15:18correctly and is working as I defined
481. 15:20earlier on inside of my answer key. And
482. 15:22yeah, with a setup kind of like this, I
483. 15:24can deliver really reliable results. Now
484. 15:26the second job that we have is
485. 15:27maintaining our pro it skills because
486. 15:29they can become stale over time. Now to
487. 15:32go through a real example if I go to one
488. 15:34of my previous cohorts that I ran I have
489. 15:36like my office hours page over here
490. 15:38where the recording of the office hours
491. 15:40appears and the problem that some people
492. 15:41are facing was that video playback
493. 15:43sometimes broke on Safari. So I made a
494. 15:46change to the video player on the
495. 15:47website and then it worked fine on
496. 15:49Chrome but it was broken on Safari and
497. 15:51sometimes even Firefox. And the reason
498. 15:53why it went unnoticed was basically
499. 15:55because my proet skill was clicking
500. 15:57around the website and testing the
501. 15:59changes in playright chromium, but
502. 16:02because Safari and Chrome handle video
503. 16:04playback a little bit differently, the
504. 16:06change would have not actually been
505. 16:07verified on Safari. So when a user
506. 16:09reported to me like, hey, Safari
507. 16:11playback doesn't work that well. I
508. 16:13basically told my agent, can you spin up
509. 16:14Playrite WebKit, which is based on
510. 16:16Safari, and also use a real Mac that it
511. 16:18has access to to basically try and
512. 16:21identify the problem, fix those changes.
513. 16:23And after that was fixed, I wanted to
514. 16:25make sure that future changes don't
515. 16:27break Safari again. And that fix was
516. 16:29upstream inside of the private skill. So
517. 16:32I basically told the agent anything that
518. 16:33touches a video playback inside of the
519. 16:35private skill should also be checked
520. 16:37with playright webkit and also on a real
521. 16:39safari on a MacBook that it has access
522. 16:42to. So essentially zooming out here the
523. 16:44meta pattern is that even though your
524. 16:46pro skills and your verification skills
525. 16:48will catch many problems before they hit
526. 16:50production. There will be some problems
527. 16:52that users still come across and report
528. 16:54and start thinking okay how could the
529. 16:55agent have been prompted better? How
530. 16:57could I have written a better spec? It's
531. 16:59more of a case of how can you improve
532. 17:00your private skill upstream so your
533. 17:02agent can catch these changes before it
534. 17:04ever reaches production. And that's
535. 17:06basically what the maintain private
536. 17:07skill allows you to do. So after a
537. 17:09couple runs, you basically run it and
538. 17:10then it goes back to your previous runs.
539. 17:12It would also look through any user
540. 17:14feedback and any bugs that have been
541. 17:15reported and try to identify
542. 17:17improvements that can be made to the
543. 17:19private skill so that when a bug does
544. 17:21slip through the cracks and gets
545. 17:22reported by user or comes up in Sentry,
546. 17:25then a similar class of bugs and
547. 17:27problems can be fixed before they ever
548. 17:29reach users in the future. Now, the
549. 17:31third job is basically auditing your pro
550. 17:33skills on a regular basis to make sure
551. 17:34they're actually catching problems. And
552. 17:36I find that one of the best ways of
553. 17:38doing this is mutation testing, which is
554. 17:40where you break your app on purpose to
555. 17:42basically see if your checks actually
556. 17:43notice. So that is what the audit prove
557. 17:45it skill does. So it basically takes a
558. 17:47change that you made recently, plants a
559. 17:49realistic bug inside the code like
560. 17:52making the counter off by one for
561. 17:53example and then a fresh agent that is
562. 17:56basically told to verify the flow again
563. 17:58such as going back to a previous example
564. 18:00of adding tags to notes. It would see if
565. 18:02the fresh agent is able to identify that
566. 18:04problem without telling it what broke.
567. 18:06So if you're using a aentic verifier
568. 18:08instead of a deterministic verifier, if
569. 18:11it catches a bug, then that's really
570. 18:12great. If it doesn't, then it means that
571. 18:14your skill would need some kind of
572. 18:15fixing. So, I think going back to where
573. 18:17we started earlier in the video, so many
574. 18:19lines of code are being written and so
575. 18:21many changes are being merged that no
576. 18:22one is really reading every line of code
577. 18:24anymore. And depending on the industry
578. 18:26you're in, that can be fine as long as
579. 18:28something else is checking it. So,
580. 18:29ideally, that should be some kind of
581. 18:31verifier. And you want to kind of ask
582. 18:33your agent before you make a big change,
583. 18:35what exactly is a verifier here? How
584. 18:38would we know that this is correct once
585. 18:39it has been implemented? But
586. 18:40essentially, as we saw from the podcast
587. 18:42episode, a lot of the best engineers are
588. 18:44focusing on building verification
589. 18:46tooling around their agents. So with the
590. 18:48limited attention that they do have
591. 18:50every single day, they can basically
592. 18:51make sure that their agent has gone
593. 18:52through enough verification itself
594. 18:54before it finally shows a change back to
595. 18:56them. And often they're not even reading
596. 18:58the code of the change. They're watching
597. 19:00videos and looking at graphics and HTML
598. 19:02files. So yeah, all three on my previous
599. 19:04skills will be down below if you do want
600. 19:06to use them. And I will be making
601. 19:07updates to them over time. And if you do
602. 19:09want to hear more ideas from me on a
603. 19:10regular basis, then you may also want to
604. 19:12join my newsletter that will be linked
605. 19:14down below,