New Skills! v1.3 brings /pr, /implement-spec, and /retro — Transcript

Full transcript

1. 0:00Hello folks, version 1.3 of my skills
2. 0:03repo is out and it brings three very
3. 0:06important skills that have become
4. 0:07central to my workflow. Today I'm going
5. 0:09to show you about implement spec. I'm
6. 0:11going to show you about PR and I'm going
7. 0:13to show you about retro. And retro I
8. 0:16think is the one that I've just seen
9. 0:19absolutely go nuts on socials. People
10. 0:21going crazy for this skill and it has
11. 0:24felt incredibly powerful to me too. And
12. 0:26the first place I want to start is the
13. 0:28implement spec skill. The main flow of
14. 0:30my approach for landing huge chunks of
15. 0:33work at once is essentially two
16. 0:35different entities. You have a spec and
17. 0:38then you have the tickets that implement
18. 0:40that spec. The spec kind of acts like
19. 0:42the place where you're going and then
20. 0:44the tickets break that destination down
21. 0:47into a set of sessions that you can then
22. 0:50run in a single coding agent. If you try
23. 0:52to implement the spec all in one coding
24. 0:54agent, you would go into the dumb zone
25. 0:56of that coding agent, you might end up
26. 0:58hitting auto compact. And you know,
27. 1:00while that works sometimes, and it is
28. 1:02getting better, I still think that
29. 1:04breaking things down into individual
30. 1:06tickets is just a lot cleaner. Now, the
31. 1:08way that you orchestrate those tickets,
32. 1:09the way that you give them each to one
33. 1:12coding agent has always been a little
34. 1:13bit confusing for users. And I've kind
35. 1:16of been sort of going back and forth
36. 1:18over the way that I recommend you do
37. 1:20this. The most annoying and manual way
38. 1:23to do this is managing it yourself where
39. 1:25you the user say okay uh agent we are
40. 1:28going to implement our first ticket. You
41. 1:30wait for it to be complete. You clear
42. 1:32the context and then you say okay
43. 1:34implement ticket two. You wait. You
44. 1:36clear the context. You say implement
45. 1:38ticket three. You're kind of acting like
46. 1:40a for loop there. It's not really
47. 1:42workable. So that is what I would
48. 1:44describe as the manual loop there which
49. 1:46is not much fun. But you can change that
50. 1:49by having a deterministic loop where you
51. 1:51have a script that reach uh reads each
52. 1:53ticket and then runs implements on it
53. 1:55itself. This is how I recommend that
54. 1:57most people do this because once you've
55. 1:58set up your deterministic loop, it's
56. 2:00going to run the same way every time.
57. 2:02It's very reliable and it's cheap,
58. 2:05right? Because you're not using an agent
59. 2:06or you're not using a human to run that
60. 2:09loop. You just essentially get a script
61. 2:11to do it for you. But setting this up is
62. 2:13quite complicated and requires a lot of
63. 2:16patience as you sort of refine your loop
64. 2:18and get it better and better. And so I
65. 2:20see this as out of range for beginners.
66. 2:22And so I wanted some way of allowing
67. 2:26beginners to see not have to manually
68. 2:29loop it themselves. And so I decided,
69. 2:31okay, why don't we try a middle ground
70. 2:34where we have sub agents go through and
71. 2:37implement the loop. In other words, an
72. 2:38agent does the babysitting instead of a
73. 2:41human. This means that all of the
74. 2:42tickets will be implemented by sub
75. 2:44agents. And this has recently gotten
76. 2:47possible because sub aents can
77. 2:49themselves spawn sub aents. So before
78. 2:52sub agents were kind of a little bit
79. 2:54nerfed whereas now they're actually as
80. 2:56powerful as orchestrator agents. And so
81. 2:58this is what implement spec does is it
82. 3:01the agent sits there looking at all of
83. 3:03the tickets orchestrating them and then
84. 3:05implements them in sub aents. This is I
85. 3:08would say worse than doing a
86. 3:10deterministic loop because the
87. 3:11deterministic loop again it's
88. 3:12deterministic. It works the same way
89. 3:14every time. But implement spec is a good
90. 3:16way to get started with AFK workflows
91. 3:19away from keyboard workflows where you
92. 3:21step away and it implements the entire
93. 3:23spec. Before you use any skill, you
94. 3:25should read the entire skill. And it's
95. 3:27not particularly long here. There are
96. 3:30nine steps in its process and it's got a
97. 3:32little bit of reference material just at
98. 3:33the top. One crucial thing is it goes
99. 3:35for parallelization where it can. And so
100. 3:37this says the tickets are not a list of
101. 3:39steps. They're a task graph with
102. 3:41blocking relationships between them.
103. 3:43That means there's always a frontier of
104. 3:44tickets that are ready to be grabbed. If
105. 3:46we take a look at the steps, we can see
106. 3:47it reads the spec and tickets. It does a
107. 3:50little bit of exploration in its own sub
108. 3:52agent. It creates an integration branch
109. 3:55and then it goes and uses implement sub
110. 3:57aents to implement each ticket using TDD
111. 4:00and using work trees too. Once that's
112. 4:02done, it merges its work to the
113. 4:04integration branch with a sub aent and
114. 4:07then pings off more implement sub aent
115. 4:09until all the work is done. And once it
116. 4:12the work is done, it goes and calls code
117. 4:14review on the integration branch. Then
118. 4:16it cleans up all of its work and makes
119. 4:18the PR ready for review. So you get a
120. 4:20single PR off this huge great big spec.
121. 4:23I have been reaching for this a lot more
122. 4:25than I thought I would, especially on
123. 4:28systems where you haven't set up these
124. 4:29deterministic scripts yet, where you
125. 4:31don't have your like software factory
126. 4:33dialed in. I suppose implement spec has
127. 4:36been really really useful. It's been in
128. 4:37in progress for a while now, but I
129. 4:39thought now's the moment. Let's graduate
130. 4:41this so people can use it and fiddle
131. 4:43around with it and build their own
132. 4:44version of this, too. This is a basic
133. 4:46version of how you might implement your
134. 4:48own spec. You could do something totally
135. 4:50different to this. You could use stacked
136. 4:52PRs. You could do all sorts of stuff.
137. 4:54And in fact, talking of PRs, one of uh
138. 4:56my most used recent skills has to be the
139. 4:59PR skill. So, PR skill.md. I've been a
140. 5:02huge fan of Dex Hies for a while now. I
141. 5:04recently got to meet up with him in
142. 5:06person and he has this gorgeous skill
143. 5:08called Show Me. What show me does is it
144. 5:10helps the user understand the current
145. 5:12topic visually with concise diagrams and
146. 5:15it uses all of this like uh really nice
147. 5:18pseudo code. This looks a little bit
148. 5:20garbage, but actually when you see it in
149. 5:22the context of a PR, it really makes the
150. 5:25PR a lot easier to read and review. PRs
151. 5:28are still the main bottleneck for work
152. 5:30getting to Maine. And my goal with this
153. 5:33PR skill, kind of inspired by Show Me,
154. 5:37is to make human review as easy as
155. 5:40possible. And in the skill, we provide
156. 5:41essentially a template for writing a PR
157. 5:44body. That's all the skill does. We
158. 5:46first start with a summary based on the
159. 5:48kind of show me template which we
160. 5:50include in the skill and then we ask for
161. 5:53evidence. So before and after. This
162. 5:55evidence is an essential part of
163. 5:57learning to trust your agent, learning
164. 6:00to trust your agents outputs. and
165. 6:01actually prompting it to provide
166. 6:03evidence often means it runs an extra
167. 6:05test or it takes an extra screenshot if
168. 6:07it has that ability and it provides you
169. 6:10runtime information on that it's actual
170. 6:14change is doing what it thinks it's
171. 6:15doing without asking for hard evidence
172. 6:17it's very easy for agents to say yeah
173. 6:19that probably works cuz I've read the
174. 6:20code but no verification is something
175. 6:22I'm beginning to be really obsessed with
176. 6:25especially after my recent interview
177. 6:26with Lauren and then the third heading
178. 6:28here is merge danger so how dangerous Is
179. 6:31this PR going to be to merge? One
180. 6:33metaphor I've really like here is, is
181. 6:35this a one-way door or is this a two-way
182. 6:38door? A two-way door is a door that you
183. 6:40can walk back through. In other words,
184. 6:41we can easily revert this PR. But a
185. 6:44one-way door is a, you know, a PR that's
186. 6:46going to do something out in the world.
187. 6:49It's going to delete some data or it's
188. 6:51going to be expensive in some way to
189. 6:53roll back. And then finally, you have
190. 6:54blast radius here. So the potential
191. 6:56ramifications of the changes. So, if you
192. 6:59have a very small blast radius on a
193. 7:01two-way door, then, you know, you really
194. 7:03don't need to like review that PR very
195. 7:05hard. And so, having this at the bottom,
196. 7:07probably the, you know, first or second
197. 7:09thing the person is going to see is
198. 7:11essential. So, for instance, here's one
199. 7:13of the PRs in my repo that has been
200. 7:15provided by the PR skill. This is
201. 7:16essentially merging two algorithms that
202. 7:19were basically the same. And we can see
203. 7:21the files deleted and the files added in
204. 7:23the right context. We understand the
205. 7:25functions, the sort of the call stack
206. 7:27that's being structured here, but
207. 7:28crucially, we have the evidence. We have
208. 7:30a before and after on a real entity. So,
209. 7:33there's no intro in this one, but
210. 7:35afterwards, it adds the intro, which is
211. 7:37kind of the behavior that we want here.
212. 7:39And finally, we can see that the merge
213. 7:40danger is two-way. It's just a pure
214. 7:42refactor plus one deliberate behavior
215. 7:44change, and the blast radius is small.
216. 7:46So, this PR skill is just become
217. 7:48absolutely indispensable to my flow. It
218. 7:50is also one of the most consistently
219. 7:52invoked model invoke skills that I've
220. 7:54seen. The you you the user don't have to
221. 7:57worry about it at all. It seems to just
222. 7:59invoke it every single time, at least on
223. 8:01Opus 5.5. So you should just be able to
224. 8:03download it and just see your PR bodies
225. 8:05get better. And of course, if you
226. 8:07already have a PR body skill, then why
227. 8:09not just steal what you can from this?
228. 8:11Now, before we talk about the retro
229. 8:12skill, we should mention that there has
230. 8:14been a big change in how my especially
231. 8:17grill with docs function. So let's take
232. 8:19a look at domain modeling here and
233. 8:21domain modeling skill now instead of
234. 8:25writing to context.md writes to
235. 8:27glossery.mmd. The inspiration for this
236. 8:29skill came from DDD came from Eric
237. 8:32Evans's book domain driven design. And
238. 8:35[snorts] for that I was essentially
239. 8:37using a file name based on the concept
240. 8:40of bounded context from DDD. And so this
241. 8:43used to be context.md like this. But now
242. 8:47I've sort of moved away from the DDD
243. 8:50style flow. I mean, I still use DDD in
244. 8:53all but name, but I wanted a slightly
245. 8:56less overloaded term here. Context.md
246. 8:59just felt way too vague. It didn't sort
247. 9:02of trigger the agent to pull it in at
248. 9:03the right moment. And for users, it was
249. 9:06pretty confusing. I also ended up
250. 9:07reducing the amount of stuff in glossery
251. 9:10or in context.md until it was literally
252. 9:12just a glossery. So there really is no
253. 9:14point anymore. glossery.mmd is now what
254. 9:16we're going this is also updated
255. 9:17everywhere and so loads of my skills
256. 9:20rely on seeing a glossery.mmd in order
257. 9:23to behave properly in order to use the
258. 9:24domain language at all points and so you
259. 9:27will need to definitely update this.
260. 9:29There's no other behavior changes about
261. 9:30the skill. It's really literally just a
262. 9:32file name change and I think that this
263. 9:35is one that folks have been wanting for
264. 9:36a while. So I'm glad to ship it. But now
265. 9:38let's take a look at the retro skill and
266. 9:40let's actually do it by running it. What
267. 9:43I noticed with my skill set is that I
268. 9:46impose a lot I ask a lot of you the
269. 9:49user. And what I was kind of asking of
270. 9:52you is you were responsible for overtime
271. 9:58making changes to the repo that would
272. 9:59improve it. In other words, I was asking
273. 10:01you to manage your agents.mmd to manage
274. 10:04your skills, to make sure that your repo
275. 10:06was token efficient, to make sure that
276. 10:08your environment was good and safe for
277. 10:10the agent to write code in, that you had
278. 10:12good lint rules, that you had good
279. 10:13navigation. And I realized that I
280. 10:15personally was doing a lot of stuff to
281. 10:17my repos to my agents in order to make
282. 10:20them easier for me to work with to make
283. 10:22them more performant. And so I realized,
284. 10:24why don't I bundle that up into a skill?
285. 10:27And this is what retro does. It is short
286. 10:29for retrospective. You run it on a
287. 10:32coding agent session, either the current
288. 10:34one or, you know, 10 that I've currently
289. 10:37got or let's say the previous one, and
290. 10:39it will suggest improvements for next
291. 10:41time. The cool thing about actually
292. 10:42reading real sessions here is that it's
293. 10:45able to look at what happened with the
294. 10:48agent. It's able to see the
295. 10:50inefficiencies that maybe the agent
296. 10:51actually hides from. I really like this
297. 10:53post from Andre Retro by Mogers. Game
298. 10:55changer. My agents are bumping into all
299. 10:57kinds of problems. that went undetected.
300. 11:00But due to persistence, the features
301. 11:02still got built one way or another. In
302. 11:03other words, the agent doesn't complain,
303. 11:05I think, as much as it should, and
304. 11:08doesn't try to fix its own mistakes. You
305. 11:11kind of need to get another agent to
306. 11:13look back on those sessions and see
307. 11:14where it went wrong and see how to help.
308. 11:16And so, this is what our agent has found
309. 11:19here. The agent ran an irreversible
310. 11:20public action before you picked a fix.
311. 11:23The agent found why the 1.3.0 release
312. 11:25was missing and then it ran a release
313. 11:27without actually asking. This is why, by
314. 11:29the way, when you actually look at the
315. 11:30releases, you'll see there's no 1.3.0
316. 11:33release, only a 1.3.1 release. You know,
317. 11:35Claude kind of screwed this up. I don't
318. 11:36honestly care that much about it, so I
319. 11:39don't mind this one in my context. We
320. 11:41can see that this particular repo in has
321. 11:43no actual guard rail. So, there is a
322. 11:45PNPM check script, but nothing actually
323. 11:47runs it. So, it's suggesting adding a CI
324. 11:50pipeline here. In here, we can see
325. 11:52there's context loss happening between
326. 11:54compactions in long sessions. So, I use
327. 11:57animatics to kind of create my videos or
328. 12:00early drafts of them now. And it's
329. 12:02saying that there's some loss when you
330. 12:04compact. And so, it's saying move some
331. 12:06of the repeated instructions into an
332. 12:08animatic skill. This is definitely a
333. 12:10really good one, and I'm definitely
334. 12:12going to implement that for sure. Oh,
335. 12:13look at this one. A custom CLI that I've
336. 12:15created for my course video manager
337. 12:17wastess tokens. So, Retro looks at token
338. 12:20inefficiency too. Wiki, which is another
339. 12:22custom CLI, is not on path locally and
340. 12:24Wiki AI hero has gaps. So, Retro is
341. 12:27merciless. It will look at your actual
342. 12:30sessions and see what is going on and it
343. 12:32will pretty much always find ways to
344. 12:34improve. Now, what I'd say with Retro is
345. 12:36you want to use your human judgment when
346. 12:39you look at these. I know, for instance,
347. 12:41the thing that it thinks is most serious
348. 12:43is actually not that bad. It's just a
349. 12:46release. I can always create and cut new
350. 12:48releases. And so the way retro works is
351. 12:50it's designed to be human in the loop.
352. 12:52It's not designed to just keep creating
353. 12:54these fixes random. Pretty much everyone
354. 12:56when I show them this, they go, "Ooh,
355. 12:58I'd love to automate this." No, you
356. 12:59don't want to automate this. Automating
357. 13:02this means that the agent will get
358. 13:03itself into a loop where it continually
359. 13:06finds false positives. It continually
360. 13:08tries to fix things and you'll end up in
361. 13:10this weird trajectory where it just
362. 13:12takes the repo and your agent into a
363. 13:14place it shouldn't really go. So, I
364. 13:16recommend you run retro on a sampling of
365. 13:19your agent sessions. Every time you've
366. 13:21got a free moment and you realize, "Oh,
367. 13:23I haven't run retro in a while." Just
368. 13:25try getting it to run through your most
369. 13:27recent sessions. Maybe a session that
370. 13:29has gone wrong, let's say, or you feel
371. 13:31the agent has done something weird, and
372. 13:33retro will usually find a fix. We can
373. 13:35see that inside the steps in the retro
374. 13:37skill, it has several categories for
375. 13:39improvements that it looks for. How easy
376. 13:41is your codebase to navigate? Are there
377. 13:44automated checks that we could add?
378. 13:46Could we add coding standards for the
379. 13:47automated reviewer to enforce? How
380. 13:50healthy is the global agents.mmd? Is the
381. 13:52tool economy any good? Are there any
382. 13:54noops in the instructions in steering
383. 13:56files? Does the agent have access to all
384. 13:58the information it needs? Overall, I've
385. 14:00seen my token efficiency and just the
386. 14:02quality of work that I've been able to
387. 14:04produce massively increase by running
388. 14:06retro regularly. And if you want to see
389. 14:08the quality of your work increase too,
390. 14:10then I recommend you take my AI coding
391. 14:12crash course which is on aihero.dev. I'm
392. 14:14updating this today with a new lesson
393. 14:16focusing on implement spec. And if you
394. 14:17want to take control of your coding
395. 14:19agent and utilize my skills better, then
396. 14:21this is the place. But folks, thank you
397. 14:23so much for following along. Hopefully
398. 14:24you enjoy Skills V1.3. Enjoy seeing the
399. 14:28uncomfortable truth of how bad your
400. 14:30agent sessions are with retro.