"Software Fundamentals Matter More Than Ever" — Matt Pocock — Transcript

Full transcript

1. 0:07[music]
2. 0:14>> Hello everyone. Having a good conference
3. 0:16so far? Yeah. Are you having a good
4. 0:18conference so far? Yeah. Good.
5. 0:20Wonderful.
6. 0:22I have a message for you that I hope
7. 0:24will be um a comforting message for
8. 0:27folks who believe that uh
9. 0:29their skill set is no longer worth
10. 0:31anything in this new age, which is I
11. 0:33believe that software fundamentals
12. 0:35matter now more than they actually ever
13. 0:37have.
14. 0:39And
15. 0:40I'm a teacher,
16. 0:42and I've been recently teaching a course
17. 0:44called Clojure Code for Real Engineers.
18. 0:47Nice and provocative. And
19. 0:49in the process of kind of working on
20. 0:50this course, I had to come up with a
21. 0:52curriculum about
22. 0:54AI coding, which is a bit of a nightmare
23. 0:57because things are changing all the
24. 0:59time, right? AI is a whole new paradigm.
25. 1:02We need to chuck out all of the old
26. 1:03rules, surely, so that we can bring in
27. 1:05the new stuff.
28. 1:08And there's
29. 1:09a kind of movement that has come up
30. 1:12around this, which is the specs-to-code
31. 1:14movement. And the specs-to-code movement
32. 1:16says that, "Okay, you can write a
33. 1:18specification about how an application
34. 1:19is supposed to work. Then you can use AI
35. 1:21to turn it into code. If there's a
36. 1:23problem with the application, you then
37. 1:26go back to the spec. You don't really
38. 1:27look at the code. You just change the
39. 1:29spec, you run the compiler again, and
40. 1:32you end up with more code." Raise your
41. 1:34hand if you've heard of that.
42. 1:37Keep your hand raised if you've tried
43. 1:38it.
44. 1:39Okay, I've tried it, too. You can put
45. 1:41your hands down.
46. 1:42And what I noticed was I would run it,
47. 1:45and I would try not to look at the code,
48. 1:48but I would look at the code, and I
49. 1:50realized I would get code out, first of
50. 1:51all, and then I would run it, I would
51. 1:53get worse code. And I did it again, I
52. 1:55got even worse code. I got it again, I
53. 1:58kept running the compiler, kept running
54. 1:59the compiler, and I would just end up
55. 2:00with garbage.
56. 2:03You know, raise your hand if that's
57. 2:04happened to you.
58. 2:06Yes. I don't think this works. The idea
59. 2:09that we can just ignore the code and
60. 2:10just have the code let it manage itself
61. 2:13is just sort of vibe coding by another
62. 2:14name.
63. 2:16And I didn't believe that back then. I
64. 2:18thought, "Okay, how do I fix the
65. 2:20compiler? How do I make it so that it
66. 2:22doesn't produce bad code each time, or
67. 2:23worse code?"
68. 2:25And so I thought, "Okay, I need to
69. 2:26explain to the LLM in English what a
70. 2:30good code base looks like." Let me dig
71. 2:32out one of my old favorite books, which
72. 2:34is a Philosophy of Software Design by
73. 2:36John Osterhout.
74. 2:37Go on Amazon, get it.
75. 2:39Um
76. 2:40and he has a definition for what bad
77. 2:43code looks like.
78. 2:45He calls it complex code. Complexity is
79. 2:47anything related to the structure of a
80. 2:48software system that makes it hard to
81. 2:50understand and modify the system, right?
82. 2:53So a a bad code base is a code base
83. 2:55that's hard to change. If you can't
84. 2:58change a code base without causing bugs,
85. 3:00then it's a bad code base. Good code
86. 3:01bases are easy to change.
87. 3:04So I thought, "Ooh, that was good.
88. 3:05Let's try another book. Let's try The
89. 3:06Pragmatic Programmer."
90. 3:08Go on Amazon, get it.
91. 3:10They have a whole chapter on something
92. 3:12called software entropy. And this is
93. 3:14exactly what I was seeing. Entropy is
94. 3:16the idea that things tend towards um
95. 3:19disaster and uh floating away from each
96. 3:21other and collapse. And this is exactly
97. 3:23how most software systems behave, too,
98. 3:25is that every time you make a change to
99. 3:27a code base, if you're only thinking
100. 3:28about that change and not thinking about
101. 3:30the design of the whole system, your
102. 3:32code base is going to get worse and
103. 3:34worse and worse. And that's what I was
104. 3:35seeing.
105. 3:36Everything inside the specs-to-code idea
106. 3:39that you just run the compiler again and
107. 3:40again was making worse code.
108. 3:43Now, there's an idea that sort of drives
109. 3:46the specs-to-code movement,
110. 3:47which is that code is cheap. Raise your
111. 3:50hand if you've heard that phrase before,
112. 3:51that code is cheap. Yeah.
113. 3:55Well, I don't think this is right.
114. 3:57I think code is not cheap. In fact, bad
115. 4:00code is the most expensive it's ever
116. 4:02been.
117. 4:03Because if you have a code base that's
118. 4:04hard to change, you're not able to take
119. 4:07all of the bounty that AI can offer, cuz
120. 4:10AI in a good code base actually does
121. 4:12really, really well.
122. 4:15And this means good code bases matter
123. 4:16more than ever, which means software
124. 4:18fundamentals matter more than ever.
125. 4:20That's the thesis of this talk.
126. 4:22So let's actually get into practical
127. 4:23stuff.
128. 4:25I'm going to talk about different
129. 4:26failure modes that you may have
130. 4:27experienced, or you may not have
131. 4:28experienced yet with AI, and how you can
132. 4:30avoid them by just going back to old
133. 4:32books and looking at good software
134. 4:34practices. Sound good?
135. 4:36So the first one is that the AI didn't
136. 4:38do what I wanted.
137. 4:40You know, I I thought I had a good idea
138. 4:42in my head, and the AI just did
139. 4:43something totally different, or it did
140. 4:45some uh like specs that I, you know, it
141. 4:47just made something I didn't want. Raise
142. 4:49your hand if you've hit this mode.
143. 4:51Cool. Okay.
144. 4:53Well,
145. 4:54this is what they say in The Pragmatic
146. 4:55Programmer, is that no one knows exactly
147. 4:57what they want. It's that you and the
148. 5:00AI, there is a communication barrier
149. 5:02there, right?
150. 5:04And so when you're talking to the AI,
151. 5:05that's kind of like the AI doing its
152. 5:07requirements gathering. It's basically
153. 5:09working out from you what it is that you
154. 5:11need.
155. 5:12And
156. 5:14I realized that there was another book,
157. 5:16Frederick P. Brooks' The Design of
158. 5:17Design,
159. 5:19and it talks about this idea called the
160. 5:20design concept.
161. 5:22It's that when you have more than one
162. 5:23person designing something together, you
163. 5:25have this idea sort of floating between
164. 5:28you, this ephemeral idea of the thing
165. 5:30that you're building. And that thing
166. 5:32that you're building, or the idea of it,
167. 5:34is called the design concept. It's not
168. 5:36an asset, it's not something you can put
169. 5:37in a markdown file, it is the invisible
170. 5:40sort of
171. 5:41theory of what you're building.
172. 5:44And so I thought, "Okay, that's what's
173. 5:46going on. Me and the AI don't share a
174. 5:48design concept." So I came up with a
175. 5:50skill.
176. 5:51The skill is very, very simple. It's
177. 5:53called Grill Me,
178. 5:54and it looks like this.
179. 5:57"Interview me relentlessly about every
180. 5:58aspect of this plan until we reach a
181. 6:01shared understanding. Walk down each
182. 6:03branch of the design tree, which is
183. 6:05another thing from Frederick P. Brooks,
184. 6:07resolving dependencies between decisions
185. 6:09one by one."
186. 6:10This skill is like uh the repo
187. 6:12containing this skill has like 13,000
188. 6:14stars or something. Like, it just went
189. 6:15nuts, went viral. People love this
190. 6:17thing. It These couple of lines means
191. 6:20the AI asks you like 40 questions, 60
192. 6:23questions. I've had it ask uh people 100
193. 6:25questions before it's satisfied they've
194. 6:27reached a shared understanding. And it
195. 6:29means it turns the AI into a kind of
196. 6:32adversary, where it's just continually
197. 6:34pinging you ideas and trying to reach a
198. 6:36shared understanding.
199. 6:38And that means that the conversation
200. 6:39that you then generate, you can take
201. 6:41that and turn it into a product
202. 6:43requirements document or something. Or
203. 6:45if it's a small change, you can just uh
204. 6:48do
205. 6:49uh turn it directly into issues.
206. 6:52And then your AFK agent will then pick
207. 6:53it up.
208. 6:54And
209. 6:55don't at me on this, but I personally
210. 6:57believe this is better than the default
211. 6:59plan mode in uh the
212. 7:02tool that I use, which is Clojure Code.
213. 7:04Plan mode is extremely eager to create
214. 7:07an asset. It really wants to uh just
215. 7:09create a plan and start working.
216. 7:12Whereas I think it's a lot nicer to
217. 7:15reach a shared design concept first.
218. 7:18So that's tip number one.
219. 7:21Now, failure mode number two is that the
220. 7:22AI is just way too verbose.
221. 7:25It's like you're almost talking at
222. 7:27cross-purposes with the AI. Raise your
223. 7:29hand if you uh feel this. If you've ever
224. 7:31experienced that failure mode. Yeah.
225. 7:33It's kind of like the AI is like talking
226. 7:34just using too many words to try to
227. 7:36communicate what it's doing. It's not
228. 7:38like you're talking uh using the same
229. 7:40language.
230. 7:41And this to me felt very, very familiar,
231. 7:44right? If you've ever been a developer
232. 7:46for a long time, and you've worked with,
233. 7:48let's say, domain experts, someone
234. 7:49building an application, um let's say
235. 7:52the domain expert wants you to build
236. 7:53something on uh I don't know,
237. 7:54microchips. You have no idea what
238. 7:55microchips are.
239. 7:57You need to establish some kind of
240. 7:58shared language, right? Cuz otherwise,
241. 8:00they're going to be using terms you
242. 8:01don't understand. You're going to be
243. 8:02translating that into code that maybe
244. 8:04you don't even understand, and certainly
245. 8:06the domain expert won't.
246. 8:07And so there's this kind of language
247. 8:11gap between you and the domain I went
248. 8:14back to domain-driven design, DDD.
249. 8:17This is something I'm still kind of on
250. 8:19the edge of exploring, but everything
251. 8:20I'm reading about DDD is just music to
252. 8:23my ears. I freaking love it.
253. 8:25And DDD has a concept of a ubiquitous
254. 8:27language.
255. 8:30With a ubiquitous language,
256. 8:32conversations among developers, and
257. 8:34expressions of the code, and
258. 8:35conversations with domain experts are
259. 8:37all derived from the same domain model.
260. 8:39It's essentially a markdown file full of
261. 8:41a list of terms that you and the AI have
262. 8:43in common. And you really focus on those
263. 8:46terms, and you really make sure that
264. 8:47they're aligned with what it actually
265. 8:49means, and you use them all the time in
266. 8:51the code, when you're talking about the
267. 8:52code, when you're talking to domain
268. 8:54experts, or in our case, when you're
269. 8:55talking with AI.
270. 8:57So I made a skill.
271. 8:59This skill is the ubiquitous language
272. 9:01skill. Basically just scans your code
273. 9:03base, looks for terminology, and then um
274. 9:07creates a markdown file. Creates the
275. 9:09ubiquitous language markdown file, a
276. 9:11bunch of markdown tables with all of the
277. 9:13terminology.
278. 9:14And this, then I pass it to the AI,
279. 9:17and I'm able to read it, too. And I
280. 9:19actually have it open all the time when
281. 9:21I'm grilling with the AI and planning
282. 9:22and that. What I noticed by reading the
283. 9:24thinking traces of the AI, it not only
284. 9:26improves the planning, but it allows the
285. 9:29AI to think in a less verbose way, and
286. 9:32actually means that the implementation
287. 9:33is more aligned with what you actually
288. 9:36planned. So this has absolutely been a
289. 9:38powerhouse. It's been unbelievably good.
290. 9:41So that's tip number two. Create a
291. 9:42shared language with the AI.
292. 9:45So okay, let's imagine that you've
293. 9:47aligned with the AI. You know what it is
294. 9:49you're supposed to be building. The AI
295. 9:51has built the right thing,
296. 9:53but it doesn't work.
297. 9:55Raise your hands if that's happened to
298. 9:56you.
299. 9:57Yeah, just doesn't work.
300. 9:59Well, there's an obvious thing that we
301. 10:01can do to make that better, which is we
302. 10:03can use feedback loops. We can use um
303. 10:06static types, you know, if you're not
304. 10:07using TypeScript, uh
305. 10:09that's crazy. Uh if you're not using uh
306. 10:12if you're building a front-end app and
307. 10:13you're not giving it the LLM access to
308. 10:15the browser so it can look around,
309. 10:17absolutely needs that.
310. 10:19And you obviously also need automated
311. 10:21tests.
312. 10:23And one sort of
313. 10:26thing I notice here is that even with
314. 10:28these feedback loops, the LLM doesn't
315. 10:30use them very well. It doesn't kind of
316. 10:32like get the most out of its feedback
317. 10:34loops in the way that a veteran
318. 10:35developer would. And so it does what it
319. 10:38tends to do is just does way too much at
320. 10:40once. It will produce like a huge
321. 10:42amounts of code and then think, "Oh, I
322. 10:44should probably type check that
323. 10:45actually." Or I should uh you know,
324. 10:47maybe check a test on that or maybe do
325. 10:48something like that.
326. 10:50And this in the Pragmatic Programmer
327. 10:52they describe as outrunning your
328. 10:53headlights. It's essentially driving too
329. 10:56fast because
330. 10:58the rate of feedback is your speed
331. 11:00limit.
332. 11:02The rate of feedback is your speed
333. 11:03limit, which means that you should be
334. 11:05testing as you go, taking small
335. 11:07deliberate steps. And the AI by default
336. 11:09is really not very good at that.
337. 11:11So, skill number three is TDD.
338. 11:14You should be using test-driven
339. 11:16development
340. 11:17because TDD forces the LLM to
341. 11:21really take small steps. You create a
342. 11:24test first, you make that test pass, and
343. 11:27then you refactor the code to make it
344. 11:29nicer and consider the design.
345. 11:32The issue here
346. 11:33is that testing is really hard.
347. 11:36Testing has always been hard.
348. 11:38And the reason for that
349. 11:41is there are a ton
350. 11:43of different decisions you need to make
351. 11:44when you write a test.
352. 11:46You need to figure out how big a unit do
353. 11:48you want to test?
354. 11:50You need to figure out what to mock. You
355. 11:52need to figure out what behaviors do you
356. 11:54even want to test in the first place?
357. 11:55And all of these decisions are
358. 11:56dependent. So, if you are testing a
359. 11:58really big unit like an entire uh
360. 12:00massive application, then it might be
361. 12:03quite flaky. You might not want to test
362. 12:04that many behaviors. You know, if you
363. 12:06only test this unit, you need to mock
364. 12:08this unit, you know. It's all
365. 12:09interlinked. And I've been thinking
366. 12:11about this for years, for my entire
367. 12:12development career.
368. 12:15And what we notice is that good
369. 12:17codebases are easy codebases to test.
370. 12:20Right? So, here we're starting to get
371. 12:22back to the idea of code being
372. 12:24important. It's that the better your
373. 12:26codebase is, the better your feedback
374. 12:27loops are because you're able to um
375. 12:31give better feedback to the LLM, it
376. 12:33produces better code.
377. 12:35And so I thought, what does a good
378. 12:37codebase, what does a testable codebase
379. 12:38look like? Again, we go to John
380. 12:41Ousterhout. [clears throat]
381. 12:42He talks about having deep modules in
382. 12:45your codebase. Not shallow modules, not
383. 12:47lots of modules that expose type kind of
384. 12:50um lots of functions.
385. 12:52They should be relatively few large deep
386. 12:54modules with simple interfaces.
387. 12:57Let's compare them quickly.
388. 12:59Deep modules, lots of functionality
389. 13:01hidden behind a simple interface. Hiding
390. 13:04the complexity.
391. 13:05You can look inside the deep module if
392. 13:06you want to, but you don't need to. You
393. 13:08can just use the interface. Shallow
394. 13:10modules, not much functionality, complex
395. 13:12interface.
396. 13:13And
397. 13:15I'll just wait for you to take the
398. 13:16photos.
399. 13:18Shallow modules in a codebase kind of
400. 13:19look like this, where you have a ton of
401. 13:22different tiny little blobs that the AI
402. 13:24has to walk through and navigate. And
403. 13:26this is really hard for the AI to
404. 13:29explore actually.
405. 13:30And so often what you'll see is if you
406. 13:32have a codebase like this, which AI is
407. 13:33really good at creating codebases like
408. 13:35this,
409. 13:36is that you'll have a situation where AI
410. 13:38doesn't understand what your code is
411. 13:40doing. It will attempt to explore the
412. 13:42code, but because it's poorly laid out,
413. 13:45filled with shallow modules, it doesn't
414. 13:47maybe get to the right module in time or
415. 13:49doesn't understand all the dependencies,
416. 13:50all that stuff. It doesn't understand
417. 13:52your code.
418. 13:53And so what does a
419. 13:54codebase full of deep modules look like?
420. 13:57Well, it looks like this.
421. 14:00Where it's the same code, but it's just
422. 14:02structured inside boundaries, where you
423. 14:04have these interfaces on the top.
424. 14:08And these interfaces, you should
425. 14:10probably have a lot of control over them
426. 14:12and design them really well. Otherwise,
427. 14:14you know, AI might mess up the design.
428. 14:17But the implementation, you can kind of
429. 14:18leave that to the AI bit.
430. 14:20So, how do you turn a codebase that
431. 14:23looks like this into a codebase that
432. 14:26looks like that?
433. 14:29Well, I've got a skill for that. Improve
434. 14:31codebase architecture. Turns out this is
435. 14:33not It's it's quite complicated to do
436. 14:35this, but it's a
437. 14:36like a set of steps that you can
438. 14:38reusably do again and again. You just
439. 14:40sort of explore the codebase, look for
440. 14:42opportunities where there's code that's
441. 14:44kind of look um
442. 14:45related, and wrap all of that in a deep
443. 14:47module.
444. 14:50And this is a testable codebase because
445. 14:52the boundaries around this code are so
446. 14:54so simple. You test at the interface,
447. 14:56you verify using that interface,
448. 14:59and you're good to go. And so this is a
449. 15:00codebase that rewards TDD.
450. 15:04But how about failure mode number six?
451. 15:06Which is your Okay, let's say your
452. 15:07feedback loops are working. Let's say
453. 15:09that things are kicking into gear.
454. 15:11You're able to ship more code than you
455. 15:12ever have before, but your brain can't
456. 15:14keep up.
457. 15:16Right? Uh raise your hand if you've felt
458. 15:18more tired than you have ever before in
459. 15:20your development career.
460. 15:22Yeah, me too. It's knackering.
461. 15:25And I think that this is a codebase that
462. 15:27actually makes it harder for your brain
463. 15:30because you, as well as the AI, need to
464. 15:32keep all of that information in your
465. 15:33head.
466. 15:34Whereas this, not only is it simpler
467. 15:38for you to read and understand, it also
468. 15:40means you can kind of treat these
469. 15:42modules, or these deep modules, as gray
470. 15:45boxes.
471. 15:47You can kind of say,
472. 15:49"Okay,
473. 15:50I'm going to just design the interface,
474. 15:52but I'm not going to worry too much or
475. 15:53not review the implementation too much."
476. 15:57You can do this obviously with uh things
477. 15:58that are less critical in your
478. 15:59application. Can't do this with uh you
479. 16:01know, various things like finance or
480. 16:03whatever, but in many many modules in
481. 16:05your app, you don't need to think about
482. 16:07the implementation too much as long as
483. 16:09you have a testable boundary outside the
484. 16:11module, and as long as you understand
485. 16:13its purpose and can design it from the
486. 16:14outside. I have found this has really
487. 16:17saved my brain because I can just go,
488. 16:19"Okay, the AI, I'll let you handle
489. 16:21what's inside the big blob. I'm just
490. 16:23going to test from the outside and
491. 16:24verify it."
492. 16:26So, that's tip number five. Design the
493. 16:27interface, delegate the implementation.
494. 16:32But this means that whenever we're
495. 16:34touching the code, whenever we're
496. 16:35planning stuff, we need to think about
497. 16:37and be aware of the modules in our
498. 16:39application. We need to know that map
499. 16:41really well. It needs to be part of our
500. 16:43ubiquitous language. We need to build it
501. 16:45into our planning skills as well. So, my
502. 16:47write a PRD, inside the PRD I'm specific
503. 16:50about the module changes and the
504. 16:52interfaces inside those modules, how
505. 16:54they're being modified. I'm thinking
506. 16:55about them all the time. And this comes
507. 16:57from Kent Beck.
508. 16:58Invest in the design of the system every
509. 17:01day.
510. 17:02And this is the core of it, right?
511. 17:03Because specs to code, we are not
512. 17:06investing in the design of the system.
513. 17:08We are divesting from it. We're getting
514. 17:10rid of that.
515. 17:12Whereas this, I think, is absolutely
516. 17:13key.
517. 17:16And so
518. 17:18code is not cheap. That's the message I
519. 17:19want you to take away. Code is
520. 17:21important.
521. 17:23And if we think about AI as a really
522. 17:25great on-the-ground programmer,
523. 17:27a kind of tactical programmer, a
524. 17:29sergeant on the ground making the code
525. 17:32changes, you need someone above that.
526. 17:35You need someone thinking on the
527. 17:36strategic level. And that's you.
528. 17:39And that requires software fundamental
529. 17:41skills that we've been using for 20
530. 17:43years, for longer.
531. 17:46Now, if you were interested in any of
532. 17:48the skills I put up here, it's in the
533. 17:49GitHub repo macpocockskills.
534. 17:52And if you're interested in the training
535. 17:53that I do or any free stuff, I'm on
536. 17:55YouTube, I'm on Twitter, but I'm also at
537. 17:57aihero.dev, where I have a newsletter
538. 17:59you can check out.
539. 18:01Thank you so much. I hope that this
540. 18:03gives you confidence in this new AI age
541. 18:05that you can actually make a good
542. 18:06impact.
543. 18:07Thank you.
544. 18:09>> [music]
545. 18:10[applause]
546. 18:15[music]
547. 18:21[music]