How to Actually Code with AI (ft. Matt Pocock) — Transcript
Full transcript
- 0:00Would you say that anyone needs to learn a specific programming language right now? Nah.
- 0:04I think that software engineering has always been easier than people think it [music] is,
- 0:08right? Coding has always been easier than people think. I would have relatives coming
- 0:12to me and say, "Yo, you must be so smart. You're a coder. You know, you know how
- 0:14all that stuff fits [music] together." You know, but you just learn it. I do not have
- 0:18a software development background. Yet, I can now do it all. If I ask AI to code something,
- 0:22it can code it all day. But I can feel the limitations and I think everyone else does.
- 0:26Engineers tend to produce better products than non-engineers right now because they have the
- 0:32domain expertise. They [music] know what to ask for. They have the right language. Let's
- 0:37take a high school kid. They want to become a software developer. And in this day and age,
- 0:41it's like what does that mean? Should they? What I [music] would say is
- 0:49[music]
- 0:54first of all, thank you for making the time to do this. No worries. I know you've been busy.
- 0:58Like you just launched your new uh AI crash course and your new skills like was it 1.3
- 1:03you released? That's right. Yep. That's awesome. Y congrats. I've got a I've got a baby coming in
- 1:10uh late November and I'm trying to get a course out before then. It's unusual. I have this hard
- 1:16of a deadline, you know, but the baby is coming. Let's go ahead and get started. So, first for
- 1:22people who don't know who you are, which I'll kind of give you an intro. We can just kind of
- 1:25cold start here. Um you are Matt PCO and the where I've heard about you was initially your skills,
- 1:30which I think is what most people um that's their gateway drug to Matt PCO and um everyone uses
- 1:36them. It's like the de facto thing to use for AI coding now. But everyone just kind of adopts
- 1:42your skills to their own workflow without exactly looking at everything you do. I think that's kind
- 1:47of my my that that was kind of my approach. But then I saw that you had a course and I'm like I'm
- 1:52going to buy the course. So I bought your course and it's awesome. Um but it's also challenging and
- 1:58which is why I wanted to have this conversation with you. So um real quick, if there's anything
- 2:01I missed about you, can you go ahead and tell the audience who you are and what you're about? No, I
- 2:05mean I right now I'm the AI guy. I suppose I'm the skills guy. Um I've not always been this person.
- 2:10Uh before I was the TypeScript person actually. So I was a hu influencer in the TypeScript space.
- 2:17I was teaching Typescript courses and that came out from uh working in open source on very very
- 2:22very challenging TypeScript projects like XATE. I don't know if you've heard of Xate. It's a tiny
- 2:26little library. Well, not actually that tiny but it's about state machines and state charts
- 2:31in Typescript. And I was trying to make that type safe and that was a blooming nightmare. Uh taught
- 2:36me a lot about TypeScript and I released a course about that and went full-time teaching based on
- 2:40that course which was called total typescript and that meant that when a AI came around I was
- 2:46looking for my next project. Basically I didn't know what I was going to do and AI came and I
- 2:51realized ah this seems interesting. took me a while to get there because I was pretty skeptical,
- 2:56but I have risen through my skepticism and now yeah, I'm definitely one of the skills people
- 3:04pushing their skills out there and just trying to write down interesting processes I find and things
- 3:09that work and shipping them as markdown files. So, the reason I'm so excited to talk with you
- 3:16is I think I'm in a place of a lot of other people where I I have now been given a magic wand. I can
- 3:23do whatever I want, but I need to go to Hogwarts to learn how to use this thing. [laughter] And I
- 3:31uh I I'm coding like crazy. I'm probably coding as we speak. I I I'm sure I have an agent doing
- 3:36something, but I know I'm not doing it well. And I know there's a way to do it well. And and
- 3:42and where I'm really curious, and I think we're going to end up talking about this at some point,
- 3:46is where is coding sitting right now? how much can we give away to the agent and trust it versus how
- 3:53and I think you the term you use is putting agents in a straight jacket which I love that because
- 3:56it's like they do go off the rails and they do get kind of crazy. So that's why I'm excited. I think
- 4:01a lot of people in my audience are in the same boat. Like I do not have a software development
- 4:05background yet I can now do it all. I don't have to know how to write any code but I can feel the
- 4:11limitations and I think everyone else does. Like I made this app but it's on my computer. How do
- 4:16I get other people to use it? I don't know what's the best way to do it. let me ask my AI agent and
- 4:20then they spin up Versell or they do something random and it's it's their opinion and it's not
- 4:24good. So what I would love to know and we I guess we can just go ahead and jump into this. Can you
- 4:30tell me what actually you tell me what what is the best way to approach software development
- 4:36right now with AI coding? Cuz like you you teach the course, you're the master. How can how can
- 4:41we step into this? I don't like this framing. I don't like the framing of me being the guru and
- 4:46you guys being the students. You know what I mean? Like that's that's it's that's not what I'm going
- 4:49for at all. Like I you know I've got a bald head, right? I live near Oxford. I've got glasses. That
- 4:53doesn't make me smart. Doesn't make me smarter than you. Right. And I've got the British accent
- 4:57too which helps. But it it helps so much. You have no idea. So much right. [laughter] So okay,
- 5:04Johnow in his book a philosophy of software design talks about the difference between tactical and
- 5:10strategic programming. Right? Tactical versus strategic. Tactical is like the on the ground
- 5:15sergeant trying to win the battle and the strategic stuff is the general trying to win
- 5:21the war, right? The long view. And whenever, you know, preai, whenever you were working,
- 5:26you would try to be working with a bit of tactical and a bit of strategic all the time, right? You'd
- 5:32be thinking about, okay, how does this change make a difference in the long term? And also,
- 5:37how does this change like how does it accomplish my short-term goals? How does it help me win
- 5:41the battle of today? Now AI is really good at tactical programming and terrible at strategic
- 5:48programming. It's just garbage at it. It because of the way it's trained and because of the way
- 5:53like you think about an AI in a benchmark, right? It has a task to fulfill and it's being judged on
- 6:00that specific task. In other words, its focus is 100% tactical. Whereas it's not you're not
- 6:06going to get better scores on the benchmark by thinking long term and thinking strategically.
- 6:10And even how would you grade that, you know, it's very difficult to grade that because your
- 6:15strategic mistakes don't usually come back at you until like 9 months down the line,
- 6:19you know. So to in order to get good at working with AI, you need to get good at the strategic
- 6:26stuff because AI can take the tactical stuff, can eat it for you, but unless you're careful,
- 6:32it'll just focus on the stuff for now and won't care about the stuff long term. Okay. And I I do
- 6:39feel that because you're right, like if I if I ask AI to code something, it can code it all day and
- 6:45it it will even go in a very bad direction because it just wants to be that agreeable assistant for
- 6:50me. But strategy is where it's weak. Um would you say that's the the biggest weakness you see right
- 6:56now with people jumping into vibe coding? Totally. I mean with vibe coding is a people have like
- 7:04there's this term called semantic diffusion where like words stop meaning the thing they originally
- 7:08meant right and so vibe coding is a stupid um term that now means all sorts of different things
- 7:13I take the original definition which is coding without looking at the code without thinking about
- 7:18the code with considering that the code is just like a an artifact we don't care about now vibe
- 7:23coding I don't think has ever worked if it's the only thing you do right if you're only vibe coding
- 7:30It means that your code will slowly get worse and worse and worse because again this tactical agent
- 7:35is just thinking about how to solve the current problem. It's not thinking about the long term and
- 7:39it gets worse and worse and worse. And because the code is the environment your agent operates in,
- 7:46the environment your agent operates in slowly gets worse. It's like the dog meme of the house
- 7:50burning around them, right? And the dog is just sort of sat there going, "Oh, okay." Right? the
- 7:54agent just just watching the house fall down as they're trying to get their jobs done. So, you
- 8:00need someone to keep the house in order and that person is probably at the moment you. I do think
- 8:05that agents might be getting better at this, but they still need to be prompted and told to think
- 8:12about strategy, to care about strategy long term, and right now they just don't. So, you you don't
- 8:20call your course a a vibe coding course. And I and I see why you would steer away from that because
- 8:24that that vibe coding definition still has that negative connotation even though everyone is kind
- 8:29of vibe coding if you if you um if you call it AI assisted vibe coding. Vibe coding has a negative
- 8:36connotation. Uh do you think that developers right now all of them should be using AI? Like are they
- 8:45foolish to not use AI? Yeah, it's ridiculous not to use AI because so much of our job before was
- 8:52tactical programming, right? It was it was just mostly tactical and you would get these nice uh
- 8:57relaxing breaks where you could just focus on the tactical stuff, just solve the problem today. Oh,
- 9:01I get to kind of just go into a nice flow state and just solve it. But now you can pay less than
- 9:06minimum wage for someone else to do it, right? It's unreasonable for you to be spending your
- 9:10time doing that. And to be honest, the stuff that always interested me most as you know a developer,
- 9:16as someone who's interested in increasing velocity on our team, increasing the um just the quality
- 9:23of code that we're putting out, the in quality of features for people. The strategic stuff was
- 9:27always the most interesting stuff for me. Anyway, so I I mean I love this new age. I don't mind that
- 9:32tactical programming is dead. I'm fine with it. You know, just get me thinking about the long
- 9:36view. let me just sort of cook and create this beautiful um environment for my agents to operate
- 9:41in. I find that stuff really exciting. M okay. So that's that's where I want to live and I'm hoping
- 9:47that throughout this conversation you will give me and the audience a direction on where we can
- 9:52start actually becoming I guess becoming software engineers in a way cuz like right now I I don't
- 9:58see a place where I'm going to actually go out and hire a software engineer. Like I know I can
- 10:02do it myself. I just know there's a few things I'm missing, a few gaps I need to fill in. Um would
- 10:07you say that's where where people can be right now? Like can we just become software engineers
- 10:11with a few like a course that you produce? I think that um software engineering has always been
- 10:17easier than people think it is, right? Coding has always been easier than people think, you know?
- 10:21Uh you know, I would have relatives coming to me and say, you must be so smart. You're a coder. You
- 10:26know, you know how all that stuff fits together, you know? But like it's just you just learn it,
- 10:30right? you know, you just spend like a couple of weeks learning it and you get better over time and
- 10:34you make mistakes and you know, and you've got all these tools to help you out and now agents
- 10:38can help you too. And it's always been easier than it looks. And strategic programming is really just
- 10:45starting to think and care about different things than you're used to caring about. And honestly,
- 10:51so much of strategic programming is just caring a bit more than the person next to you,
- 10:57right? is caring about the quality of the thing you're producing, of the code, the environment or
- 11:02the agent operates in. And so so much of that like I could always tell which of the my team
- 11:09members were strategic programmers that had that long-term view because they just cared more. And
- 11:14I think honestly if you're just interested in it, if you want to produce better code and you want
- 11:19your agents to do it, then that's the first place to start. It's just, you know, can I can I swear
- 11:24on this? I'm not sure. Can I curse? Yeah. Yeah. Go ahead. Yeah. Okay, you have to give a you know
- 11:27what I mean? You just got to give a [ __ ] And so that's the first place to start. Hey, time for a
- 11:32coffee break. And this coffee break is sponsored by Threat Lucker and their learning event Zero
- 11:36Trust World. Did you know about this? It's a thing and it's kind of sick. It's happening February
- 11:4017th through 19th, 2027. Gosh, is it really already 2027 coming up? At Zero Trust World,
- 11:45you're going to get access to hands-on hacking labs where you get firstirhand experience hacking
- 11:49stuff. And it's okay if you're new to hacking. They have every skill level. They're also going
- 11:53to have mainstage talks, breakout sessions, and technical deep dives. These are experts
- 11:57in the field sharing their knowledge with you. And speaking of experts, you're going to be around a
- 12:01bunch of them at this event. Networking is one of the main reasons you want to go to any type
- 12:04of learning event. You'll be spending time with people who are dealing with the same threats that
- 12:07you are. And it's people who are practitioners and also executives. You're going to be making
- 12:11some connections and those sessions you take can be continuing education credits. This event has so
- 12:16much and it grows every year. And the afterparty, I heard, is something you don't want to miss. Why?
- 12:20You got to go and find out. So now I know you want to go. I got a link below. You'll get $200
- 12:24off or use the code ZTW networkchuck 27. So go to ztw.com and use the code ZTW networkchuck 27. And
- 12:32thank you to Threat Locker for sponsoring this podcast and thank you for taking a coffee break
- 12:35with me. Okay. So I I know I didn't prep you for this um and you can say no, but would you be able
- 12:41to walk me through an example of how you approach a coding problem and with with your skills? Yeah,
- 12:47totally. So what you want me to screenshot? Yeah. Yeah, that'd be great. Yeah. I think that's that's
- 12:52the one thing I think most people are missing. Anyone can jump into Codex now and go make me this
- 12:57thing. And Codex will start doing lots of things using lots of words people have never seen before.
- 13:01They assume it knows what it's doing. And then they spit out some code and you're like that's
- 13:05cool, but it's not strategic. And I want people to kind of feel what that feels. Oh yeah. See
- 13:11what that feels like as you do this. Totally. So essentially my skills have evolved into a process,
- 13:18right? And that process is different depending on the kind of task that I'm trying to do. And uh I
- 13:27think to answer your question is how am I thinking strategically is that the strategic thinking is
- 13:32sort of blended into every stage of the process. It's not just what am I solving today. It's how
- 13:37my how is the thing I'm solving going to be um uh you know going to benefit this project long term.
- 13:45So one I'm I'm inside a repo right now which is I have a sort of course video manager repo which
- 13:52is essentially this huge great big project which is like a video editor. It's an AI writer. It is
- 13:59um essentially where I produce all of my videos, all of my content. And the first place to start is
- 14:05by thinking very very carefully about the language you use when you're working with agents. Now,
- 14:12this comes from a software practice called domain driven design. Right? I've actually literally I've
- 14:16been filming this part of my course today. So, I've got the book next to me. This is Eric Evans
- 14:20domain driven design. Absolutely. Um it's a bit of a tome, right? But is the first couple of chapters
- 14:25that talk about the importance of language in development is so so cool. And so with a couple of
- 14:31my skills, I've got a skill called uh grill with docs, let's say. and grill with docs. What it does
- 14:37is it essentially quizzes you, grills you about the thing that you're about to build. You might
- 14:42have heard of my grill me skill, which is like my sort of most famous skill. Um, which is a sort
- 14:47of simpler version of this. So, let's say I go in with Grill with Docs. I'm going to say, I need to
- 14:52improve the teleprompter part of my video editor. I think it's just gotten a little bit cluttered
- 14:57and I think there's some stuff I no longer use in there. Can you help me out? So, I'm just going to
- 15:02give it a vague prompt. Basically, I'm going to send this off and it's going to do a couple of
- 15:05things. It's going to first of all explore the codebase and figure out what we're even dealing
- 15:11with and what we're looking at. And it's going to look in this glossery here for stuff that
- 15:15I've defined in terms of what is actually in my application and what the entities are. So inside
- 15:21here, you can see there's language, there's course structure, right? So I have a course and then
- 15:26we have sections inside courses. We have course versions, right? because courses change over time
- 15:31and I keep track of them. And then lessons are single learning units within a section. We have
- 15:37uh draft versions, whether it has changes, pending versions, and all of this stuff is language that
- 15:42I've developed with the agent over time. And that pays off because it means, you know, like
- 15:47how Opus 5 was just garbage to talk to, right? I didn't have that much of a problem with it because
- 15:53it was using the terms in these this glossery, right? It was using terms that I'd already agreed
- 15:58on. It wasn't that verbose because it was talking the language of my application already. And so,
- 16:05and by the way, are you using Whisper Flow right now? I'm using Whisper Flow. Yeah. Exactly. Okay.
- 16:10Love Whisper Flow. I use it all the time. Yeah. I'm a huge dictation fan. Like, um,
- 16:13if you're not dictating, you're just going way too slow, basically. So, it's got a sub agent
- 16:19sort of like looking in the background and sort of peeking at stuff. And then what it's doing is it's
- 16:24quizzing me about the thing we're about to build, right? So, this is essentially a grilling session.
- 16:28and it's going what does the teleprompter read from? When you film today, what's on screen? Now,
- 16:32this stuff is actually it's probably better if I just wait for the explore to finish because it's
- 16:36kind of asking me stuff that it can figure out in the code. When do I use it? What's on screen at
- 16:41once while you're reading? What clutter have you noticed? So, it's asking me sort of like vague
- 16:45questions that I can provide answers to. And we're sort of slowly getting aligned here. So,
- 16:50this is actually more tactical, right? This is about how do you get the agent to do what you want
- 16:55it to do? How do you align with the agent before you begin? Because I think what most people do is
- 17:00they just prompt the agent. And they don't they underestimate that there's a communication gap,
- 17:07right, between you and the agent. The agent is just like a developer that you're hiring
- 17:12in order to get the work done. And they've been dropped into this world like the guy from Momento,
- 17:17right, with no memory whatsoever. Just totally blank slate. How the hell am I going to like
- 17:22uh figure out what I need? So it immediately dives into the codebase and asks in in this case asks
- 17:27you good questions because usually what they'll do is they'll just kind of their way through and
- 17:33they'll just sort of say oh yeah sure that sounds good and I'll build something like what you want.
- 17:37Whereas if you force it to ask you questions if you try to get over that communication barrier
- 17:42it's much more effective and you're going to get the result that you want instead of the
- 17:46result that you sort of hope for. And I I feel that often. um I I won't be thorough enough in
- 17:53the initial ask and I'll get the result of it. Actually, just before this meeting, I I have I
- 17:59built a a tool to help me do interviews and keep track of my documents and I noticed the feature. I
- 18:05didn't ask it to build. It's just there. And I'm like, I kind of like that, but it wasn't in the
- 18:08original design. It just made the choice for me, which I'm like, uh I'm not sure if I like that it
- 18:14did that. [laughter] Exactly. Yeah. And you have this sort of when you're communicating something,
- 18:19you have this like you kind of know some parts of what you want, but maybe you don't know everything
- 18:24you want, right? And maybe it feels clear to you, but it probably isn't that clear to an agent,
- 18:30right? I've been a teacher for uh since I was 21 years old. So, well, that that's 14 years I've
- 18:37been a teacher. I'm quite good at knowing when someone understands what I'm saying or not. Right?
- 18:44But even me, I will try to prompt something. I will get complacent and it will just build me
- 18:50something out of scope or like won't build the thing that I wanted. You know what I mean? So
- 18:54there's always that communication gap and that's not going to go away if the model gets smarter,
- 18:58right? That's just always there because it's the human that's the in the wrong essentially.
- 19:02So what we've got now is we've got our grillings come back and it's quizzing me on all of these
- 19:07sources and it's saying, "Okay, it's using my language." So we've got beats and scripts and
- 19:13animatics. Does the script still autoscroll? Is the status area at the top end space etc. So this
- 19:19is kind of like to answer your original question. This is the tactical stuff coming through. We are
- 19:24fixing the problem and it's benefiting from this nice glossery that I've been building and working
- 19:29on as I've been working on this app over the last 6 months. So when you say grill with docs because
- 19:35I'm very familiar with grill me. I use the grill me skill. I'm not familiar with grill with docs.
- 19:39So you're saying the docs part is this document that you've created or is there more docs that
- 19:44we're not seeing? I'll show you. So this is my skills repo and if we take a look at the grill
- 19:51me skill. So if we take a look at the grill me skill, it's essentially very short now. It's just
- 19:58call the skill tool with grilling. So it's going to find you this grilling skill and call it. And
- 20:04this is the grilling part of the grill with docs and grill me. Now what grill with docs looks like
- 20:13is it looks like this. It says use grilling but also use this second skill called domain
- 20:18modeling. And domain modeling is this extra docs. It's this glossery and also this extra
- 20:24little line about architectural decision records as well which is something I've added in here.
- 20:29ADRs what they do is they essentially document the stuff that you can't document in code. You know,
- 20:36code is like the current view of all of the decisions that went into building this product,
- 20:42right? But it's only the present. It doesn't tell you about the past. And if you don't have
- 20:47like a small record about the past, then the agent will just continually make the
- 20:51same mistakes over and over. And so these ADRs, this little list of markdown files,
- 20:56essentially they're quite small. I don't know if I've got any in in this repo. Docs ADR. Yeah,
- 21:02here we go. So these ones here. Uh yeah, I've got what 31 ADRs just knocking about in the repo.
- 21:09And these essentially just track the important decisions I've made that are hard to reverse,
- 21:14expensive to take back. And it means that the agent just has a lot richer context when it goes
- 21:19to build features and grills me. Okay. So when you're grilling or when it's grilling you, it will
- 21:24go through those it's instructed to go through those ADRs. It'll go through your glossery. Um,
- 21:29is that something that you had to like within that skill you you have it written in there go to the
- 21:34ADRs, go to the glossery. Is that built into the skill or is that something you added custom for
- 21:38you and this one repo? Um, this is built into the skill and it's also it's basically in every skill
- 21:44I have it says reference the glossery. Use the glossery when you're building specifications. Use
- 21:49it when you're building modules too. So the code will actually be using the same language as when
- 21:54I speak about the code, right? That makes the repo incredibly easy to navigate because everything it
- 22:00just pulls up everything about a course version and the database is aligned. The uh uh codebase,
- 22:05the front-end codebase is aligned, everything just reads properly. And it's amazing how many repos
- 22:12you'll see and that I've worked with in my career where the front end will call, let's say we have a
- 22:18refund or something, the front end will call it a reversal. the email that sends will call it a
- 22:24pending chargeback and the code will say something completely different, right? Code will call it a
- 22:30refund in the database. This sort of language, it just makes the app so much harder to understand,
- 22:35so much harder to work on for you and for the agent. And I think we underestimate again these
- 22:41communication barriers between you and the agent. And if you use a shared language, it just feels
- 22:46like the agent is in your brain. You know what I mean? It's just a totally different experience.
- 22:50So when you let's say you initially started this project and you're starting from scratch,
- 22:56are you by hand typing out or are you dictating the glossery and telling it what to make or are
- 23:02you kind of just giving the agent a general idea and letting it create the glossery for you. So the
- 23:09cool thing about this is you really like if you're thinking, okay, I want to use this on my project,
- 23:14but I don't I I don't want to have to go through the whole rigma roll of having to set all this up,
- 23:18right? That sounds like a pain in the ass. What you can do is you can just start grilling and it
- 23:23will mint the terms and quiz you about the right terms as you go. And that way you can just sort of
- 23:29slowly build it outwards just focused on the hot spots that you're actually working on. And then
- 23:34it will say, "Okay, would you like to update the code so that it matches the glossery?" And slowly
- 23:38slowly it's like you're peeling like a little window of sanity in your vibe coded slot base,
- 23:43right? It's just beautiful. And like as as you do it, it just feels like magic because you just
- 23:49realize, God, I understand what's happening in my project now. Everything reads as if it's like
- 23:55as if I wrote it. So, it's hard to describe without actually walking through and doing it
- 24:00and seeing how this benefits you long term, but it's the first strategic thing that I'd
- 24:04recommend folks do. So, for anyone just like watching this now, and just like by the way,
- 24:09what you said, I'm sure someone's like, "Oh, that sounds awesome. and that's what I need right now
- 24:13for this mess I'm coding. What's the quickest way to apply that? Is it just to go out to your skill
- 24:20repo and just say install or use it? Point your agent in that direction. What's the best way to
- 24:25do it? Yep. You just go to aihero uh.dev/skills and I've got the um essentially a full set of uh
- 24:33documentation on all of these skills. And for your next feature, just use grill with docs instead of
- 24:38uh let me just get the screenshot so it's open. What was that? What [laughter] was that? I think
- 24:45that's tracking how much time I spend on Twitter. That's what that was. And it's Guess what, guys?
- 24:50The uh the verdict is not good. Um so this is Grill with Docs. So essentially, you just install
- 24:56the skill and you get started with the skill. Okay. Okay. That's that's pretty simple. Okay.
- 25:02That's awesome. Um and by the way, I had the same problem with Twitter. I had to um uninstall it
- 25:08from my phone and I'm now coding a thing that will keep me off Twitter but keep me up to date. So,
- 25:13it's a thing. Anyways, it's a bad bad uh bad addiction. So, once we've once we've grilled and
- 25:21we get to that point where the the AI understands what we want and it understands the feature,
- 25:26what's next? So, you've really got two choices. It depends if the thing you're building is massive.
- 25:33If the thing you're building is uh relatively small and you think it can fit inside the same
- 25:39context window as in the grilling, then you go to the implement skill. And the implement skill is
- 25:45essentially you turn the idea that you've just grilled about into code. Now when I talk about
- 25:51context window, I'm sort of talking about I'm not really talking about the 1 million token context
- 25:56window that Opus has. I consider there to be a smart zone and a dumb zone inside these context
- 26:03windows which is as you add tokens to the context window. The model makes uh dumber and dumber
- 26:10decisions sort of happens slowly as essentially you add tokens and the relationships between the
- 26:17tokens increase quadratically. you get this sort of crazy attention degradation where it's having
- 26:22to hold all this information in its head but also listen to the most important stuff in a
- 26:27room that's getting full of people right it's very very difficult task and I don't models are
- 26:34getting better at it but they're still not great at it so I get start getting nervous around 150k
- 26:39tokens right really that's early very early right because I I want the good stuff right I want my
- 26:45models making smart decisions and I don't want them like if I like ask a follow-up question,
- 26:51I don't want to be paying again for the um you know 300,000 tokens that I've got in my context
- 26:57window, right? I really like optimizing for small context windows. It keeps things very nimble,
- 27:02keeps things very fast as well because actually latency increases the more tokens you add,
- 27:07you know. So for me, it just feels like a general principle of health keeping my agent in the smart
- 27:12zone and keeping it very very early. Now the actual number I think doesn't matter. It's more
- 27:17about the instinct of context paranoia of keeping things small, keeping the agent uh smart. And so
- 27:24if you feel like your thing you're building is going to be bigger than 150,000 tokens, then you
- 27:32create a spec. So you take the conversation that you've just had in Grillwood docs and you turn
- 27:37it into a specification. And this specification lives either in a markdown file or it lives in
- 27:42your GitHub repo as an issue. And then that spec sort of defines where you're going, right? And
- 27:48from there once you've figured out where you're going and you can kind of go, okay, I can check
- 27:52the final product against this specification, then you turn that spec into tickets. And those
- 27:59tickets are the thing that the agent is going to walk through and actually implement one by one.
- 28:04So each ticket is going to be a session. So each ticket gets implements on it and then you can use
- 28:11my code review to do a sort of adversarial review on the thing that was built to check that it
- 28:15matches up against that spec. So that is the whole flow, right? So you've got two choices. Either you
- 28:21go down this spec and tickets route or you just implement the thing that you grilled. Okay. Um I
- 28:28have a few questions on context. How do you what what are your tactics to manage like sometimes
- 28:36when you when you're doing a grill me I can imagine is if you have a big code base it could
- 28:41fill up that context window very quickly. So how do you have sub agents doing I mean we saw a bit
- 28:48the sub agent was doing explorations are using sub agents to do a lot of that to manage the context.
- 28:52So yes, I think I think there there so there's like four things there, right? The first thing
- 28:59is I think people equate big code bases as big code bases always use more context, right,
- 29:05to explore. I don't think that's true. I think that hard to navigate, poorly organized code bases
- 29:12are harder to explore. And you can have a small codebase that's actually hard to navigate, right?
- 29:17And you can have large code bases that are easy to navigate. And so we can talk later about how
- 29:22to improve navigability of your codebase because that's again something that we've got here. Well,
- 29:26you said there were four things that you do to manage the or some tactics to manage context. Yes.
- 29:32And I can't remember what they were. [laughter] Oh no, you compacted your context. Oh no, I've
- 29:39compacted at the wrong [laughter] moment. Damn it. Damn it. What is going on with me? Um I've been
- 29:46filming all day and my brain is mushed. That's That's why I completely understand that. That is,
- 29:50by the way, thank you for taking the time to do this interview after filming all day. I know that
- 29:54you spend all your tokens in that those moments. Um, now I did have another question on that. So,
- 30:00when you're doing Grill Me, uh, which agent are you using? Are you using Fable? Are you using
- 30:05Opus 5.5? What do you trust with that? Yeah, I'm using Opus 5.5. And I think I remember one of the
- 30:13points I wanted to make which was I think there's an epidemic of people using too high an effort
- 30:19right I think most folks are using way too high effort either like high x high or max whatever
- 30:26and what that means is when they do an exploration the agent like if you think about effort an effort
- 30:33like producing um let's say a painting right or a animation you know you get opus 5.5 five and you
- 30:40maybe want it on like a higher effort because it's going to put more effort in and produce a
- 30:45better painting or produce a better animation. But if you think about the task of exploration like
- 30:51like is is more effort going to get you a better exploration, you know what I mean? Sure, it might
- 30:56sort of look in more files, but does more files it really just needs to read the right files,
- 31:01you know, like and find the right information and give you that information back. So I think
- 31:06people are using way too high efforts to explore their code bases when mostly what I just use for
- 31:11pretty much everything is Opus 5.5 median, right? That's just really Yeah. So for me, I'm like and
- 31:17I I think there's still massive confusion on this because I think effort well I want my AI model to
- 31:22do his best every time. I that's that's what what I would expect from a human like do your best
- 31:28always explore that with 110%. Um so I keep mine on extra high. Yeah. And I think I think that is
- 31:35a big mistake because what you're doing there is you're just telling it to spend tokens on tasks
- 31:39where it doesn't need to spend tokens and so you end up getting into the dumb zone faster because
- 31:44it's just and and it's going to perform worse because it spent all of its effort on too early
- 31:49a stage. So yeah, Opus 5.5 medium is what I'm on and I'm even thinking of dropping that right to
- 31:55low genuinely. Wow. So, are there any moments when you're specking something out that you feel the
- 32:02need to go to a higher reasoning or higher model? Not particularly. I think maybe research tasks are
- 32:07interesting one, especially when it's the web, right? Because the web is very hard to navigate.
- 32:12It's hard to find information in. And so, I think probably high XI is a good example of that.
- 32:18when I'm doing performance testing, I think, or maybe debugging where it needs to try lots and
- 32:24lots of different things, I don't want it to stop early, that's when I might go to a higher effort
- 32:29level. But in practice, usually I just default to medium bis because I I'm thinking about other
- 32:34things and it's not something I'm constantly twiddling or tweaking. Gosh, that is crazy. And
- 32:40I'm I'm probably going to switch to medium after this just to see how it feels. Um, and I imagine
- 32:45it's a lot faster and more token efficient. That would be that would be fun. So Opus 5.5 is rapid
- 32:51as well. Like it's fast anyway. And so when you reduce the it's it's a very good model,
- 32:56guys. It's a really good model. Do you switch back and forth between uh Anthropic and OpenAI? I've
- 33:02got an anthropic subscription and I I just haven't really picked up a codeex model or like I haven't
- 33:09had a fiddle around with it. I tend not to minmax between models that much. I tend to think like I I
- 33:16feel I think this is a little bit sort of I I like holding contrary opinions, right? It's it's fun to
- 33:23hold an opinion that most people don't hold. I tend to think that if you keep the model static
- 33:27and you stop the room around you from burning down, you're going to get much better results. In
- 33:33other words, if you fix the environment your agent operates in, there's often so much lowhanging
- 33:37fruit there that people just underestimate basically and they think, "Ah, the new model
- 33:42will fix all my problems, right? Or if I crank it up another effort level, then I'll get better
- 33:46results." And sometimes you will, but you'll probably just burn a ton of tokens there. I think
- 33:50instead of waiting for the next model to fix all your problems, why not just fix all your problems?
- 33:55That's my take. Interesting. Okay. saying you're saying that going through a method like you have a
- 34:01methodology with your skills and getting your code base organized that will probably fix the problems
- 34:06that you're hoping a new agent that's smart enough to just do that for you would do. Yeah, exactly.
- 34:11And we can touch on a skill that does specifically that if you like. Yeah. And if you want to keep
- 34:17keep on context. Yeah, let's let's stay on context before we um start a new session. We're just using
- 34:22a bunch of AI stuff. It's I'm so AI pilled. That's how I think now. I'm like, hold on, we start a new
- 34:28session because this context is already full. And it's so funny how humans are kind of like that.
- 34:32Um, so we we've done we we can take a large idea. If it's a large idea, it'll put it into tickets,
- 34:38which I imagine at that point you're having one orchestrator agent divvy that up between sub
- 34:44aents and they're working that in like work trees. Is that is that what's happening? Yeah, exactly.
- 34:48Um there is a skill here called implement spec which basically you pass in the spec that you've
- 34:53just created in a new session and it goes and implements it all with parallel sub aents doing
- 34:58it in work trees merging all back together until you get one mega PR ready for you to implement or
- 35:03ready for you to review. Wow. Okay. That's I like that. And I just can't get over the fact that you
- 35:09um you don't AI max [laughter] because like I do you only have one anthropic subscription because
- 35:16that would surprise me. Just one. That is crazy. 200x. Yep. My goodness. Um, well,
- 35:23I don't I'm not token efficient, so I have three of those bad boys. And I also have probably a
- 35:28couple uh codeex ones, too. But I'm just always testing stuff. But the way you approach this,
- 35:34how simple. Well, it's it's not exactly simple, but it's just very straightforward.
- 35:38It's refreshing. Yeah. I'm I think that again it's this sort of slightly wanting to have a different
- 35:46view than everyone else. I was just sort of surprised by how well this works and by how
- 35:52token efficient it is actually. Um I think a lot of people would say it's not token efficient you
- 35:57know like uh sure implementing sub aent like people have different appetites for how many
- 36:01tokens they want to spend and that's fine. Um, but yeah, I think just keeping it relatively simple,
- 36:08having a clear coherent philosophy about the way you want to use AI is important. And I find the
- 36:14smart zone dumb zone thing has been very helpful in clarifying my decision-m and understanding. And
- 36:19this is not my idea, by the way. This is Dex Hy um came up with this who you should have on actually.
- 36:24He's very very clever with this stuff. And um yeah, just having that as like a guiding principle
- 36:30has been really nice just to clarify my thinking about how to use it. Now, with um with creating
- 36:38those tickets, I'm sure the agent is smart enough to divvy up those tickets to where they're small
- 36:43enough that the sub agent itself won't get into the dumb zone. Correct. Yes. They're not amazing
- 36:49at it, but yeah, they do a good enough job. Um, and I think of it as having like if you imagine
- 36:54each ticket should be around 150,000 tokens, then that's about right. Okay. How do you keep track of
- 37:02the tickets that's created? I guess it creates it inside the repo, right? So, it creates worked on
- 37:08just as GitHub issues basically. So, you can just go on the [clears throat] GitHub issues. They're
- 37:11tracked as sub issues underneath each spec. And so, then you can just see the spec being
- 37:16um completed as it's being worked through. Okay. So then after after it all completes and the
- 37:22work trees are finished and it's it's merging all the code, what's next? Yeah. So I do two things.
- 37:28So what we do is we go on a code review. So the implement spec actually calls code review on your
- 37:35um the code that it's done. And here two things happen. So it does a standards review and a spec
- 37:42review. So spec review is it compares the code against the original spec and it checks whether
- 37:48it did the right thing right did it build the thing that I wanted or has it gone over scope
- 37:53or under scope and then with standards it reads the standards files that you have in your repo so
- 37:59let's say you have a a codingstandards.mmd file and I can pull one up for you if you like these
- 38:05coding standards.mmd this is full of information on how I want my applications to be built right
- 38:11so I use effect which is a very cool TypeScript library and it has a ton of stuff in here for
- 38:18all the information I want to review. So every any in Typescript is a leak. Uh specific stuff
- 38:24about course versions, you know, this is really dense stuff. I think a lot of people would have
- 38:30this in their agents.mmd file, right? I imagine maybe is this something you do as well? You know
- 38:38what's funny is I don't even pay attention to that. I trust the agent with it. Yep. Right.
- 38:45You just think, okay, I'm not worrying about these steering files or not worrying about what
- 38:49the agent is reading or thinking about the agent's environment. Yeah. I'm more like I I I definitely
- 38:53tell my agent, I want to have a standard, but I want you to create the standard for me and I don't
- 38:58want to have to look at it. Got you. Yeah. I think that's um I would probably describe that as vibe
- 39:05coding. I think I think that's vibe coding, right? Because it's not really caring about, you know,
- 39:10I want you to be good, but I don't care how you're good, you know? [laughter] Yeah. It's it's like
- 39:15that that meme I or some kind of Tik Tok I saw a while back. It's just like think faster. Deep
- 39:20research. Do it. Do just be smart. Make it happen. Like it's just making wishes and and um yeah,
- 39:26ultra ultra think you know um ultra think ultra think do it right until you're um you know or
- 39:31I'll fire you or something. You know what I mean? Just like so that is what we're that's the second
- 39:36axis that we're comparing on. Is it built correct according to my coding standards? because I sort
- 39:41of assume that when my agents implement something, they're going to get stuff wrong. And for me,
- 39:46that's fine because I've got a code review step that goes again and does this adversarial review
- 39:51and checks it against the standards. And so once that's done, then I'll go and human review,
- 39:58right? And I don't do this for everything. I do it for some things, for very important work. And if I
- 40:03was like being professionally paid to like build an application, then I would probably do it for
- 40:07most things really. So describe to me what the human review looks like. Is it you actually in
- 40:13the code looking at I mean is it line by line? Is it just like an overview or using an agent
- 40:18to help you look at it? Yeah. So what I do is I have a PR skill which is I think in reference
- 40:25skills here which basically builds me a PR body that does a couple of things. It's this is a nice
- 40:31skill by the way because you can just install this and the agent will call it. It's unlike
- 40:35my other skills. has just sort of improved your PRs and it does a few things. It gives me like a
- 40:40sort of visual summary of the thing that's being built. It gives me evidence that it's working. So,
- 40:46it has to take like a screenshot or have like execution based evidence. And then it tells me
- 40:52the merge danger. What is the danger of me merging this? Whether it's a one-way door or a two-way
- 40:57door. So, if it's a one-way door, then it's a door we can't walk back through, right? we merge it and
- 41:02then there's data loss or like you know an email gets sent out to 100,000 people where if it's a
- 41:08two-way door then we can merge it and come back and then also it's blast radius too so whether
- 41:14um whether if it goes wrong it's going to be a big bad thing or a small thing and so I use
- 41:20this merge danger as basically like do I need to review this because if it's a small blast radius
- 41:25and it's a two-way door then I probably don't need to like review it that hard you know what I mean
- 41:29I can probably just like test it in production So, wherever if it's a one-way door with a big
- 41:34blast radius, you better believe I'm reviewing not necessarily line by line, but I'm looking at
- 41:39um I'm trying to break it. So, that's kind of what I do for human review. And I like human
- 41:45review because it's not only a chance to make sure the stuff I'm shipping is high quality, but I then
- 41:52take the stuff that I'm saying in human review and I pull that back into my coding standards into my
- 41:59uh environments that my agents in. Right? So I start thinking, how can I improve my environment
- 42:05so I never have to write this comment again? And that's what I'm thinking when I'm reviewing. Now
- 42:10is that like an automatic thing? So when you're so let's say you're reviewing it human review you're
- 42:16looking at the code what is your method for giving so I imagine you're not going to fix it yourself
- 42:21you'd probably just tell the agent to fix it so it has that context um is it automatic to take
- 42:26what you're saying and add that to your your your coding standards or do you have to tell it that
- 42:31to actually initiate the skill there is a skill for that so we have a retro skill so this looks
- 42:37back on a session looks back on a PR and suggests improvement ments based on what the agent did,
- 42:43right? So, it's another look back on the agent session, on your review comments, on everything
- 42:48that just happened. Let's improve it next time. M okay. Okay. And I'm sure we'll get to this
- 42:54because it's more of an architectural and design question, but like your coding standards, is that
- 43:00just something that you as a software engineer with that background just know what you want to
- 43:05have? And how would someone who doesn't have that background even go about doing that? Yeah. Um,
- 43:11this is this is where the strategic stuff comes in, right? Because this is I don't want this to
- 43:16happen next time. I'm thinking about the long view. And you need to be able to recognize what
- 43:20good code bases and bad code bases look like. And that's really hard. I like I'm I'm we want to dive
- 43:28into that now because we totally can. Um, let's wait a moment because I and I'm like I'm like yes,
- 43:35but also no because I know we we probably want to keep going through this the different skills
- 43:39we have as we're adding features because I I do have a question like once we once we review it
- 43:45and it looks good. Let's say it's a two-way door. Blast radius is small. It's it's an easy thing
- 43:50to deploy. Um, what's next? Do do you deploy it? Does your agent deploy it? What's your approach? I
- 43:58mean what's next is I press merge or I say okay if there's a bunch of them I just say okay guys merge
- 44:04all these five PRs or something and you figure out the right order you merge them in together
- 44:08and then we're kind of done right you know we've done the loop and now we're back to wanting to um
- 44:12ship some more features either do another grilling session or just like plain ship something. Um,
- 44:18so that is the loop. We've covered the entirety of the main flow where you grill with something,
- 44:22you either then to spec and tickets, implement it, code review, and then retro so that it's better
- 44:27next time. So I think that that's probably the the part where most people stop cuz anyone can
- 44:34go Matt PCO's a smart dude. He's got skills that will make my agent smarter. I'm just going to copy
- 44:40paste and go. But the next part we're going to talk about is the actual thing the thinking that
- 44:45needs us to understand architecture design and a little bit of coding. So walk me through that.
- 44:52Okay. So before we think about we like we've got the tactical stuff kind of done right. We now know
- 45:00how day-to-day we're going to be um shipping code. But how do you then take the long view and think
- 45:06how do I um improve this over time and get my agents to a point where I can trust them? And the
- 45:12point of this is that uh Lauren Tan um who goes by Potato, who works at Grockbot, who's uh kind of
- 45:19famous now for shipping like 2,500 PRs a month or something, she talks about how do you essentially
- 45:25scale up your trust in your agents? Because the more you can trust your agents, you sort of walk
- 45:32up this trust ladder. And if you don't really trust your agents, then you're never going to
- 45:37be able to scale beyond like four agents at once, you know, because you're [clears throat] going to
- 45:41be watching them everything they do, checking all of their work, human reviewing every part
- 45:45of it. And even if you walk through this whole flow, you're still going to be doing that, right?
- 45:49So her question is, if you can go up the trust ladder and you can trust maybe some PRs to be
- 45:56merged without human review, you know, the two-way doors, how are you going to get your agents to
- 46:01show you evidence that they've done their work? How are you going to constrain them so that they
- 46:06do better work every time? And if you've got essentially a really great environment for those
- 46:12agents to operate in, then you can start trusting them to levels where you can have hundreds of
- 46:16agents operating at the same time comfortably. and trust that they're doing good work. And so
- 46:22all of this is about trust. And what you need is a codebase that's a good codebase. And if you read
- 46:30like the classics that I've got on my shelf, they tend to converge around one definition of a good
- 46:35codebase, which is a codebase that's easy to make changes in. A codebase where if you make a change,
- 46:41it's probably not going to screw anything up. And that's really hard to do actually because
- 46:47or actually I don't know if it's necessarily hard to do but you have to care about that in
- 46:51order to make your codebase easy to change. And so what that means for me is that you need a ton of
- 46:59feedback loops so that the agent can grade its own work and can verify that what it has done
- 47:05is good. And you asked me before about effort. I would say that the point where I really want
- 47:11a high effort level is when it's verifying its own work. When it's checking that what it's done
- 47:16actually functions and it's clicking around a real application where it's running like performance
- 47:23traces on its code to make sure it hasn't degraded anything, that's where I want it because if I can
- 47:28trust that the agent has done that work and has verified its own stuff and that the codebase is
- 47:32easy to make changes in, then that is stuff I can just like merge without thinking about it. I can
- 47:37just go, "Oh yeah, that looks good. merge it. You know, I don't have to worry about the code.
- 47:42Are you there? Like, do you trust your agents right now? I'm in an environment where I can,
- 47:48right? I can absolutely because I'm not working on a medical application. I'm not
- 47:53working on a finance application. My domain is I'm building personal software for myself, right? So,
- 47:58I have been in those situations before. I've worked on, you know, banking finance applications.
- 48:02And in those situations, I would not or or I would be looking very very carefully at at certainly the
- 48:11one-way doors, right? I would probably be asking my agents to verify everything from a security
- 48:16point of view really heavily as well. I would be burning a lot of tokens on verification so that
- 48:21I could get to the point where I trusted them. [clears throat] So, what are some examples of
- 48:27feedback loops you have right now that build your trust in your agents? Great question. So there are
- 48:33a ton of them. And by the way, um I have this really cool resource. It's called the AI coding
- 48:38dictionary. I've not really talked about it very much. And it's essentially every term that I use
- 48:42is somewhere on this graph, right? So we have uh model providers, context windows, and all of them
- 48:49have a definition associated with them. We have uh somewhere in here will be automated checks.
- 48:57And so automated checks are essentially what I'm about to talk about which is strict typing um you
- 49:04know a fast test suite and a bunch of other stuff. So the main automated and this is this is free by
- 49:10the way you just you know access it there's a GitHub repo if you want to point your agents
- 49:12at it. Very useful for just like this is kind of like the glossery that I showed you earlier but
- 49:18applied to AI coding right this is where I got obsessed with it. So all of my resources use the
- 49:23same language. Now, automated checks essentially you want strong types. So, you want to be using
- 49:29things like TypeScript and not JavaScript because types tell your agent um very very quickly that
- 49:36it's made a mistake. Then you want tests. So, you want actual runtime tests that verify that
- 49:43it's doing what it thinks it should be doing at runtime and that you don't break anything.
- 49:48You want things like llinters as well that uh for those who don't know a llinter is essentially a
- 49:55program that checks your code to make sure that it looks right that you can do all sorts of things
- 50:00with it like check that you're using the right functions check that you're importing from the
- 50:03right modules that you're not using bad patterns that the authors of the framework know are bad and
- 50:09are trying to protect you against. So llinters are very important too and what you want is to
- 50:15get the most value out of those you can. So you want to use like types really effectively. This
- 50:21is why I have let's say a this little rule inside my coding standards here says don't
- 50:28use the any type in Typescript because I know as a developer that if you use the any type
- 50:34it's actually a cheat. So it's trying to cheat around the type system and it's going to reduce
- 50:38the quality of my type checking further down. So I am trying to improve those automated checks so
- 50:46that they work better and catch more errors next time. Interesting. Okay. Now these all these
- 50:53automated checks the llinter the um the CI the testing is this something that anyone can easily
- 51:00implement? What's the process for that? You just you just have to care, right? You just tell your
- 51:05agent about them. You say, "I want to constrain my application more. I want you to find harsh lint
- 51:11rules." And this is something that retro does as well. So what retro will do is it will say,
- 51:17"Okay, we we found like a comment in um the PR um and it tries to find ways that it can prevent
- 51:24it deterministically with an automated check next time." So every time it sort of ratchets
- 51:29up the power of these automated checks every time you run retro or find something. And so
- 51:35eventually you get to the point where the agent is so boxed in. They're in their straight jacket,
- 51:40right? They literally cannot make a mistake. And what you worry about then is of course improving
- 51:46the speed of those checks to make sure that you can still ship work. But agents are really good
- 51:49at that as well. So all of these individual parts agents are good at. They're good at finding lint
- 51:54rules to constrain themselves. They're good at um making those tests run really quickly. It's
- 52:00just you have to care. You have to tell them to do that. And if they if they don't know to do it,
- 52:05then they won't care about it either. Is that a skill you have? I'm thinking about building
- 52:11a skill in this direction. Actually, I do have a skill um which we can talk about in a minute
- 52:17that sort of prompts the agent to improve its own environment a bit. But essentially, this is
- 52:23retro. That's what retro does is the more often you run retro, the more sessions you run it on,
- 52:28um, the better your codebase is going to be. Okay. Okay. So, let's say let's say we've got the
- 52:37average Joe. They have an idea for an application. Let's say it's a website that, I don't know,
- 52:41makes ordering coffee and tea better and easier. You can find the cheapest tea and coffee in the
- 52:45world. Doesn't matter what the idea is. How would they approach that idea? So, obviously,
- 52:51Cloud Code is your favorite. I think it's it's very good too. It's one of the best harnesses.
- 52:55So cloud code. Um would you say they need something like VS Code or can they just stay in
- 52:59the terminal? What's your what's your preference on that? I would recommend VS Code. It's nice to
- 53:05be able to easily access the code. I don't always use it to like read code or think about code.
- 53:10I would say that what what my stack is and it's very approachable like it's free is GitHub is VS
- 53:18Code is Git obviously and Claude code or whatever agent you like it doesn't really matter what the
- 53:24agent is to be honest as long as it can use skills and it can use um uh my skills u then you're good
- 53:30to go and pretty much every harness out there is you know can use skills because they're a common
- 53:37standard. Okay. So, doesn't matter what the what the harness is really. Um, and I think most of
- 53:42them use sub agents. Like everything's kind of the same now. They've all copied each other. Uh,
- 53:46so once they have that set up, what are the things they have? So, when we get to the
- 53:52strategic thinking, when they're building their app, obviously they'll they'll start talking with
- 53:56their AI. Do you have skills that ensure that they do that? Is that what the grill me is for? Like,
- 54:02is that does that ensure they have that strategic thinking they need, or is there something more
- 54:06they need to know? So it is baked into different parts of the process. So let's say when you call
- 54:12to spec, it's going to ask you sort of what shape of modules you want, which might be a little too
- 54:18technical for some people, but just sort of keeps you in the code a little bit. I don't necessarily
- 54:23think you need like a like full awareness all the time of everything that's happening,
- 54:29right? You can people talk about like software factories and they talk about a dark factory,
- 54:35right? which is kind of like, you know, you have your factory, your agents are working in there,
- 54:38and you never turn the lights on, right? You have no idea what's going on. I don't like the
- 54:41dark factory. I do think you need to look in there and care, but you can have little dimmer switches,
- 54:47right, for different rooms and stuff. There are rooms in your codebase that you never really need
- 54:51to walk into, right? You can just sort of have a little peek in there. And I think one thing we
- 54:57should talk about before we like go here is that there is another skill that I think really helps.
- 55:03So, I sort of talked earlier about um wanting to get the most out of your automated checks.
- 55:09Mhm. And [clears throat] when you one thing that I think people really underestimate is that when you
- 55:15write tests for your application, so specifically the things that test your application at runtime
- 55:21that actually run parts of your code and like verify that they work. Tests are really hard.
- 55:26They're really hard to get right. And this is this has been true, you know, before AI.
- 55:31The reason tests are hard is it's really easy to write a test that really doesn't verify very
- 55:36much. It can like, you know, you look at like a line of code and it just says um the the amount
- 55:45of characters you can have in a tweet is 240 or something. I can't remember what the actual number
- 55:49is, but let's say 240. And then you'll have a test that just checks that number and checks that that
- 55:56number is 240. Right? The test is just repeating the implementation. It's not actually like what
- 56:02you want from your tests is you want a big chunk of code that I'm going to call a module and you
- 56:07want that module to basically be hidden from the tests. The stuff inside the module is hidden from
- 56:12the tests because that means it can just test it via a very narrow little um sort of entry point
- 56:19to that module. And it means that you can change everything inside that module without needing to
- 56:24alter the tests. [snorts] Because if I like have a single line of code that's tied to a single test,
- 56:29then every time I change that line of code or I restructure it or do something with it,
- 56:33then that test will have to change as well. And it's just sort of endless crappy churn.
- 56:37Whereas if I have these big what I call deep, what what John Asterout calls deep modules,
- 56:42then you can actually sort of turn the lights off inside that module. You don't have to care
- 56:47about what's in there. As long as the tests happen from outside and everything sort of seems to work,
- 56:52then it's probably going to be fine. And I have a skill that can basically turn your codebase
- 56:59into this shape. It's called improve codebase architecture. So you can fire this off at any
- 57:05repo and it will give you a visual report on the bad stuff that it finds and it will suggest
- 57:13improvements for making it better and then you can take the result of that and turn that into a spare
- 57:18tickets and run it through the normal loop. Ah, so using this skill, it will it obviously
- 57:23will show you the the holes and the errors and all the bad things inside your your codebase,
- 57:27but it'll also help you build those those tests. And are we talking about unit tests? Is that the
- 57:32same thing? Yeah, unit tests. Exactly. Okay. Which I I as I was learning coding before AI,
- 57:37I never got to that point, but I heard it was always the worst part of coding,
- 57:41just creating really good tests. Totally. It's really really hard. But agents agents
- 57:46make it easy to type out tests, but they don't necessarily think about structuring the codebase
- 57:51so that it's easy to change, right? So that those tests actually catch things that are important.
- 57:57It's very easy to write tiny tiny little tests when what you need are tests that actually like
- 58:03take out huge chunks of the application and put it all under test at once. So that's what this skill
- 58:08does. Okay. Okay. And that I think that would be a massive help for a lot of people. I mean,
- 58:14I'm going to run this skill on a few of my code bases right after this interview just to see what
- 58:18happens. Um, okay. So, we we've we can produce good tests and that's a good place for us to go to
- 58:26um to make sure our code base is is healthy and and it's um it's and you said it was we could be
- 58:33able to dim the the lights or turn lights off on a module as long as the tests are really good,
- 58:38right? Yep, that's right. you have like this feeling of um I don't know when when
- 58:45you work on a codebase that's been vibe coded I always feel this terror right this I don't know
- 58:51what's in there all right and I think it's bad and what you start to feel is right how will I
- 59:00know if something goes wrong right will I have to eventually like look at this code and review this
- 59:03code like how am I going to get to that whereas if you have this if you have improved codebase
- 59:09architecture and your codebase starting to be put into these big beautiful modules. You can go,
- 59:14okay, I understand the shape of this and my app understands how to use these modules as well,
- 59:19how to like um leverage them and how to call their functions and things like this. And you have these
- 59:25lovely big chunks of stuff that you don't need to review that's all using your domain language,
- 59:30all using your glossery terms and stuff. So, you know exactly what's going on in there. and you
- 59:34realize, okay, I can start thinking of my app's codebase in these big chunks, right? In like,
- 59:41okay, that's the bit where we do authentication, that's the bit where we do um transcripts,
- 59:46that's the bit where like the whole video editor interaction lives in one big chunk and it's all
- 59:51testable. So, when once you have that, you start realizing, okay, cool. I don't need to worry that
- 59:56much about this codebase because it's structured in a way where it's more robust. Hey, real quick,
- 1:00:01let's take a coffee break, you and me. [snorts] This coffee break is sponsored by Coveron. Look
- 1:00:06at Zach. We talk a lot about AI on this channel. I mean, everyone's talking a lot about AI. And
- 1:00:11while AI has made you pretty good at your job, you know who else has made pretty good at their
- 1:00:14jobs? Scammers. They can clone a voice, write a perfect fishing email, spin up a fake fake
- 1:00:19website in minutes, actually maybe seconds. And they can do that at scale. And they're not
- 1:00:23just coming after you. They're coming after your family, your parents, your kids, everybody. And
- 1:00:26that's where Coveron comes in. Their family plan watches the dark web for you because that's where
- 1:00:30all that stuff ends up. So, they're watching the dark web and they will alert you when you
- 1:00:34or anyone in your family has like a new account set up. And if something does slip through, you
- 1:00:38got real people you can contact for help. They'll help you clean it up. Plus, up to $2 million for
- 1:00:43identity theft and up to 100 grand for online fraud, cyber extortion, and home title fraud. One
- 1:00:48scam can cost you everything and they're moving fast. Protect yourself right now. Get 71% off this
- 1:00:54Black Friday deal. Coveron.com/chuckoveron. That's coveron.com/chuckcoveron.
- 1:01:01Okay. So, do you think we're at a place now where we can get to the how do you structure
- 1:01:05your codebase? How do you structure your project? Yes. I think like there's no one like genius way
- 1:01:15to do it. Essentially, you use these deep modules. You if you have I what I I really want to actually
- 1:01:23just run this and um because it's very very easy to run, right? because we can just literally go
- 1:01:28claude and then I'm just going to run improve codebase architecture. And what this does is it's
- 1:01:34going to um pull in the codebase design skill which sort of um if you want to just read that
- 1:01:39that's got all of my uh like how I like to design code bases in there and it's finding the hotspots.
- 1:01:45So it's going to find my recent commits and look inside the repo to uh explore that and explore
- 1:01:51what might need to be improved. And then we're gonna get a HTML report back of the things that
- 1:01:57it wants to change. So once this is done, I'll be able to like exactly tell you how to do it. There
- 1:02:04is I think what people need to get is there's no one perfect codebase, right? All you can do is
- 1:02:10just improve it over time and get it better over time. And what I do is I have a cron job that runs
- 1:02:16on each of my repos where once a day it will run improve codebase architecture and it will give
- 1:02:21me a top suggestion back and then it will go and make a pull request based on improving that thing.
- 1:02:27And because those are two-way doors then I don't really need to worry about it. I just know that
- 1:02:31it's making decent improvements and slowly pulling the code into where it needs to be. And so you're
- 1:02:37essentially just trying to fight back against the wave of slop, right? because agents will
- 1:02:42make tactical mistakes and strategic mistakes and you just need to constantly be pushing it to this
- 1:02:47position where they use these deep modules. Uh when you have these cron jobs running,
- 1:02:55do you trust it enough just to fix things without your approval? Yeah, I what I do is I do sampling,
- 1:03:03right? I will just sort of check in every five PRs or so. And this this works for my application
- 1:03:08because it's again not finance, not medical. So I will do sampling. I will just sort of check
- 1:03:14that the thing is running. I will occasionally um retro it. So just say retro look at the last five
- 1:03:20of these and see if they're actually reasonable. See if they make sense. See how we can improve
- 1:03:23them next time. And so yeah, basically I do. Hm. Interesting. Now do you do you use like the
- 1:03:31Cloud Code app on your phone? Do you do you do things remotely? Do you text it? Use Telegram?
- 1:03:36What what's your stack for that? So, I have been like in all sorts of like I've tried all sorts
- 1:03:43of setups basically. I've like my recent one is I liter I literally tore it down today which I built
- 1:03:48a uh a custom Discord bot for myself. Kind of like my own Hermes basically. I put it up in the cloud.
- 1:03:53I was like texting from my phone and stuff. I think phones are cigarettes basically. Phones are
- 1:03:57poison. Um so I tried to make my phone as boring as possible. So I'd sort of go away from that. I
- 1:04:01didn't want to do it anymore. So right now, as of now, I'm just using claude code and using um
- 1:04:06GitHub actions for some scheduled jobs as well. So this is an open source repo. GitHub actions allow
- 1:04:12you to run cron jobs and I just run claude inside the GitHub action and it updates and creates pull
- 1:04:18requests for me. And agents know GitHub actions really well. It's actually free for open source
- 1:04:24repositories and so it's a really good option for me. Okay. I I love what you said that phones are
- 1:04:30poison. And what's funny is I I um I have it to where I can use iMessage for my agent right
- 1:04:36now and it's I was just thinking this morning as I was on my walk. I'm like this is so fun
- 1:04:41just being able to text my agent to do whatever I wanted to do and it it does feel freeing. So I I
- 1:04:46find it interesting that you're like the opposite. You're like no phones are poison. I want to make
- 1:04:50my phone as boring as possible. Why? Well, because I had that too, right? I would have Whisper Flow
- 1:04:57running on my phone. I would be chatting to it on Discord as I sort of went, you know, for my
- 1:05:01walk and stuff. Um, but this is more, this is just a personal thing really. Like I think we'll look
- 1:05:07back on this generation and think of it in the way that we thought about cigarettes in the 50s, you
- 1:05:11know, like you know, we're giving, you know, like phones to 13-year-old kids and stuff, you know,
- 1:05:15like it's it's miserable. Like these kids like I it's just, you know, life is there to be lived,
- 1:05:24you I I I I just I don't having this constant access to information and uh especially the social
- 1:05:32side of it as well and this constant hankering of people wanting to pull you in and scroll basically
- 1:05:39just scroll scroll scroll scroll a bit more. I just hate it. I'm with you there. And I I I
- 1:05:45I like the idea of keeping my agents close to me, being able to text them. But like I said,
- 1:05:50like just this past weekend, I deleted Twitter. I deleted or X I deleted Tik Tok and YouTube and
- 1:05:55everything off my phone because it was sucking that that time away and it's and it's so funny
- 1:06:00when you take that stuff away you notice you kind of have that itch like I'm bored for these
- 1:06:053 seconds I have to wait let me go check no I don't I can't I don't have it and you you feel
- 1:06:09that urge start to like it's and it's nasty. I've even gone further which is I've uh removed the web
- 1:06:14browser and I've removed the email from my phone as well which is apparently something you can do
- 1:06:19on Android and so it is as boring as possible. And what I noticed was I would hope that or I
- 1:06:25hoped that when I did that I would get all this extra inspiration. I'd feel so much more energy.
- 1:06:30I would like become this whole new person and then nothing happened. Right? Essentially
- 1:06:36nothing h nothing rushed in to fill that gap. You know what I mean? It turns out that yeah,
- 1:06:40these things are really addictive and you you know I've been trying to read more books, I guess,
- 1:06:46but it nothing is as good or feels as good as checking Tik Tok, right? Or checking Twitter. So,
- 1:06:54you just have to kind of live with that. [laughter] It's like a you're you're talking like
- 1:06:59a former addict or something like I'm always a drug addict and there's nothing better than that,
- 1:07:02but it's ruining my life. [laughter] Absolutely. Uh so, what is this? Well, this is the report
- 1:07:08that we just got back from uh improved codebase architecture. So, what we're essentially seeing
- 1:07:14here is a before and after of each of the um modules. So, what we have is a problem. Submit and
- 1:07:22duplicate course contain 150 near identical lines. So, there's some duplicated code inside our repo.
- 1:07:29And what it's basically proposing is instead of this horrible sort of setup where we have a bunch
- 1:07:34of shallow thin modules on the left hand side, we say, okay, all of these different parts of
- 1:07:39our application are all going to rely on a single deep module and we can then test that module. And
- 1:07:45so this one feels like a like a total win for me. So this is cool, but honestly I think and I speak
- 1:07:53for myself as well as probably my audience and many people watching. I'm like, um, I see that I
- 1:07:59found stuff and it's very interesting, but also I probably have to ask AI about a lot of these
- 1:08:03things. Like, what does that mean? What does that mean? Why did you show me that? What is this? No,
- 1:08:07absolutely. Absolutely. So, this kind of maybe goes into another section of our our conversation,
- 1:08:13but um I'll ask it now and maybe you can tell me where we should ask or answer it. But for
- 1:08:20people who are wanting to get into this world and what it means to be a software developer,
- 1:08:25what is that knowledge that you still need to learn? What's the thing? Because obviously it's
- 1:08:29not it's not tactical coding anymore, but I look at this and I'm like, I don't know this, so I
- 1:08:34probably should learn this. What What is it? So, there's a skill for that. Um, [laughter] so, uh,
- 1:08:42do you have a t-shirt with that? Like, that should be a t-shirt. [laughter] I have a teach skill,
- 1:08:46right? I'm a teacher. I was bored on the tube one day and I like I thought, why don't I just
- 1:08:51write down everything that I know about teaching um, and turn that into a skill. And what I would
- 1:08:56say is that let's say you have improved codebase architecture. You get a report and you look at
- 1:09:00the report and you think, I can't read that. I don't understand what's going on there. I don't
- 1:09:04understand the terms it's using. Um, I've thought very carefully about the terms that I've I've used
- 1:09:09in the skill so that they point out to um books and they point out to well-known methodologies.
- 1:09:15For instance, let me just I've got a big pile of books next to me. This is one of the books that
- 1:09:20I recommend called uh working effectively with legacy code by Michael Feathers. Um, wonderful
- 1:09:26wonderful book. Teaches a lot of this stuff. And what teach can do is it you basically just say you
- 1:09:32pass it the report you've just received. you say, "I don't understand this. Teach me about this."
- 1:09:36And it goes and finds a like a workspace on its own. It keeps track of what you know and then it
- 1:09:43quizzes you and builds these again HTML lessons so that you learn things slowly. So it's essentially
- 1:09:49a build your own course for uh whatever you want. So it taught me how to um do a Rubik's cube
- 1:09:56actually. So this was how I tested it and uh I can now do a Rubik's cube which is wonderful. Um,
- 1:10:02but you can also I've tried it out by teaching strategic programming as well and it's really
- 1:10:06really good at it because my um these reviews here they're basically like they use a common language.
- 1:10:13So they talk about locality. So you probably have no idea what locality is. A lot of software
- 1:10:18developers don't know what locality is. It means how if I make a change to a piece of code, how
- 1:10:25likely am I that all of the changes are going to be in the same location? In other words, are all
- 1:10:29of the related pieces of code together in one file or in a couple of files? So, locality is good. You
- 1:10:35want locality? And yeah, like you just basically give it this and you say, I want to understand
- 1:10:43this. Teach me about it. And then it's going to do a great job. So, you're saying, which by the way,
- 1:10:48that teach me skill is such a fun idea. I love that. So, you're saying that it's important for
- 1:10:54us to know all this. um at what point is it not important for us to know this? Like are the models
- 1:11:01going to get so good to where it doesn't matter? So it comes down to caring again, right? And all
- 1:11:14you need to know is you need to know enough strategic programming to be dangerous and to
- 1:11:18know that it's effective and that improves things over time, right? because the models are still not
- 1:11:24going to care about it, right? I maybe there's a world in which they are um prompted to do this.
- 1:11:31And I I've been thinking about this myself as well, but for now it seems to me that people who
- 1:11:37are really good at something are going to be able to get models to be really good at that thing,
- 1:11:43right? M so engineers tend to produce better um products than non-engineers right now because
- 1:11:51they have the domain expertise they know what to ask for they have the right language right um you
- 1:11:58know you might say uh what's a good example let's say that you have a problem in your application
- 1:12:05where the you have some sort of flashing stuff coming in when you load the app right it sort
- 1:12:10of shifts around in terms of layouts right and I know what would cause that. I would say um fix the
- 1:12:18hydration error on the homepage, right? Or fix the uh layout shift on the homepage. Whereas someone
- 1:12:25else might need to say, okay, the when I load the layout page or the homepage, um it loads and then
- 1:12:31it shifts everything around. It just allows me to communicate more effectively and prompt the agent
- 1:12:36in a better way because I know the domain language because I'm an expert in it and I've seen that
- 1:12:41kind of thing before and I can communicate what I want much much better. H that does make sense.
- 1:12:49Um and I'm sure I'm sure you've seen this term uh the bitter pill engineering idea. Uh bitter
- 1:12:56lesson or bitter lesson. Yeah, I I forget it's it's then been used interchangeably, but the idea
- 1:13:01that all this all the um architecture, all the the the straight jacketing we're doing um will
- 1:13:08be replaced by better, smarter models and we just need to kind of get out of their way because they
- 1:13:13might know better than us how to architecture a given thing. But you're saying that it's it's the
- 1:13:17caring that matters the most. It's the caring, right? And I always say like I'm not a pundit,
- 1:13:23right? I have no idea where things are going. you know, I'm just in ox like rural Oxfordshire. I'm
- 1:13:28not in Silicon Valley. I don't know anything. Um I'm my job is not to predict the future. My
- 1:13:33job is to tell you how to get these things working now. And they're very effective now. Um but yeah,
- 1:13:40I and I think there probably is some sort of harness, some way of doing this that will make the
- 1:13:47agents care, right? Because you can get the agents to care. It's just you need to care yourself. So,
- 1:13:53that's something I've been thinking about and working on too is how do you get agents
- 1:13:56to think strategically? And I think that probably is something you can do in the harness, just not
- 1:14:02um it doesn't have to be in the model. But for now, this is what I consider the state-of-the-art
- 1:14:08way to do this. Okay. [clears throat] And then once you've I think what a lot of people don't
- 1:14:16understand how to do right now is they don't understand the architecture or rather the
- 1:14:22infrastructure like where to actually put this app they're building um do they use a database
- 1:14:28what type of database um do they put it on AWS or Google cloud like all these things are big things
- 1:14:35that normally are done by DevOps engineers or or software engineers who have that background.
- 1:14:40Now, how should someone approach that? Especially if it's someone who like only has done vibe
- 1:14:46coding. Let's say they they're watching your course and they're coming into it. Do they
- 1:14:50need to learn all that stuff or can they just ask the agent to do it? And do you have skills
- 1:14:54that help with that? I don't have skills that help with that. And I would say we don't really
- 1:14:59need skills to help with that. What's a little bit of a history lesson which is over the last 10 or
- 1:15:0420 years DevOps has moved from need like being something that you needed a degree in to like
- 1:15:12figure out how to do you know like uh hosting uh self-hosting your own Linux box or something and
- 1:15:17doing really complicated stuff or stuff that could really go wrong and really bite you into a set of
- 1:15:22products that have been built by Google cloud by AWS um especially you know serverless platforms
- 1:15:27if you want to do some googling that really take a lot of that stress away from you and has resulted
- 1:15:33in products like Versel for instance who I used to work for and um Netlefi and all this stuff that
- 1:15:40basically take all the pain away from you right and what you really need to do is just figure
- 1:15:46out where your data is going to live whether it's going to be in a database you should probably just
- 1:15:51use like a Postgress database if you don't know what you're doing and once you've got that and
- 1:15:55there are many services that will host and manage your Postgress database for you planet scale
- 1:16:00Superbase. Once you've got your data, you can basically deploy your app anywhere. You know,
- 1:16:06just Netleify, Versel, whatever you want, and it'll be fine. You know, like this has
- 1:16:12gotten so much easier that an agent really can just do it for you and it will be good. So,
- 1:16:18as long as you're running some security checks and you're doing relatively regular security reviews,
- 1:16:23which agents are also pretty good at, you'll be fine. It's a lot easier, just like coding. It's
- 1:16:29a lot easier than most people think it is. Yeah. And I can speak to that. Like I I'm very much a
- 1:16:36home labber. I've got Proxmox and all kinds of things in my environment. But as soon as
- 1:16:41I felt comfortable enough to give the keys over to my agent, I can simply text him and go, "Hey,
- 1:16:45I want to deploy this app, make a new VM, make it happen, just do all the things. Give it an IP
- 1:16:50address, give it a domain name, and boom, it just does it." And it's magic. It feels like magic. But
- 1:16:56I did need to have a bit of that domain expertise to even know how to tell it to do that and to give
- 1:17:02it those keys. Um, but you're saying that a lot of the DevOps things um, so things like databases
- 1:17:10and and and Versell hosting now they've abstracted that away to where you don't really have to know
- 1:17:14it. Your agent can do it. Do you is that where you host your stuff? Is it is it Versel? Yeah,
- 1:17:19I'm on Versell. Yeah. M like I would say just use the thing that's easiest to use basically
- 1:17:27especially when you're getting started. I wouldn't worry about you know like going into AWS manually
- 1:17:32or figuring out all that stuff. There's stuff you can like there's foot guns there right which is
- 1:17:36why these companies like Versel exist. So like they just sort of make that easier and you don't
- 1:17:42have to worry about you know manually deploying or anything. You just push to your GitHub repo and it
- 1:17:46deploys for you. Um it really is very very simple. Um, you can even use something like GitHub pages
- 1:17:51or something. And all of this stuff is even when it comes to like domains and um, uh, DNS records
- 1:17:58and things like that. All of that is really easy for agents to figure out information on. And often
- 1:18:03um, these sites like, you know, Cloudflare as well. Um, you know, again, it's just something you
- 1:18:09have to care about. You just basically say to your agent, make this deploy as good as possible. Um,
- 1:18:14give me everything I might need in terms of security. um uh distributed denial of service
- 1:18:19attacks, DOS attacks. Um try to make it secure and just give me the basic level of what I need.
- 1:18:24And agents are really good at that. Time for a quick coffee break. I want to talk to you about
- 1:18:27something. You've probably seen one of these before. A video of a celebrity or a CEO giving
- 1:18:31away some kind of scammy thing. Actually, I just saw that with the Roadster announcement. You know,
- 1:18:36Elon's flying car. I was trying to find the live stream and I actually just found
- 1:18:40a scammer live stream that had a fake Elon sitting there talking. To me, it looked fake,
- 1:18:45but for many people, it would not. And there were like 30,000 people watching that stream,
- 1:18:48and he was pumping some crypto scam, but it looks stinking real. Voice, face, everything. Like,
- 1:18:53I was showing it to my family. They're like, "Oh, are you sure? Are you sure it's not him?" It's
- 1:18:57a deep fake. Scammers are literally using them all the time. And in a survey of 7,000 people,
- 1:19:01that's their biggest concern. Deep fake scams are so bad now, and one in seven have claimed they've
- 1:19:06fallen for one. So, here are some red flags you should look for. when the lips are a little bit
- 1:19:09off from the words. They have some weird blinking and an offer that's like too urgent or too true.
- 1:19:14I mean, come on. Most of these announcements we see from founders are like pre-order in 3
- 1:19:18months. They're not like buy this right now. So, when you see that, pause, don't click,
- 1:19:21don't send any money, don't share it, and then check it cuz Bit Defender has a free app called
- 1:19:27Real Check. You can drop in a link from Tik Tok, YouTube, Instagram, Facebook X, upload the clip,
- 1:19:32and it tells you if the video looks altered and how likely it is to be a scam. And honestly,
- 1:19:36not every AI video is a scam. Real Check will help you discern which ones are. Now, you may
- 1:19:41be thinking, "Well, Chuck, I don't get fooled by these scams." Well, what about your mom and your
- 1:19:45grandma and your dad and your brother and your sister? Do they know enough to spot these things?
- 1:19:48Give them a tool that will help them do it. Run suspicious clips through Bit Defender Realcheck to
- 1:19:52see if it's a deep fake meant to scam you before you click or send money or share it. Links in the
- 1:19:57description. Bitdefend.me/realchecknc. That's bitdefend.me/realchecknc.
- 1:20:04And it's free. So like what do you have to lose? Just try it. What do you do for security? Um
- 1:20:10because I know that that's one thing everyone actually most people don't think about but
- 1:20:14they should is all these AI agents are just making things very quickly. They want to make it work and
- 1:20:19that's the primary objective. But then security is a whole realm. Do you have skills for security?
- 1:20:23No. I'm the wrong person to ask because um no the project the projects I work on are personal
- 1:20:28software essentially and I have a platform team that handles my site where we might have some
- 1:20:32security concerns. Uh so no I don't don't think about it that much but I know it's a concern.
- 1:20:37What I would recommend is if you are interested in security is to add something to code review so
- 1:20:43that it does instead of just um to access instead of just thinking about standards and spec you also
- 1:20:50do a security review and many harnesses have a built-in security review um skill. So that's what
- 1:20:56I recommend. H have you had any security issues with anything you've coded specifically with? No,
- 1:21:06but I haven't had those cons. I I I've definitely screwed up in many many ways, but I don't think
- 1:21:13I've ever uh especially preai, but I don't think I've ever caused a security issue. I don't think.
- 1:21:22That's good. That's good. We can trust you now. No, I'm just kidding. Um, now with your with your
- 1:21:30skills, how do you make sure what what's your method for making sure the skills are good?
- 1:21:36Because, you know, anyone can make a skill, but how do you test them? How do you make sure they're
- 1:21:41they're effective? Yeah. A lot of people ask me this and especially how do you like eval your
- 1:21:46skills, right? So for those folks who never heard of evals, they're basically benchmarks that you
- 1:21:50run against some system by pushing a bunch of data through the system and then seeing what happens.
- 1:21:56So you might eval coding agent by saying okay, is it better with this model or this model by running
- 1:22:02like um a benchmark of something? You you get it to do the same task every time and you see if it
- 1:22:08um succeeds or fails. So that could be a 0% score or 100% score on the eval. Now skills
- 1:22:15are incredibly hard to eval incredibly hard and incredibly expensive to eval. How would I eval
- 1:22:22my like grill with docs or something? You know, like how would you eval u domain modeling? How
- 1:22:28do I know if I've chosen the right term? A lot of these skills are just sort of augmenting what
- 1:22:32the model does anyway and what the harness does anyway. Trying to kind of get out of the harness's
- 1:22:37way and just give it basically trying to get it to care about different things. How do you eval that?
- 1:22:43And if essentially I have the fortune of having a ton of data going through my skills anyway,
- 1:22:51right? Because I have, you know, you know, millions of users all around the world who are
- 1:22:55using the skills and reporting issues back. I've got very active Discord where people talk about
- 1:22:59skills all the time. So I'm sort of running evals except my evals are people, you know, they're just
- 1:23:04So what I do is I listen to them and I react and I try to improve my skills all the time.
- 1:23:10Um, but I don't burn tokens trying to build evals that might or might not give me good data. So
- 1:23:18that's the decision I've chosen. If I was in a model lab and I was being paid to do this
- 1:23:22and I had unlimited tokens, I would build evals. Okay. And which I think even those eval wouldn't
- 1:23:30be as helpful as what you're experiencing now with people. Exactly. if I didn't like I'm very
- 1:23:35grateful to my users because they do report stuff really really effectively and that's the actual
- 1:23:40stuff I need right if I try to synthetically create that with evals it would be worse so
- 1:23:46when they do give you feedback how do you go about implementing that how do you verify it how do you
- 1:23:51test it uh I have an agent that um looks at it and looks at the issue it tries to recreate that issue
- 1:23:58so it runs it in like a dummy repo and then it says yes this seems good and then and it creates
- 1:24:03a PR. So I have that on a schedule running um every day I think because I get quite high volume
- 1:24:09of issues. Um but I also get a ton of crap issues as well. So that agent also filter stuff out. Uh
- 1:24:15did we ever talk about like what happens after the PR like do you deploy the changes yourself to your
- 1:24:21infrastructure or do you let the agent do it? Um I use a continuous integration platform. So on let's
- 1:24:27say Versel for instance, it will just whenever GitHub detects a change to the default branch, it
- 1:24:34will just ping off to Versel and then Versel will redeploy my application. We'll build it. We'll put
- 1:24:40it in the cloud. So that's something that's been solved for yonsks right before agents. And so it's
- 1:24:45just not something I think about. And it's and it's always deterministic. So it always happens
- 1:24:50the same way every time. And I don't burn any tokens to do it, right? It just does it by itself.
- 1:24:56That's yeah and I didn't think about that like yeah CI/CD has already largely solved
- 1:25:00that problem. I think so many people don't even think about CI/CD. They just think maybe
- 1:25:05uh they think about just having their agents log in and deploy the code itself. Um so you would say
- 1:25:10that people should think about CI/CD and implement that. Oh yeah. I mean if you have a choice between
- 1:25:14determinism and non-determinism, why would you ever pick non-determinism, right? Why would you
- 1:25:20do like why bother? Because determinism works the same way every time, right? your agents
- 1:25:25can always go in and fix if things go wrong. It costs you no tokens and it just costs you CPU
- 1:25:30cycles or something, you know, just costs you this resource which is incredibly cheap. So, always
- 1:25:34pick determinism. And this is this is true too when you're doing like a retro or something when
- 1:25:39you think, should I change this into like a rule where my agent is going to code review against it
- 1:25:44or should I change it into a lint rule that is deterministic and runs the same way every time.
- 1:25:50always always choose determinism when you have a choice and most people don't realize they have
- 1:25:55that choice. Yeah, I I don't think they do. And and even just me recently, I just started doing
- 1:26:01some CI/CD. I I um host I self-host Git Git T or Gideia, I'm not even sure how to say it. Um and I
- 1:26:08I use some of their CI and action functions, which is it's great. Um, now for you right now, I take
- 1:26:16it you mainly probably do mostly teaching, but you do also do some of your own development. Do
- 1:26:22you have any other projects that you're constantly working on that you're maybe going to sell or make
- 1:26:26public, make it a SAS? Nah, I'm not a big SAS guy. Basically, I'm like I'm in this teaching
- 1:26:31thing as like a lifestyle choice basically. Um, I have a young son and I want to spend as much
- 1:26:36time with him as possible. Um, being at home here, like getting to choose my own projects,
- 1:26:42choose what I care about and choose my own time is incredibly important to me. As as you can imagine,
- 1:26:48I've been approached by a fair few organizations saying, "We would love to hire you and come work
- 1:26:52for us." But I'm basically unhirable at the moment. like until my like uh my son and I've
- 1:26:58got another one coming uh go to school then like and I have a maybe a bit more time you know until
- 1:27:05that happens I'm going to stick exactly where I am so I'm I'm not doing the SAS thing but I am a bit
- 1:27:10of an open source boy so I have a project called S castle which is a kind of effort to um do agent
- 1:27:18orchestration in Typescript using essentially allows you to drop in any agent and run them
- 1:27:25in sandboxes. So like in Docker sandboxes or uh Podman or Versell or actually whatever you want.
- 1:27:31And then I've got another one which is sort of falling a bit by the wayside at the moment which
- 1:27:35is called Everlight. And Everite is actually an eval framework amazingly. So I I worked on
- 1:27:40this like a year and a bit ago I think. Yeah, it's really um fallen by the wayside to be honest. But
- 1:27:46essentially it's like a a visual eval framework that you can run locally and so yeah it's um it's
- 1:27:52open source for me and kind of that's that's where the bulk of my work is going and also into this
- 1:27:59video editor that I've been building and that I use for everything. I'm fascinated by that because
- 1:28:04I also build lots of video editing workflows for myself, you know, because we make videos. Um,
- 1:28:10we should chat about that in a separate discussion because I'm sure we can go get very nerdy very
- 1:28:14quickly and not many people are going to care about it. Oh yeah. Um, now with your open source,
- 1:28:18I love that you called yourself an open source boy. And I I think for many people, they would
- 1:28:23love to not only start their own open source projects, but contribute to open source, but they
- 1:28:27don't know how. And they never could until really now. But how do you do that? Well, like could they
- 1:28:34take your skills and start to contribute or even start their own open source project and what would
- 1:28:39they need to like what would they need to be aware of that isn't already like what we haven't already
- 1:28:44discussed? Yeah, I mean what I'd say is that this was a different conversation about 18 months ago
- 1:28:51than it is now because now we are living in a world where it's incredibly easy for anyone to
- 1:28:58just ping up a slot PR to a repo, right? to just say I want this change made here's a slot PR and
- 1:29:05very popular projects are now closing their PRs to new contributors so for instance uh if we look at
- 1:29:14uh HonoJS I think they just did this the other day um Hono is one of the most popular kind of
- 1:29:20like server frameworks in JavaScript and you can no longer create PRs against Hono unless you're
- 1:29:26a collaborator right because of just the level and I've closed uh PRs on my skills for instance
- 1:29:31as well because I want my skills to be very carefully curated and it's a rather popular
- 1:29:36repo. It's nearly up to 300,000 stars now and that is, you know, people just pinging crap
- 1:29:40into it all the time and I just didn't want to be notified about it. So, open source now
- 1:29:46is all about providing issues and providing good evidence, right? Because it's so cheap now to turn
- 1:29:52those issues into pull requests and to just get an agent to work through them and to figure it out.
- 1:29:57So the higher quality uh repros that you can do is great and like that will teach you a lot about how
- 1:30:05to communicate to people working open source and how to communicate to other developers and how to
- 1:30:09communicate to AI. But of course you can build your own thing, right? And the cool thing about
- 1:30:16having all of this code available to you is that your agent can read it. You know your agent can
- 1:30:21just like take a look inside this codebase. You can say for slashte me about the Hono codebase.
- 1:30:26What can I learn based on what I've told you about my experience already? What is cool
- 1:30:31about this codebase? What are they getting right strategically? How are they deploying
- 1:30:34their infrastructure? What infrastructure do they have? What do they need? You know, and all of this
- 1:30:39stuff, you should just look at these cool projects and just point your agent at them and say, "Teach
- 1:30:43me about this." And then like again, open source is a lot um it's a lot simpler than people think
- 1:30:49it is, you know? It's really just like deploying something to uh npm or something, you know. So,
- 1:30:56npm you can just use uh like a um a change set manager to just sort of do versioning and things
- 1:31:04like this. Very very easy to learn, very simple to get um to get right, you can just do it within
- 1:31:09GitHub actions, make it nice and secure and you go and agents know how to do it. So, if you want
- 1:31:14to build an open source um piece of software, you can and agents will enable you to do it.
- 1:31:22It's interesting that you you mentioned that people are closing their PRs to only verify
- 1:31:27contributors. Um, do you think that they'll go even further and just cut off contributors because
- 1:31:33you can just have AI do everything for you now? Why even bother with the messiness of humans? I
- 1:31:39think um some projects do need teams behind them. You know, you still need teams of people like
- 1:31:45making decisions and um some you know, some more managing triage and contributing and that kind of
- 1:31:50stuff. But I think yeah essentially what you're saying when you cut off PRs is only these five
- 1:31:5712 two people can contribute to this project using their agents and we are sort of and I think that's
- 1:32:05sort of always been true like anyone who's done um any long-term open source will have encountered
- 1:32:13the everyone who's contributed to open source has met a BDFL a benevolent dictator for which is the
- 1:32:21person in charge of the open source project who is wants it to succeed but they also only want
- 1:32:27their point of view in the repo and they are the one who is the the gatekeeper for stuff entering
- 1:32:33it. So contributing to open source has always been weird, always been political, always been a hard
- 1:32:39thing to do well. Um, it's why I have a huge amount of respect for people that I know who
- 1:32:44are amazing at it and who contribute to all sorts of open source projects, but I think it's harder
- 1:32:49to get into than it's ever been. H so what what do you think is the and actually I think we're going
- 1:32:56to get into some of the last things I wanted to ask you here but uh I think before the benefit of
- 1:33:04contributing to open source was that it would show it would be a free place to uh get experience and
- 1:33:09show off what you can do in order to help you land a job and actually make money from that
- 1:33:13because obviously open source there's no money in that. Is that still valuable to to be able
- 1:33:18to show off what you can do now that it's mostly AI doing that? AI has eaten tactical programming,
- 1:33:24but it can't do strategic programming. So, you need to show evidence that you can program
- 1:33:28strategically. You need to show evidence that you can um manipulate infrastructure, that you can
- 1:33:34build a beautiful repository, right? That you can improve your agents environment, that you
- 1:33:39can steer it properly. And I think that strategic programming skill is actually more valuable than
- 1:33:45it's ever been. You know, I I genuinely think that I think it's like because now you have a fleet of
- 1:33:51agents, tactical agents that can just go off and do whatever you want them to do. And so if you're
- 1:33:57able to raise their level and care about improving them, then it's honestly a rid like it's so much
- 1:34:04better than when you were working with humans. I've always cared about this, right? Even when
- 1:34:08I was working with um with people, I've always wanted to make the project better and make us
- 1:34:12all work faster. I've always cared about that, I think, more than maybe my colleagues had who
- 1:34:16just wanted to clock in and go home. So, not all of them, of course, but like just being able to
- 1:34:24have those skills has just felt like a ridiculous uplift in the amount I'm able to trust my agents.
- 1:34:30And I think that is an incredibly valuable skill because that can take you from a you know 5x agent
- 1:34:36user into a 100x 200 200x agent user you know okay so for someone who let's let's take the let's take
- 1:34:46a high school kid they want to become a software developer and in this day and age it's like what
- 1:34:52does that mean? Uh what what advice would you give them? How could they become a software developer?
- 1:34:58Should they? and what do they have to learn? So, I'm not I don't again I'm not I can't predict
- 1:35:05the future. I've not been on the job market for yonsks, right, since 20 or something maybe. Um,
- 1:35:12so I don't know what it's like out there. So, your mileage may vary in this advice, but what I would
- 1:35:19say is demonstrate that you care and that you care about the strategic stuff. you might not even need
- 1:35:26like great expertise in it. But if you're using language like deep modules, if you're thinking
- 1:35:32about okay, how am I going to improve things next time? If you're constantly thinking about not only
- 1:35:37improving the code, but improving the next code that comes through, then I think that is enough
- 1:35:43to start de and you demonstrate that through projects, right? That you're actually shipping
- 1:35:48stuff that may be assisting you in your own job hunts. You know, that's the classic one. um you
- 1:35:52know, you build a little uh CRM for yourself or something and you make it as beautiful as you can.
- 1:35:56You make it as performant as you can. You really care about it and you deploy it somewhere that
- 1:36:00people can click through it and actually view it and and fiddle around with it and go, "Wow,
- 1:36:04okay, this this kid knows what he's doing and he's doing it all with agents." That is what I'd say.
- 1:36:10And it is very possible to demonstrate that and demonstrate huge amounts of work. You just have
- 1:36:15to care about the strategic layout. Okay. So you would say would you say that anyone needs to learn
- 1:36:24a specific programming language right now? Nah. Uh programming languages like you know that's
- 1:36:30tactical stuff right you know actually getting good at a programming language you know that's
- 1:36:36you know it's it's important to be able to read it to understand what module boundaries are but you
- 1:36:42can just get agents to just teach you about it and to teach you about it in pseudo code. What I would
- 1:36:47recommend is start um start working through some of these classic books, you know, like a Michael
- 1:36:52Feathers book or something. Actually buy the book and start obviously it's going to be pretty dense,
- 1:37:00right? There's going to be words in there that you don't understand, but then use the teach skill or
- 1:37:04something. So, it pulls that down information for you so you can start actually going, "Oh,
- 1:37:08I see how I would apply this in real life." You know, this book is from 2003 and I I think it
- 1:37:13was wrapped in plastic in on my like bookshelf or something and I just thought, oh, like I've
- 1:37:19been prompting agents to design good code bases, but this guy was doing that before there were even
- 1:37:24agents, right? He's been writing how to what a good codebase is. And so I pulled it down and I
- 1:37:30just I couldn't believe what I was reading because it just felt so relevant to what I was looking at.
- 1:37:36and I was he was able to put into words the stuff that I sort of felt and I just started plugging
- 1:37:44that stuff into agents and that's how improved codebase architecture came about. Um so care about
- 1:37:52it read books about it use the teach skill to get your way there and that's how to do it. What other
- 1:38:00books should people read? Philosophy of Software Design by John Asterout. Pragmatic Programmer by
- 1:38:06David Thomas and Andrew Hunt. Um Michael Feathers is good. Refactoring by um what's his name? Martin
- 1:38:14Fowler. Uh Extreme Programming by Kemp Beck. Those are probably the big ones. Wow. And so would you
- 1:38:20say that people should read these to become effective at AI programming? Yeah. Uh if you
- 1:38:27actually do the reading, it puts you way ahead of everybody else in the field. genuinely like
- 1:38:33um so yeah definitely I I think I do think that experience is the only way to you know get there
- 1:38:39but it really helps when you have the experience and you have the words to explain that experience
- 1:38:45because I think like just understanding the language and the categories of things and like
- 1:38:51okay this is more important than this is that's what strategic programming is all about. It's all
- 1:38:57about finding balance between lots of different things that are all competing. You know, like,
- 1:39:02okay, I want my agents to ship features really well, and so I'm going to write 4,000 tests
- 1:39:07for them. Oh, but those 4,000 tests are actually slowing things down because they actually make the
- 1:39:12agent run really slowly. How do we winnow those down? How do we put those in a CI so that it can
- 1:39:16rely on them later? How do we shard them so that they've got, you know, running in four different
- 1:39:21um nodes instead of just one? You know, all of that that might sound confusing and complicated,
- 1:39:26but all of those are just words and you just learn the words and then you prompt the agent and then
- 1:39:31the agent can care about it. So, what you're doing is you're just giving the agent permission
- 1:39:35to care about the strategic layer and you're learning the words in order to do that. Okay,
- 1:39:41that makes sense. And it makes me wonder and I know you don't you're not in the job market,
- 1:39:46you can't really look into the future, but how effective right now are computer science degrees?
- 1:39:51I would ask how effective have they ever been actually [laughter] because I I didn't get a CS
- 1:39:57degree, right? I got a drama degree, you know, I was a voice coach for six years and I just taught
- 1:40:00myself this stuff in like 2015 and started moving over. Wow. Um so yeah, I don't know. I don't know
- 1:40:06if that they've ever been that effective. Um the the issue is people who care and people who don't.
- 1:40:13That's interesting. And I I'm sure many people will be watching this and they're in a computer
- 1:40:17science degree. Um, what would you tell them? Care, you know, just just care about the the
- 1:40:27stuff you're shipping. Care about your users, care about the other devs on your team, even
- 1:40:31though they're agents, and care to learn more, care to be curious, and yeah, uh, enjoy because
- 1:40:40it is fun. [laughter] And this is so fascinating because like it's fun to talk with someone who I
- 1:40:46didn't know you didn't have a competi degree because that's that's even more interesting.
- 1:40:49You're selftaught and I I think that's been the best way to learn anything for a while now. AI
- 1:40:54has made that even more obvious. Um for you if you were to hire a developer to join your team, should
- 1:41:03you need one? Actually, that's probably a two-part question. What would make you need one? Why would
- 1:41:08you need one? And what would you look for? what would they what skills would they need to have
- 1:41:12that would impress you? Um I mean I I hate to harp on this again but it is it is that strategic
- 1:41:19thinking right it is the environment it is the all of that and using the right language and caring
- 1:41:23about the right things. Um, for me, I do think there are bottlenecks all over the process still,
- 1:41:30right? This is not just um, you can't just infinitely accelerate and just like, you know,
- 1:41:36one person can ship 2,500 PRs a month, but if you have two people doing that, that's still 5,000
- 1:41:42PRs a month. You know what I mean? That's still doubling the amount of work as long as both of
- 1:41:46them are enabled and both of them can work on cool stuff. And honestly, the the thing that really
- 1:41:53differentiates good products these days is not the amount of tokens that have been spent on them,
- 1:41:58but the amount of attention that has been given to the product, the amount of use that it's had,
- 1:42:03and the the smart things people observe about the product in order to make it better. So, it's the
- 1:42:08testing and the QA and the planning and figuring out strategically what needs to be done next to
- 1:42:13improve the product. How and noticing problems and and essentially product management and product
- 1:42:19design and that is something you need a human for, you know, or you need a human at least to
- 1:42:26tell an agent to care about it, you know. So yeah, that's what I'd say is that people now are all
- 1:42:32about directing agents and using their attention and their care to make the agents care about the
- 1:42:39right things. Okay. And it sounds the word care about a thousand times and I wasn't planning to.
- 1:42:45I think I've just found a new word that I like. So yeah. Yeah. I think uh [laughter] and I think that
- 1:42:50is important because as we've kind of noticed uh through this, you can't just go in and just
- 1:42:56write code. You can't just go in and go, I'm going to make this thing work. You have to really care
- 1:43:00about what it's going to be. You have to think about it's it's like thinking about a child,
- 1:43:06like what do I want you to be? What how what can I expose you to? What are the standards I'm going to
- 1:43:09set for you? What's the rules for life I'm going to give you? Um that's interesting. Uh now I'm
- 1:43:15curious how much of this strategic programming um how to use AI agents and I'm going to allow you to
- 1:43:23shill your course a bit here because I I I love your course. How much of that is in the courses
- 1:43:27you create right now? Yeah, that's like I'm saying all this stuff matters and as I'm saying it,
- 1:43:33I'm trying to find ways to teach people about it. You know, I'm saying software fundamentals
- 1:43:38still matter and so people's follow-up is okay, what software fundamentals, how do I learn them,
- 1:43:43how do I apply them? And so I've got a course coming out in November. Um I haven't released
- 1:43:48this information, but it's coming out November 9th. is like a two-week cohort course that's just
- 1:43:53timed perfectly so that um I go on paternity leave like right afterwards and uh so stressing trying
- 1:44:00to get it done and all of that is about using my skills sure but skills are essentially just
- 1:44:06pointers to a process and the process I'm trying to teach is that you need to understand how to
- 1:44:12improve your agents environment. It's essentially the the end point is where you create a software
- 1:44:17factory that does a lot of this stuff for you. So a lot of the caring is sort of automated in these
- 1:44:23cron jobs in these routines that automatically try to make your agent and its environment better. As
- 1:44:29I was saying it's sort of like you know ideally the harness would improve the environment as you
- 1:44:35go and that's what we're trying to get to by the end of the course. Interesting. And I I've I just
- 1:44:41heard the term software factory for the first time last week and I know it's been thrown around for a
- 1:44:45minute. What exactly is that? Well, on the AI coding dictionary, we have a definition for a
- 1:44:51software [laughter] factory. There is a definition for that. Um, and it's essentially I I won't pull
- 1:44:57it up, but what it does is if you imagine that producing work involves several points at which
- 1:45:07the human needs to interact with the work as it's being done. So, it might be that you need to tell
- 1:45:14the agent to do something. That's the first point. Then you might do a grilling session
- 1:45:19and you might um approve the plan. That's the next point. Then it might go all the way through until
- 1:45:24the work needs to be merged. Now what a software factory does is it tries to take the human out of
- 1:45:30more points in that process. So it might be that instead of you um needing to uh initiate the work,
- 1:45:39it might be that there's some trigger that happens. Let's say a new issue is created
- 1:45:43on your repo that pings down to an agent and the agent decides whether to initiate some work based
- 1:45:48on that. Or it might be there's a slow database warning in your um in your service that checks
- 1:45:54your database that again goes down to an agent. The agent decides whether to um proceed with that.
- 1:46:00And then you might have ones where okay that might not even need a human in there all the way up to
- 1:46:05human review. But maybe you don't do human review for things. Maybe you use that oneway two-way
- 1:46:10door thing and actually only one-way doors need a human review and the agent just goes and merges it
- 1:46:17itself. So the software factory is basically just taking humans out of more points so that you can
- 1:46:23accelerate the development of software more. And the issue with that is that the more you
- 1:46:29accelerate it, you need breaks as well. You need to be able to pull the software factory back so
- 1:46:33it doesn't just become a slob cannon. which again is where the strategic layer, the thinking about
- 1:46:38the environment really makes a big difference. Okay. And that that sounds great. Um but it also
- 1:46:46sounds really hard to do. Yeah. Yeah, I think so. I think um there are you're basically having
- 1:46:53to plug together a bunch of services in order to do it. I think as um the tools and the harnesses
- 1:47:00improve and make those things easier then you know you know like people are obsessed with like loop
- 1:47:06engineering like um few months ago or whatever this is basically loop engineering what you're
- 1:47:11doing is you know a loop is one way to do it where you just schedule the agent to just like pull the
- 1:47:16GitHub issues and just sort of initiate work on its own. Um but you can have multiple different
- 1:47:20ways of doing it essentially. So this is just loop engineering by another name essentially and it's
- 1:47:25it's very exciting and it means that you can churn out a ton of work and really improve the quality
- 1:47:30of what you're working on. Uh and that's what software factory is. Interesting. Okay. And then
- 1:47:36you I'm curious about how you approach so like for like right right now if I went to your course
- 1:47:43website is that a platform or is that something you guys designed? Yeah, that's something we
- 1:47:47built from scratch. Um I work with a team called badass.dev. dev and they work with a few course
- 1:47:54creators and they uh build this uh course platform from scratch basically. Oh dang, that's pretty
- 1:47:59cool. So you're not you're not running the AI coding for that right now? Uh no. What are some
- 1:48:05do you have any projects you're working on right now? Uh the open source stuff? No, just S castle.
- 1:48:12just um trying to get this course built and trying to uh yeah ship this video editor that's uh always
- 1:48:19always constantly improving. Okay. Okay. I can't resist. Tell me about this video editor. I I've
- 1:48:26seen you post about it a few times and I'm like that looks interesting. Okay. He's he's having
- 1:48:30it create demos for him before he records it. That's interesting. So what is this thing? Oh,
- 1:48:36it's it's outrageous. So it's [laughter] uh it is it is outrageous. It's uh uh what it is. It came
- 1:48:44down to this idea that I hate editing video. And so what I wanted was I wanted to as I recorded,
- 1:48:50I would essentially leave a little gap and then it would cut that and that would go be transcribed
- 1:48:57automatically and just appear as a clip in the finished editor essentially. So there's
- 1:49:02no editing. So I edit as I record. And from there that developed into, okay, I need like a I use a
- 1:49:09lot of diagrams in my setup. I'm going to build like a custom diagram playground as well that
- 1:49:13links to the clip so I can see from the clip which diagram I was viewing that so I can go
- 1:49:17back in time and just edit it and stuff. And then maybe I Oh, yeah, Elgato teleprompter looks quite
- 1:49:22nice. Maybe I need a teleprompter, too. Um, oh, maybe I can customize the teleprompter based on
- 1:49:26what I'm seeing. Oh, maybe I can get an agent to write scripts. Maybe I can get an agent to write
- 1:49:30animatics as well. Opus 5.5 looks really good with actually building out um custom videos for you. So
- 1:49:36why don't I watch the video I'm about to record before I record the video. That seems good. And
- 1:49:40then from there I you know it just grows and grows and grows and like and now like every edit I make
- 1:49:45get automatically synced to my course platform. Um I have you know version history. I have linting. I
- 1:49:51have uh AI writing. Like it's just a a ridiculous wonderful project. And it's it's open source too.
- 1:49:57So, it's in course video manager. We're 2,500 commits deep now. Um, and it's just all in here,
- 1:50:03you know. So, if you want to uh use this as like I I don't take um pull requests and um like I don't
- 1:50:10even take issues on this actually. Uh but if you want to just have a look at it, it's there. Yeah,
- 1:50:14I'll take a look. And I I I do something not quite as streamlined as you, but I do
- 1:50:19um have it to where when I finish recording, my AI has been trained enough back and forth with me to
- 1:50:26where it knows how to edit me. It knows what kind of cuts I like. It knows that when I cuz I I I've
- 1:50:31been doing this for a long time. I used to edit my own stuff. Now I have editors. So like when
- 1:50:34I say something, I'll freeze so I can remember what to say again and we don't have to do a weird
- 1:50:39cut and then I'll keep talking. Um so it knows to watch for that. So when I finish recording,
- 1:50:44I'll just tell, hey, go go go to the place we where we have our files recorded, offload it,
- 1:50:50ingest it, and what that means is it will um transcribe it. It'll use the Gemini video API to
- 1:50:56watch it so we know exactly what's happening. And then um it will start to actually produce an edit,
- 1:51:02and it will edit it to like a very clean draft. We coded our own frame to IO. I'll watch it to make
- 1:51:08sure the story is right and type and then it will export an XML for Adobe Premiere where my editors
- 1:51:14just open up and there's the draft ready to go and they can make it look pretty. Um, it's it's very
- 1:51:19cool, but I want to get some more where you are. That sounds very interesting. Uh, it's just so
- 1:51:24great. It's so great. Um, yeah. I mean, it's, you know, it's there if you want to have a have a play
- 1:51:29around with it. Um I I just wanted a project that I knew that I would constantly use and constantly
- 1:51:36update and like so it's just always in the back of my head that okay I'm going to work on this going
- 1:51:39to work on this and I've you know I've got agents right now like playing around with it improving
- 1:51:43the environments that the code operates in. This is my main playground really for trying out my
- 1:51:47ideas. Now you you mentioned that you have AI scripting. How do you feel about that? Do do
- 1:51:54you do you uh speak word for word from what the AI scripts for you? No, I mean I did a drama degree,
- 1:52:01right? So like I'm good at improvising words and I just use the script as like a
- 1:52:07uh it has like commands that I can copy and paste in order to set up the environment and stuff and
- 1:52:11like go to the right commit in the the history for the project I'm working on. Um so mostly it's just
- 1:52:17a guide essentially. So I'm always improvising what I'm saying. Interesting. So I I mean I create
- 1:52:23courses as well. So I'm I'm I'm very curious about your process. Uh is AI pretty much part of every
- 1:52:29process of your course creation from like ideation to to episode or um video management and all that?
- 1:52:36So, well, one thing I do and one thing uh that I I actually recommend to anyone who has an audience,
- 1:52:42who has people pinging them all the time is I have a a personal wiki which is for my audience.
- 1:52:49And every 3 hours or so, I have a chron job that um looks at all my tweets people have sent me,
- 1:52:56all my mentions, uh all my Discord messages, all my Slack messages, my Gmail, basically every sort
- 1:53:01of input I can find and then writes that or sort of ingests it into uh Carpathy style LLM wiki.
- 1:53:09So that means that I can go into this, it's just a repository, it's private obviously,
- 1:53:13and I can just ask it, hey, what is my audience interested in? Right? what um questions do they
- 1:53:19have about these skills? And from there, I can create learning goals from that. So, things that
- 1:53:25people are interested in learning. From there, I have to actually get off AI. I can't use it for
- 1:53:30this next bit because it's too hard. I print off those learning goals onto these little car bits
- 1:53:35of card, right? Oh, I love And I I like rearrange them physically in space until I realize, okay,
- 1:53:41this feels like a lesson. This feels like a lesson. This feels like a lesson. figure out
- 1:53:45that can take a couple of hours actually just off computer off AI um just for one section
- 1:53:51and my courses often have eight sections. So and from there I go back and plug that back in so it
- 1:53:57turns into lessons and turns into videos and then I start writing the scripts and getting agents to
- 1:54:04dry run it as well. I create these animatics and then I'm ready to record. So you know it it can
- 1:54:10be months planning a course. It really can. It's very very slow. That's fascinating. So, I mean,
- 1:54:15two things that just jump out at me. First, the the wiki idea is such a good idea. Um,
- 1:54:19I'll probably steal that. Uh, the second one is uh that you have to go off offline to do
- 1:54:25this kind of thinking, which like I've got pen and paper and this is I find the best thinking happens
- 1:54:31with this. And I do use AI quite a bit to bounce ideas off of and like try to form an idea more,
- 1:54:37but it's it's those very human moments where I'm disconnected that things the magic happens. So
- 1:54:44I'm curious how often do you do you go back to that kind of thing where you're disconnected?
- 1:54:49It's mostly on tasks where there is a lot of information and I need to keep it all in my head
- 1:54:54at the same time because if I'm like planning if I am like talking to an agent and we're having a
- 1:55:02back and forth about a plan and maybe shaping the plan in certain ways then it is going to write a
- 1:55:08ton of stuff right in response to me and I lose so much time by trying to get over that communication
- 1:55:14gap right constantly trying to go okay what do you mean And and in doing that, I lose context,
- 1:55:22right? I lose information that I could if I just slowed down and I just kept everything in my head
- 1:55:29at a speed that works for me. L, you know, writing stuff on pen and paper. It sort of slows you
- 1:55:34down so that you can consider things at the right speed. It's it's me paced, right? It's paced at a
- 1:55:39human speed. Whereas agents just work too fast for that. So for specifically planning, that's what I
- 1:55:47have to do. And I've not found a better way to do that with agents. That it's it's interesting you
- 1:55:51mentioned it that way. When you're talking to an agent, you're like trying to bounce an idea off of
- 1:55:55it or just like try to do anything. It does feel too fast and you're it'll say a bunch of stuff
- 1:56:01and it's so dense you're like or too slow as well, right? Or you're just waiting for a response for
- 1:56:06like 2 minutes and you think, "Oh, I just check Twitter." And then half an hour later you go, "Oh,
- 1:56:09what did the agent say?" You know? Right. Oh, yeah. Yeah. Like like 7 7 o'clock at night and
- 1:56:14you're like, "Oh, I would did ask that question earlier. at night and I don't even care about that
- 1:56:19anymore. Exactly. You just got to get out of the conversation. Just keep it in your own head for
- 1:56:22a while and then I usually just brain dump it back to the agent. Um just, you know, use whisper flow,
- 1:56:27dictate out a ton of stuff and then um then it just pulls it into its own structure and its own
- 1:56:33data and then I just usually chuck the cards away because I don't don't need to go back to paper
- 1:56:37there because for small edits, for little stuff, once you've got the main structure, AI is really
- 1:56:42good. Do you use AI in a way to where because I know we've been looking at coding agents and and
- 1:56:49getting the coding agents to know your repos and your code, but do you use anything for your life?
- 1:56:54Do you have like a Hermes agent or any kind of agent that like has the context of you? Um, yes.
- 1:56:59This audience wiki has is sort of a bit of a blend of like me plus my audience, right? Because those
- 1:57:04things are often closer than you think. And so, yeah, I I have that repo sort of serves as like my
- 1:57:10home base for all of my agents. And so I have like a daily standup with my agents where I say, "Okay,
- 1:57:15what tasks need to be done today?" Um, I have a like a pixel watch that I dictate stuff into and
- 1:57:22that goes on my to-do like an agent triage to-do list and the agent then pulls it into um it was
- 1:57:29like uh deciding every minute of the day for me like scheduling out all that stuff. And I have
- 1:57:34like a little um little thing in front of me. This is called a terminal that basically Yeah, they're
- 1:57:39kind of they're all right. I don't know if they're any good, but I've got two of them actually,
- 1:57:43one downstairs and one upstairs. And uh I was just sort of ignoring it. So, I'm I'm still trying to
- 1:57:48find the right uh interface for the agent to help me run my life, you know, but it's a it's a me
- 1:57:54problem, not not an agent problem. The agents are really good at it. I find the same thing. I'm I'm
- 1:57:59trying to figure out the best way to like like help me schedule my day or help me do this and
- 1:58:03help me work with my team and help it's it's I don't think anyone's quite solved it yet. Well,
- 1:58:08it's really hard to figure out. No. And I think it's actually much harder than coding. Actually
- 1:58:13using it for knowledge work is much harder than coding because coding is so verifiable, right?
- 1:58:18You can use these automated checks to actually make sure that the thing you're doing is correct,
- 1:58:23that you have high quality on the output. But with knowledge work, with planning a course, how
- 1:58:29do you how do you know that you've built a better course than you did two months ago? You know, it's
- 1:58:33really hard. There's no way to verify that. How do you know that this video script is better than
- 1:58:39um your previous one? So, yeah, I think knowledge work is really difficult, way harder than people
- 1:58:45think it is. And I think we're going to be hitting that over the next few years as we try to force
- 1:58:50AI into all of these holes. And I love that this entire call you everything coding related you're
- 1:58:57like it's easier than you think with knowledge work you're like it's actually harder than people
- 1:59:02realize. [laughter] Yeah, maybe I'm wrong there today. Maybe it's just because I know coding more
- 1:59:07than knowledge work. Maybe. No, I think I think you're right though. It's that very very distinct
- 1:59:10point you made is that coding is verifiable and that you can learn it and it's it's deterministic
- 1:59:16and you can test it and like you you know it's good, it works or I mean that's that's why agents
- 1:59:21are amazing at math problems and can solve math problems way better than humans because every
- 1:59:27part of it is verifiable. All the inputs are text and all the outputs are text. But it's also why
- 1:59:32they still can't draw hands, right? Because like it's just different inputs than they're used to,
- 1:59:39right? They can't write a convincing tweet that we don't know is is AI, right? Because that is just
- 1:59:44a much harder job because it's so so much less verifiable. And it's something I don't even really
- 1:59:49care the AI can do. I think there's there's human things that I want to keep human that just make us
- 1:59:53who we are. And communication is like one of those things that make us who we are. Absolutely. Yeah.
- 1:59:57I play a ton of board games. Um keep agents out of my board games. I don't want to see any AI art. I
- 2:00:03don't want to hear any AI music. I don't want to uh I get a thousand AI tweets a day and I hate
- 2:00:08them, you know. [laughter] Yeah. Yeah. I think we feel the same way. Yes. Absolutely. Well, we are
- 2:00:14near the Actually, no, we are at the end of time. Um tell me where can people find you and with your
- 2:00:20courses where should they start? Go to aihero.dev. It's the place. Um find my skills. Uh install my
- 2:00:27skills, enjoy them. Um, if you have any questions, go to the Discord, find me on Twitter. Um,
- 2:00:31you know, I'm pretty active in all those places and I'm on YouTube as well. Um, recently crossed
- 2:00:35400,000 subscribers, which I'm very pleased about. Congrats. And, uh, yeah, look me up. I'm there. Do
- 2:00:41you uh do you have like a wait list for your new course? Uh, there's a newsletter. Um, but yeah,
- 2:00:47sign up and I'll see you there. Cool. Awesome. Well, Matt, this has been so fun and informative
- 2:00:53and you've given me so many ideas. Thank you for taking the time to not only teach me but my
- 2:00:58audience. This has been very very cool. Appreciate it, pal. Hey, made it to the end of the podcast.
- 2:01:03At the end of my videos, I like to pray for you, my audience. Kind of weird, I know, but just go
- 2:01:08with it. You're not here for um I mean, you are here for a reason. It's not an accident. And yeah,
- 2:01:15let's just go with it. Uh I put it at the end, so you have the option to just, you know, duck out.
- 2:01:19The information has already been given, but if you want to chill out and stay here with me right now,
- 2:01:23I'd like to pray for you and your life and your family. Ready? 1 2 3 pray. Uh God, I thank you for
- 2:01:28the person on the other side of this screen, this camera. I thank you for them, who they are. And
- 2:01:34um I thank you that they stuck to the end of this and that they are learning about coding and and um
- 2:01:40technology and just all this AI stuff, which can be stressful, but also is very life-giving if we
- 2:01:45approach it the right way. So God, I ask in your name that you shift any anxiety, any stress uh
- 2:01:52about AI or the future, about jobs. I I ask that you would just flip that switch in their brain to
- 2:01:59be joyful and excited and let them be at peace about it and just give them energy and and give
- 2:02:06them ideas and a path forward. If they're looking for a job right now, I pray you provide that job.
- 2:02:11If they're looking to start a company, give them that idea and the and the steps forward. Bless
- 2:02:15them right now. Bless them and their family. Watch over their families. Um give them peace and joy in
- 2:02:23their lives right now. And help them to take the knowledge from this video and apply it well. I
- 2:02:29ask this in your name, Jesus. Amen. Thank you for letting me do that. I'll catch you guys next time.
About this transcript
This page contains the full transcript of How to Actually Code with AI (ft. Matt Pocock) by NetworkChuck, generated from the public captions YouTube serves with the video. The transcript has 24,631 words across 1,326 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.