YT YouTube2Text

YouTube2Text → Transcripts → I Built A Self-Improving AI Software Factory

I Built A Self-Improving AI Software Factory — Transcript

by Owain Lewis · 3,138 words · 447 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00Today we're talking about software
0:01factories and how they can make your
0:02agentic coding more consistent, more
0:03measurable, and more scalable. This
0:05isn't going to be one of those hype
0:06videos where I tell you to stop using
0:07coding agents locally because I think
0:09that would be terrible advice. I'm still
0:10using local coding agents for a lot of
0:12my work. But I think there are many
0:14advantages of using a software factory.
0:16One of the big challenges with agentic
0:18coding is that when you change your
0:19models, your prompts, your harnesses, or
0:21your skills, [music] it's not very easy
0:23to know whether those things make any
0:25valuable difference. So the advantage of
0:26a software factory is it gives you a
0:28repeatable way of measuring and
0:30improving your process over time. So in
0:32this video I'm going to break down a
0:33software factory that I built. I'm going
0:35to show you how to set it [music] up and
0:36configure it. I'm going to show you the
0:37architecture of a software factory. And
0:40more importantly, we're going to talk
0:41about how you can use evaluations and
0:42software factories to improve your
0:44coding process over time. You can make
0:46some drastic improvements in terms of
0:48your own [music] efficiency and learn a
0:51lot about your own agentic coding
0:52process by following some of the ideas
0:54in this video. This is going to be a big
0:56topic. So all of the resources and
0:58everything you need to get started, I'll
0:59link for free in the description below.
1:00So let's get into it. I personally think
1:02about a software factory as
1:03infrastructure for running coding agents
1:05through a defined process. So what I
1:07mean by that is you put work into the
1:09system and it will run through a defined
1:11process the same way every single time
1:13and you record metrics and data about
1:15that that particular run. So this allows
1:17you to see everything that's going on to
1:20manage coding without having to think
1:22about you know work trees and branches
1:24and how to set up the local environment.
1:26You just delegate the work and the
1:27factory or the system will take care of
1:29it for you. And I think about two parts
1:31of the architecture. The data plane and
1:33the control plane. So the control plane
1:35is where you manage the work. And then
1:37the data plane is made up of the
1:39workers. The workers are essentially
1:41delegating to coding agents. They're
1:42setting up your environments. They're
1:44doing the work. So the useful
1:46distinction really is between the worker
1:48and the agent harness. So the worker
1:50lives on the data plane. You could have
1:51multiple workers across multiple
1:53different virtual machines. You could
1:54run this on a Kubernetes cluster if you
1:56wanted to. So you admit work to the
1:58control plane which will queue up the
2:00work. And then the workers are just
2:02going to pull tasks off the queue and
2:05then delegate those to the right coding
2:06agents and make sure the work gets done.
2:08And then the workers are going to, you
2:10know, record the results of that work
2:12and then send it back to the control
2:13plane so that you can actually see
2:15what's going on in a cycle. So the way
2:17that I personally like to set this up is
2:18using an SSH tunnel because I'm just a
2:21solo developer. I don't need to deal
2:22with any security headaches or complex
2:24infrastructure setups. I just put this
2:26onto a virtual machine. I run it on the
2:28virtual machine, but then I use an SSH
2:29tunnel to forward to access this
2:32machine. So essentially what this does
2:34is we can run our software factory and
2:36our agents on a virtual machine with no
2:39public internet access. So we can have
2:40these completely secure and private, but
2:43then we can use an SSH tunnel to connect
2:45to them on our local machine. So I'll
2:47show you how that works now. So I have
2:49this deployed on a virtual machine. So I
2:51can go onto the machinist virtual
2:53machine and this is just an abundu
2:56instance. And then if I type the
2:57machinist
3:00command, you can see here we have a CLI
3:02tool that has a bunch of different
3:03options. We can run our workers straight
3:05away. You can see here I'm signed in
3:07with my subscription, but the agents are
3:09running on a virtual machine, not on my
3:11laptop. So when I run this command, it
3:14will allow me to go to my browser and
3:16now access the UI. So this is the
3:19machinist UI. You can see here we have a
3:21bunch of work that's been completed, a
3:22bunch of work in progress. So you can
3:24see a list of all of the stuff that
3:25we've been working on. Uh you can see
3:27analytics here about what's going on. So
3:30you can see the average task time is 18
3:32minutes and we have 15 million tokens
3:33used so far. I only set this up
3:35yesterday. And we have workers. You
3:38could have multiple different workers.
3:39You could run the workers on, you know,
3:41cloud instances, Kubernetes jobs,
3:44however you want to set this up. I only
3:45have one worker right now which is
3:47running on the same virtual machine as
3:50the control plane. This is connected and
3:52the worker will then just take the tasks
3:54and dispatch them to coding agents. If
3:56you're working on many things at once or
3:58you're working in a big team, you might
3:59want to have more workers added to the
4:00pool. And then you can see here the
4:03prompts that we're running. These are
4:05the actual agent definitions. So every
4:07single task when we send a task through
4:09machinist is given to a foreman
4:12coordinator. The coordinator will be
4:14responsible for taking the task and then
4:17basically working through it in a
4:19sequence delegating the work to coding
4:21agents or sub agents. The goal here is
4:23to have this one agent spin up sub aents
4:27implement the code change and follow a
4:29defined sequence. So even down to how we
4:32manage labels. It's all configured here.
4:34It's all defined. But this is the
4:36repeatable process that every single
4:38coding task will go through in the
4:41system. So as you make changes to this
4:43specific prompt, every single task that
4:46you do will be using the same prompt
4:49every single time. So this is how you
4:50build the consistency. I have two
4:53triggers set up. You can define triggers
4:55within the factory. So this first
4:59trigger will monitor the repository. So
5:01every so often it will run a health
5:03check. It will scan the codebase using a
5:06cheaper model, find bugs, find issues in
5:08the codebase, and raise tickets. And
5:10then the other trigger that we have is a
5:12GitHub watching trigger. You can
5:14configure all kinds of different
5:15triggers within your repositories. And
5:18this will then monitor for a specific
5:20label on a task. When the agent sees
5:23that there's a particular label, it will
5:26then pull that work into the factory. So
5:27it's a way of getting work into the
5:29system. And so the nice thing about
5:31having a factory system like this is you
5:33get a historical record of all of the
5:35work that you've been doing. And so you
5:36can see as you start to change your
5:38models and you start to change your
5:39prompts, you can see the difference it
5:41makes on the system overall. So if you
5:43want to make a change or use a different
5:44model, you can run that for a couple of
5:46days, collect a bunch of data, and then
5:48start to see what difference it makes in
5:49terms of the cycle time, how effectively
5:52the tasks are being completed. So this
5:54is kind of a way of experimenting a lot
5:55with your coding setup. So just quickly
5:58within the codebase, you can see here
5:59that we have our configuration. So let's
6:02go ahead and take a look at some
6:03examples. So this is the configuration
6:06for the server. So you can see here we
6:08have the database which is just using
6:10SQite. So we have the max concurrent
6:12jobs. Then we have a forman job or a
6:15foreman agent. And this is a prompt
6:17definition. We're using codecs. And then
6:19we have a audit job as well. Again these
6:22are just prompt definitions.
6:24And if we go into the worker
6:26configuration you can see here we're
6:27configuring the executors. So we're
6:29using codeex or claude and we've defined
6:32the models that are available to the
6:34executor. You can build in any executor
6:36here. You could have multiple different
6:38coding executors and you can start
6:40experimenting then with different setups
6:42to see what makes a difference. This is
6:44the foreman prompt. This is the prompt
6:45that every single task inside the system
6:47is going to be running through. It
6:49controls the workflow and it delegates
6:51work to sub aents. So we're just working
6:52on a single GitHub issue. I also have a
6:55bunch of somewhat deterministic evals as
6:57well. So I can do automated checking and
6:58automated testing to make sure that the
7:00the right labels the right workflow is
7:02being followed. So you can also define a
7:04bunch of automated tests as well in the
7:07spirit of like evaluating and improving
7:09your system over time. So when it comes
7:11to software factories, you can either
7:12run your infrastructure locally or you
7:13can run it in the cloud. I've chosen to
7:15run my infrastructure in the cloud. So
7:17both the control plane and the workers
7:19are running on virtual machines. You
7:21could also run this using things like
7:23Kubernetes containers. You can run these
7:26using Cloud Run jobs. You can run this
7:28in a million different ways. I like to
7:30keep it simple and because it's just me
7:32working on this, I'm using a simple
7:34virtual machine setup. If you're working
7:36in a team environment, you might do this
7:37differently. Taking a look at the
7:39pricing across all of these different
7:40providers. There are a whole bunch of
7:43options when it comes to this. I
7:45personally use Google Cloud for almost
7:46all of my professional work, but I find
7:49that the virtual machines are pretty
7:50expensive. You can see here even for
7:52like an E2 medium in London is $32 a
7:56month plus the disk usage. So this
7:58starts to add up quite quickly. Whereas
7:59if you use a Hetsner virtual machine,
8:02it's significantly cheaper. So you can
8:04see here I'm now running an Iuntu 16 gig
8:06in Helsinki. So this is just a cheapish
8:09virtual machine. Like I said, if cost is
8:12a concern, you can just run this
8:13locally. You don't need to run it in the
8:15cloud. But I find running these agents
8:17in the cloud helps me think better about
8:19the infrastructure. It helps me kind of
8:22separate out my local development
8:23environment from the the infrastructure
8:25that where I'm running my coding agents.
8:27So Google Cloud is where I run most of
8:28my infrastructure, but I find that the
8:30virtual machines are relatively
8:31expensive when compared with something
8:34like HTNA where you can get a bigger
8:36virtual machine for much less money. The
8:38next thing I want to show you is the
8:40idea of eval. And this is kind of where
8:42I started with this whole experiment.
8:43the idea that I want to start improving
8:45my process over time and experiment with
8:47different models and see with data
8:49what's actually working. You can
8:51obviously do this with your local coding
8:53setup. So all of your coding agents will
8:55emit logs or JSON files containing a
8:57bunch of metadata that an agent can
8:59interpret. The difference here with a
9:02factory is that we're using the same
9:03prompt every single time. So every
9:05single task that runs through our
9:06factory goes through exactly the same
9:08series of steps. So this was a very
9:10simple example. I just said, "Can you
9:11connect to the remote virtual machine
9:13instance, review the outputs of the
9:14latest coding runs, find ways to improve
9:17our prompt to make it more efficient in
9:18terms of token usage?" And so the agent
9:22then would SSH into the machine. It's
9:24reviewed all of the output and the data
9:25in the database. And you can see here we
9:27have a bunch of interesting data. So you
9:29can see here the the times taken for
9:32these particular issues. You can see
9:34here the number of tokens used, the
9:35number of commands run, the number of
9:37agent weights as well. So what I'm
9:39thinking now is I might actually
9:40experiment with using more deterministic
9:42Python scripts. So within the machinist
9:44setup, you can run an agent, but you can
9:46also run a script. And so what I might
9:48do is experiment with something like
9:49this where we have a essentially a
9:52deterministic script that will monitor
9:55CI delegate to an agent. And this might
9:57be a lot more token efficient generally
10:00when working with agents. So by moving
10:02this to a script, it's going to be
10:04cheaper, more predictable, more
10:05testable, and gives the same structured
10:07output every time. And so I think this
10:10is something that I'm going to do next.
10:12So this is useful feedback just by
10:14analyzing our runs, looking for ways to
10:17improve. Now, what I can do is go ahead
10:19and write a script to improve. So I'm
10:22going to go ahead and cut a task to do
10:24this, and then I'm just going to go and
10:25try experimenting with this now over
10:28time. So now we found a very specific
10:30improvement to my workflow. And this
10:32would improve everything about my
10:33workflow to be honest because I'm using
10:36the same coding loop on every single
10:37task every single day. So if you can
10:39imagine that implementing this one
10:41change to my workflow could save me, you
10:43know, a significant number of tokens
10:45every single time I use a coding agent.
10:48This is going to be a massive
10:49improvement to my overall workflow. So
10:51you can see that this could save you a
10:52ton of money, make your process more
10:54reliable. Having this ability to do this
10:56in a in a kind like a repeated way is
10:58super useful. So what I'm going to do
11:00now is add a label as before. So we're
11:03going to add a machinist cued uh label
11:07or requested label, sorry. And then
11:09we're just going to basically get the
11:11agents to go and build this for us and
11:13then start to measure and experiment and
11:15see if this makes an improvement to our
11:17overall workflow. If this does make an
11:20improvement to the workflow on the
11:21factory instance, I will then also
11:23incorporate the same code, the same
11:25workflow on my local development setup
11:27as well, just day-to-day. So when it
11:29comes to software development, there's a
11:30big difference between working by
11:31yourself as a solo developer or working
11:33in a team environment. So I personally
11:35work both as a solo developer. I run my
11:37own AI business. I do a lot of side
11:39projects, but I also work in a large
11:41enterprise environment with, you know,
11:42hundreds or thousands of developers. And
11:45there's a big difference between those
11:46two situations. In a team environment, I
11:49think there's a lot of benefits from
11:51having shared infrastructure where
11:52everyone is using the same workflows and
11:54systems to implement code changes. That
11:57way, as a team collectively, you can
11:59make changes to your setup, your
12:00prompts, your configuration, and those
12:03changes benefit every single person in
12:04the team. This is far better, in my
12:06opinion, than every single developer in
12:08the team. Using different models, using
12:10different harnesses, using different
12:11prompts, different skills. There's no
12:13way to kind of have consistency across
12:15the team in terms of the workflow or get
12:17any kind of repeatability and it's just
12:20very hard to make improvements in that
12:21situation. With a software factory,
12:24everybody is using the same process. Any
12:26change you make to your software factory
12:27configuration benefits every single
12:30person in the team. And this is the same
12:32going back to the analogy about CI/CD.
12:34It's the same idea that the system can
12:36be improved and it benefits everybody so
12:39that you have massive massive leverage
12:41compared to individual people doing
12:43everything completely differently with
12:45different agent harnesses and different
12:46skills, different prompting abilities,
12:49different techniques, etc. I much prefer
12:51this idea of a common systems approach
12:54to implementing software. So I
12:56personally found it really valuable to
12:58be using a software factory. I think
12:59this idea of experimenting and trying
13:01out different techniques and seeing what
13:03works and then improving your process is
13:04probably the most valuable part [music]
13:06of the software factory process as well
13:07as having a very repeatable and reliable
13:09way of working with these coding agents.
13:12So, I'm going to be using this in my own
13:13workflow, but like I said, you don't
13:15need to use a software factory for
13:16everything you do. If you're working on
13:18design or iterative work, it doesn't
13:20make sense to me to be using a software
13:22factory cuz you need to stay close to
13:23[music] the code. You want to iterate.
13:25You want to go back and forth. if you
13:26want to see things in the browser and
13:28move very very fast. So I don't think
13:30that's a good use case. I think I'm
13:31still going to be using coding agents
13:33locally but in conjunction with a
13:34software factory. So I'm going to be
13:36using software factories for things like
13:38queuing up large amounts of work that
13:39happen overnight and then just
13:41delegating those and sequencing them and
13:43making sure they run repeatedly and get
13:45done. And then in the morning I can
13:46review those tasks. But any kind of work
13:48where I need to do experimentation, you
13:50know, I need to talk to an agent, think
13:52through a problem, um, anything like
13:54that, then I'm going to still use a
13:56local coding setup. I hope you found the
13:58video useful. If you did, please
14:00remember to like and subscribe. It
14:01massively helps out the channel. And
14:03I'll see you in the next one. Take care.
14:04[music] If you want to go deeper on
14:06building AI systems, using Aentic coding
14:08tools, leveling up your career as a
14:09consultant, as a freelancer, or inside a
14:12company, I cover all of those things
14:14inside my community. I've been a
14:15software engineer for the last 20 years.
14:17I've been both an independent AI
14:19engineer. I've worked as a director of
14:20engineering in a large organization. And
14:22I've also freelance and build software
14:25every single day. So, if you need help
14:26with any of these things, you can check
14:28out the link in the description below.

About this transcript

This page contains the full transcript of I Built A Self-Improving AI Software Factory by Owain Lewis, generated from the public captions YouTube serves with the video. The transcript has 3,138 words across 447 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.