YT YouTube2Text

YouTube2Text → Transcripts → My AI Dark Factory Dream Is Dead (Here's What Actually Works)

My AI Dark Factory Dream Is Dead (Here's What Actually Works) — Transcript

by Cole Medin · 4,400 words · 617 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00The AI dark factory is the most
0:01autonomous form of agentic engineering.
0:04We give it some kind of higher-level
0:06document for an epic or release, and it
0:08figures out how to build and validate
0:10everything and ship it straight to
0:12production. If this could be reliable,
0:14it would be the dream for building any
0:16software. But, to be a bit dramatic for
0:19effect, my hopes and dreams have been
0:20crushed here. As I've been experimenting
0:23building my own dark factory, I've come
0:24to the conclusion this is just simply
0:26not the way to do things. But, what is
0:29actually working for me, what I've
0:30turned this into, doesn't have to be
0:32that much different. I'm not actually
0:34losing that much autonomy. And so,
0:35that's what I want to cover with you
0:37right now. And let me tell you, it has
0:38been a fascinating journey to get to
0:40where I am now. There's so many lessons
0:41I want to share with you in this video
0:43for why the dark factory doesn't work
0:45and why what I'm building now actually
0:47does. And so, for the dark factory, even
0:49if we had AGI and the best LLMs in the
0:52world, this still wouldn't work. We need
0:53a system that looks more like this. And
0:56yeah, I know, it looks complicated, but
0:57I'll break it down nice and simple for
0:59you right now. And it's important to
1:00understand this kind of workflow
1:02because, let's face it, the way we work
1:04with coding agents, even in the next 6
1:06months, isn't going to look the same.
1:07Because as harnesses and LLMs get better
1:10and better, we need to give them more
1:11autonomy if we want to ship faster. And
1:13so, I truly believe in the next couple
1:15of years every person and company is
1:17going to be moving towards a system that
1:19looks like this. And even if you don't
1:21want to go and build a full AI software
1:23factory right now, which yes, that is
1:25different than a dark factory, it's
1:27still good to see what the full process
1:29looks like. There's a lot of lessons I
1:30have built in here that you can even
1:32just incorporate for your current
1:33agentic coding workflows. But overall,
1:36this is how I'm seeing the industry
1:37converge. As I work with companies and
1:39build this sort of thing, talk to people
1:41that are creating their own, everyone is
1:43building a system that is becoming more
1:45and more similar over time. So, this
1:47really is the future. This is the next
1:49step for agentic engineering. Okay, so
1:51before I get into anything else, I need
1:53to cover why the dark factory doesn't
1:54work as much as I wish it did. And so,
1:57the number one goal here is plain and
1:59simple. We just want to make our coding
2:00agents as autonomous as possible. Now,
2:03it's not vibe coding because we're
2:04putting a lot of effort up front
2:07engineering our harness. And so, yeah,
2:09we're trusting the agent at runtime when
2:10it's building the software, but it's
2:12only because we put a lot of our
2:14engineering effort into setting it up
2:16for success. And so, we define the exact
2:18process it goes through as it builds
2:21everything.
2:22Now, for my own dark factory, I started
2:24it as a public experiment more towards
2:26the start of the year. Now, watching the
2:28factory itself run is very boring cuz
2:30we're just waiting for agents to churn
2:31through work. But like I said, we're
2:32putting a lot of that engineering effort
2:34into the harness up front. So, that is
2:36what I showed with YouTube videos and
2:37live streams that I did. It was a blast
2:39and I built a pretty successful dark
2:42factory. I covered this a lot back then,
2:44but my main inspiration for building the
2:46dark factory was Dan Shapiro's blog
2:47post. He's covering the five different
2:49levels of autonomy for AI coding here.
2:51Most of us, we live our day-to-day like
2:53level three, maybe level four, but level
2:55five with the dark software factory is
2:57the ultimate form of autonomy. And so,
2:59he uses the analogy of driving a vehicle
3:02where with the dark factory, there's not
3:04even a steering wheel. You supply the
3:05final destination or like what you want
3:07to build, but nothing with the
3:09directions for how to actually get
3:11there. And that sounds like the dream.
3:12Hey Claude, hey Codex, here's a massive
3:14app I want to build. Go and create it.
3:16Don't stop until you're done. Make no
3:17mistakes. I mean, that's kind of what it
3:18sounds like. And in theory, it seems
3:21like it would be practical eventually
3:23because as LLMs get better and better
3:24and better, we would reach this level of
3:26autonomy. Like it would actually become
3:28practical. But there's a big problem
3:30with this.
3:31And this big problem relates to exactly
3:33what I was covering in my last video. As
3:35LLMs get better and better, they become
3:37better at building software, but not
3:39that much better at understanding us and
3:41our intent. The bottleneck becomes
3:43getting what you actually want because
3:45as humans, we're never going to
3:46perfectly communicate the features we
3:48want to build, how we want to maintain
3:50our codebase, those kinds of things. And
3:52so, I know it sounds kind of cheesy, but
3:54the dark factory is fundamentally flawed
3:56because as powerful as LLMs get, they're
3:58never going to be able to just read our
4:00minds. Now, I've also heard the
4:01argument, and I do find this
4:02fascinating. People say that LLMs are
4:04going to get so good that their ideas
4:06and their intent is going to become
4:08better than our own, and so we will
4:10fully hand over the development of
4:11software. But, I just don't buy into
4:14that. I think we will always need to be
4:16in the driver's seat because no matter
4:17how good LLMs are, we're always going to
4:19understand what other people want. The
4:21problems we're solving for, our
4:22customers, we're always going to know
4:24best. And to truly keep us in the
4:26driver's seat, we also need to make sure
4:27that we're a part of maintaining the
4:28code base. We can't take ourselves out
4:30of the code entirely because also coding
4:32agents have a weird way of maintaining a
4:34code base. They sprawl a lot, and this
4:37is actually the main argument of Dex
4:39Horthy. He gave a really popular talk
4:40why software factories fail a couple of
4:42months ago, and his main argument is
4:44code base maintainability. Agents just
4:46aren't able to keep it up in the way
4:47that humans do, so we have to be a part
4:49of that process still. In fact, Dex also
4:52did the same thing as me where he
4:53started with the dark factory and had to
4:55dial it back a bit, adding more human in
4:57the loop to make it actually realistic.
4:59Now, when he says software factories
5:01fail, I think that's more in line with
5:02my definition of a dark factory, right?
5:04Like this is the system that doesn't
5:06work. And so, I've taken all the lessons
5:08here that I've explained. It's an
5:09important precursor, and that leads into
5:11the system I have now that is genuinely
5:13working extremely well for me. And so,
5:15thinking about, you know, how can we
5:16maintain human in the loop while still
5:18keeping the process as autonomous as
5:20possible. How do we make it so we
5:22continue to help maintain the code base?
5:24How do we keep the agents on track as
5:26they're ripping through all these
5:26different tickets? All of this is built
5:29in. So, there's one big difference here
5:31between the software factory and the
5:33dark factory, and I don't want to get
5:35super caught up on definitions in
5:36general, but I think there is a really
5:37important distinction here. So, I'll
5:39cover that with a more simplified view
5:41of my software factory, then I'll get
5:43more into how each step works so you can
5:44really see how reliability and human in
5:46the loop is built in. So, the big
5:48difference between the AI software
5:50factory and Dark Factory, besides the
5:52obvious human-in-the-loop that we'll
5:53talk about at the end, is the scope of
5:55work that you build into the system and
5:57how much you plan that with the agent.
6:00So, with the Dark Factory, we are as
6:02hands-off as we possibly can be. We
6:04create just some kind of high-level
6:06document for this massive application we
6:08want to create, and we expect the
6:09factory to split that into issues and
6:11handle them all one at a time or some in
6:13parallel. The point is it is entirely
6:15lights out once we give that more
6:17broad-level description of what we want
6:18to create. And we're even letting the
6:20agent fill in the gap for some of our
6:22intent, right? And that's the big
6:23problem I was talking about where we
6:25don't want the agent to assume our
6:27intent, even if it is a very capable
6:29model. And so, what we're doing with the
6:31software factory is we are giving
6:34individual releases into the system. So,
6:37we're still having it rip through a ton
6:39of issues all by itself, but instead of
6:41it just being like, "Build the entire
6:42thing, go." it's let's actually figure
6:44out like, "What's the next scope of
6:46work?" Let's figure out the tickets and
6:48how we want to split up the work. Then
6:49we let the factory handle it
6:51autonomously. So, the core loop of the
6:53factory is still the same. How I handle
6:55each individual GitHub issue, Jira
6:57ticket, whatever that might be. I have a
6:58very similar way of building and
7:00validating, and it's always going to be
7:02the issue is the input, the pull request
7:04is the output. That remains the same.
7:06It's really a lot of what I do
7:07beforehand and after the fact that makes
7:10the factory a lot more human-in-the-loop
7:12and reliable. And so, with that, let me
7:13go through the three core parts of the
7:15factory here. Now, I'm going to stay
7:16pretty high-level cuz I've spent
7:18hundreds of hours building this system
7:20and testing it. Obviously, I could get
7:22really in the weeds, but I just want to
7:23generally give you inspiration for the
7:26process, talk about what's really
7:27working in the industry as a whole, and
7:29then let me know in the comments if you
7:30want me to cover anything specific here
7:32in a dedicated video.
7:34Big thanks to Port for backing today's
7:36video. They're the platform that helps
7:38teams run their own AI software factory.
7:40Cuz here's the thing, everything I'm
7:42showing you in today's video works for
7:44real engineering teams. I've built it
7:46myself. But when you start to roll
7:48factories out to organizations where you
7:50have hundreds of services, a bunch of
7:52teams, all these people and agents
7:53opening up pull requests, you need
7:55something to help you manage all of it.
7:57Essentially, a managed software factory.
7:59Your agents need to know who owns what
8:01and what standards apply. And you need
8:02to actually be able to see what your
8:04agents are shipping as they do so. And
8:06that is where Port comes in. They map
8:08your entire environment into what they
8:10call the context lake. Here your
8:12services, owners, docs, and standards
8:15all live in one catalog. You can also
8:17connect your agents with Port's MCP
8:19server. Like right here in Cloud Code, I
8:21did so with a single command. And so now
8:23my agent can query all my factories, get
8:26information like who owns what, and get
8:28other context around the factory like
8:30its readiness and which checks are
8:31failing. Cuz we're always going to have
8:33a standard to test against for our
8:35factories. And because every pull
8:36request goes in Port's catalog, I can
8:38ask my agent about a lot of context
8:40around my factories, like how many pull
8:42requests each one is merging per month,
8:44things like how long my laps are taking
8:46on average. These insights are very
8:48valuable if you want to make
8:50improvements to your factories over
8:51time. And of course, I can set
8:52permissions for my agent like this one
8:54here is just read-only. Port is free for
8:57up to 15 seats, no credit card required,
8:59and it's free to try on your own repos.
9:01I'll have a link to them in the
9:02description. So before we get into the
9:04core loop of the factory, there's two
9:06steps. We need to plan the larger body
9:08of work we want to accomplish in the
9:10next release, and then split it into
9:12tickets so we have bite-size pieces of
9:14work that can each go through the full
9:16loop that I'll cover in a second. Now,
9:18the first step here, the planning, I
9:20have intentionally left out of the
9:22factory. Nothing in the harness is going
9:23to walk you through this because I'm
9:25assuming you already have your own way
9:27of planning with your agent. I have that
9:29myself, you probably do, every company I
9:31work with. I don't really want to force
9:33that into the factory. And I want to be
9:35fully in the loop with the planning. And
9:37so, you create your PRD, your
9:39architecture document, whatever you
9:40usually do in your normal way, and then
9:42that becomes input to the factory. So,
9:44we're planning our next release. Now,
9:47the point of the software factory that
9:48makes it very different than the dark
9:49factory is we're not going to try and
9:51just build the entire application. And
9:54so, for greenfield, it's just simply a
9:55proof of concept. You want the factory
9:57to build for that first release. And
9:59then for brownfield, because yes, the
10:01factory works great if you apply it on
10:03top of existing codebases as well, it is
10:05simply a small epic that you're going to
10:07define. Now, I know that small is very
10:10much left up to interpretation for an
10:12epic. Every organization has a different
10:14way that they scope work for an epic or
10:16release. It's really up to you. My
10:18general rule of thumb is I want to
10:19create an epic document that's going to
10:21get split into anywhere between 10 and
10:2320 tickets. I know that's a large range.
10:25Every ticket is going to be a very
10:27different size, but that's just the best
10:29estimate I can give you. And so, if you
10:31build out your factory, you use this
10:32system, you'll start to just build that
10:34muscle over time for, you know, that
10:35size of epic that works well for you.
10:38But the point in the epic is you want to
10:39describe both what we are building here
10:42and what specifically we are not
10:43building. So, having things that are out
10:45of scope is also very helpful for the
10:46factory just to continue to steer it as
10:49it's churning through the different
10:50tickets. And then the other thing that
10:51we define up front that is more unique
10:53to the factory, but is also just really
10:55helpful for any agentic coding
10:57workflows, is defining the holdout
10:58scenarios. And so, essentially what a
11:01holdout scenario is is it is a test that
11:03we're going to run at the end of the
11:05implementation. But the implementer
11:08agent, the builder, doesn't actually
11:10know what the holdout scenarios are,
11:12right? We're withholding that
11:13information. So, that way there's no
11:14bias there. The agent can't overfit the
11:17implementation to make our testing
11:19scenarios pass. This is a big thing that
11:21has boosted reliability in my software
11:23factory because agents are sycophantic.
11:25If they know the test scenarios you're
11:27going to be working with, they're going
11:28to try to engineer the implementation
11:30just to get those to pass. And so, a lot
11:32of times the software isn't really built
11:34well, it's just built well for those
11:36tests. And so we don't want the builder
11:38to know what they are. So we define all
11:40of that in a document. You create your
11:42usual PRD architecture doc, whatever,
11:44and then the factory will take that and
11:47help you split it into tickets. And as a
11:48part of this process, it's also going to
11:50help you do your dependency mapping. So
11:52figuring out things that can be knocked
11:53out in parallel, dependencies that we
11:55have, so we have to, you know, have a
11:57specific order for the laps that we
11:59accomplish. The point is the factory is
12:01going to be applying GitHub labels to
12:03set up all these dependencies to really
12:05set the core loop up for success, so
12:07it's doing things and testing things in
12:10the right order.
12:11Now, we don't have to have human in the
12:13loop here. A lot of what I'm building
12:14into my software factory is sort of like
12:16toggles that we have for human in the
12:18loop, so maybe we don't review PRs
12:20before we merge into staging or we do,
12:22or we do review the split before we
12:24start building or we don't. And so every
12:26organization, every person has different
12:28needs, right? And it also depends on the
12:29complexity of what we're building in
12:31this specific release. So there might be
12:32some releases where we're dealing with
12:34compliance and regulation, we really
12:36have to make sure things are solid. So
12:37we want to review the split, we want to
12:39review the PRs at the end, but we still
12:41want the system to just churn through
12:42the issues once we do create them. And
12:45then there are other releases where
12:47we're maybe just handling a big backlog
12:49of bugs. And in that case, maybe we just
12:51want to entirely trust it to split into
12:52tickets and merge things right into our
12:54staging environment, right? Like there's
12:56there's a sort of dynamic need that we
12:58have where it's not always human in the
12:59loop needs to be this place, this place,
13:01and so I've created the factory to be
13:03very adaptable. Now, trust me, I wish I
13:05could really get into what this factory
13:07looks like in action, showing the runs,
13:09showing my plans, the splits, and what
13:10the issue lists look like, but I did try
13:13recording a version of this video where
13:14I showed all that, and it got way too
13:16long. So I have to stay high-level for
13:18now, but definitely I will be getting
13:20more into this in future videos. The
13:22point is here that after the planning
13:24and the splitting, we have a list of
13:25GitHub issues, and we're just going to
13:27rip through them with the core loop of
13:29the factory doing all the planning and
13:31validation. So, zooming in here, we
13:33start with the GitHub issue. We can run
13:35many of these labs in parallel because
13:37we did the dependency mapping, we know
13:38things that we can knock out at the same
13:40time. So, each time we run a lap, it is
13:42going to choose a single GitHub issue to
13:44work on. It'll apply a sort of claim
13:46label, so other parallel labs don't pick
13:48it up, and then we make a decision up
13:50front because something that I noticed
13:52with the first version of even my dark
13:54factory is when I did the more
13:56comprehensive workflow to handle every
13:59single implementation, there was these
14:00basic tickets that would take like 30
14:02minutes or 20 minutes to implement where
14:04they really didn't need that. And so, I
14:05make the decision up front actually
14:07using Jev, is this a simple thing where
14:09I just have a faster build workflow, or
14:11is it more complex where we do want to
14:13have the stages for iteration and things
14:15like that. And so, it makes the
14:16decision, but either way, once it's done
14:18building, it's going to go through the
14:19validation and code review that also
14:22understands, you know, based on the
14:23complexity of the implementation, how
14:25much time do we have to spend here. And
14:26so, there's both review and validation
14:29cuz they are different things.
14:30Validation means as a part of the
14:32harness, I give it the tooling to
14:33actually test the application as a user
14:35would. So, it replaces a lot of my
14:37manual testing because we do want it to
14:39be more autonomous. That's the whole
14:40point of the software factory. And so,
14:42clicking buttons, entering text in the
14:44chat input, whatever it is for the
14:45application. And then after we do the
14:48code review, and this is more analyzing
14:50like since we know that the app is
14:51working, now let's make sure that it's
14:52simple, the code is maintainable, it was
14:54implemented correctly, right? Like those
14:56kinds of more software engineering
14:58decisions, the the more classic code
15:00review. And then for anything that it
15:02deems incorrect in either one of these
15:03stages, it sends back to the
15:05implementation workflow to fix, right?
15:08So, even for a complex implementation,
15:10it's just going to go through the simple
15:11workflow to address any feedback. And of
15:13course, there are max retries, so we
15:15don't get stuck in an infinite loop with
15:17the factory. And so, then once we are
15:19done with our reviewing and potential
15:20iterating, then we allow to merge the
15:23pull request into whatever branch goes
15:26into our staging environment only if CI
15:28is green and the reviewed code is good.
15:31And then for this entire process here, I
15:33have a manager agent that runs on a
15:35periodic basis like every two or three
15:37hours and it's kind of surveying the
15:40success of all the laps together because
15:43this is actually the most recent thing
15:44I've added into the factory. Before I
15:46had this, these agents are never really
15:48communicating, right? Like each
15:49individual lap isn't really
15:50understanding if dependencies are
15:52changing or if one pull request is
15:54getting really stuck. Like if we hit the
15:56max retries and an agent really is
15:58getting stuck on a problem, we need
16:00something that kind of peers over the
16:01entire process and can realign issues or
16:04kind of like hijack a workflow and
16:06finish it if needed. And this has also
16:09really increased the reliability because
16:11also I've set it up where it will
16:13actually alert me if I really do need to
16:15come into the system. Because as much as
16:17we want this to be autonomous, it's not
16:19a dark factory anymore. We are okay with
16:21once in a while us having to come in and
16:24review the split or, you know, sort of
16:26rescue a workflow. It happens once in a
16:28while. So usually I'll just have my main
16:29Claude code agent like my second brain
16:32help me when the manager is alerting me
16:33of these issues. And then of course the
16:35holdout scenarios come in because we're
16:37running those for the app validation.
16:39And as a part of the factory, I have my
16:41core context. I call it my mission.md or
16:43engineering.md. It's the things that are
16:45out of scope that I never want to build
16:47and just higher level direction for how
16:49I want the agent to operate in my code
16:51base. So kind of that plus the global
16:52rules together. And I'm making sure that
16:54I never edit these files. That's a part
16:56of my review making sure that we never
16:57touch those because I don't want the
16:59factory to be able to change its own
17:01permissions and behavior. And then the
17:03last thing right here is for every
17:05single one of these nodes, I'm using
17:06different models and even different
17:08providers because this factory I've made
17:10it very token efficient. We don't always
17:13need the most reasoning possible. So
17:15definitely for a lot of the planning
17:16that I'm doing like right here, I am
17:18using you know Opus 5.5 or GPT-6 Astra.
17:21But for a lot of these individual nodes,
17:23like using Jev to pick the task and then
17:25for simple maybe just using Sonnet 5.5.
17:28I mean even for shipping these different
17:30nodes in here, I don't have to use the
17:31best model. And so I've done a lot of
17:33that experimentation optimizing for the
17:35sake of tokens. I'm running hundreds and
17:37hundreds of laps every single week not
17:38hitting my rate limits. All right, and
17:40then the last part of the process here,
17:42this is where we come back in to review
17:44things. And so we build up all the
17:46issues for the epic within our core
17:48loop. We merge things into the staging
17:50environment. This is where we get to
17:52validate things ourselves cuz we need
17:53the step to verify our intent. And so we
17:56test things ourselves and then anything
17:58that's off like if something wasn't
17:59built how we actually wanted it to be,
18:01all we have to do is simply create a new
18:03issue in the repository and the factory
18:05will pick that up automatically just
18:07like it did with the initial split from
18:09our epic document. And so it's very easy
18:11for us to sort of test the batch of
18:13features in the epic and then give batch
18:15feedback. We we could you know create
18:17four or five issues, whatever, and then
18:18it'll just rip through all of these and
18:20then we can test staging again. And then
18:21we go to the production once it's right.
18:24And so the dark factory pretty much goes
18:25straight to production, but the software
18:27factory, I found this to be very
18:29important. So we have true human in the
18:31loop, but at the same time we get to
18:32review a lot at once. And so we are
18:34solving that bottleneck at least
18:36partially for the human review at the
18:38end. And then once we're happy with
18:40what's in production, we can simply just
18:41go on to the next release and repeat
18:43this process going through the planning,
18:45core loop, and the staging and
18:47production. And that's it. That is the
18:49AI software factory at a high level. And
18:51so of course, like I already said, let
18:53me know if you want me to cover any part
18:54of this in more detail, but hopefully
18:56you found this inspirational just take
18:57aways for your own AI coding workflows
18:59or if you want to build a software
19:00factory yourself, it really is the
19:02future of agentic engineering. And so if
19:05you appreciate this video, you're
19:06looking forward to more things on an AI
19:07software factory and AI coding in
19:09general, I would really appreciate a
19:10like and a subscribe. And with that, I
19:12will see you in the next video.

About this transcript

This page contains the full transcript of My AI Dark Factory Dream Is Dead (Here's What Actually Works) by Cole Medin, generated from the public captions YouTube serves with the video. The transcript has 4,400 words across 617 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.