YT YouTube2Text

YouTube2Text → Transcripts → Run Your AI Software Factory Like a Real Engineering Org (Here's How)

Run Your AI Software Factory Like a Real Engineering Org (Here's How) — Transcript

by Cole Medin · 4,175 words · 589 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00My AI software factory is hands-down my
0:02biggest project right now. I've spent
0:04hundreds of hours refining it over the
0:06course of this year, and it's even
0:07gotten to the point where I've started
0:08to help companies incorporate this kind
0:10of system. Not replacing all their
0:12engineering, but using a software
0:14factory for certain teams and certain
0:16applications. But, as I've been going
0:18through these engagements, it's become
0:20quite apparent to me that the factory by
0:22itself is not enough. When you get to an
0:24enterprise level, or you just have a lot
0:25of applications, you need a sort of
0:27wrapper around the factory to make sure
0:30that it has all of the guardrails,
0:31context, and observability that you
0:33need. Because the AI software factory
0:36represents the most autonomous form of
0:38AI coding. We still have some human in
0:40the loop so we can communicate what we
0:41want, we plan that release with the
0:43agent, and then we send that into the
0:44harness to split it into tasks and
0:46implement them in a loop. Now, it's not
0:48a black box because we're engineering
0:50the harness that goes through this, but
0:52you can see how when we really start to
0:53scale the factory, we do need that
0:55harness to help us manage everything.
0:57Now, I could [clears throat] always
0:58build this wrapper myself, but like I'll
0:59show you in this video, it's not that
1:01simple. There's a lot more to it than
1:03what meets the eye. And I don't want to
1:05be focused on that part of the factory.
1:07I want to be focused on the underlying
1:08engineering loop that's doing the actual
1:10development. And so, I've been searching
1:12far and wide recently to find a platform
1:15that gives me this wrapper, but still
1:17allows me to run my own custom harness
1:19for the development. And I have found an
1:22exact match. Port is the platform that
1:24gives me exactly what I need. It gives
1:26me the layer of context and
1:28observability and guardrails over my
1:29factories. Like it's integrated directly
1:31with my pull requests, it's watching
1:33everything, it's giving me a health
1:34check across my repository, a lot more
1:37that I'll show you here. But all the
1:38while, I still get to run my software
1:40factory. And so, I'll show you how this
1:42all works right now. Not just to give
1:44you a Port tutorial, but to show you how
1:46they're incorporating all these
1:47different concepts that really start to
1:49matter once you have a lot of factories
1:51and you care about like ownership of
1:53different components and how you can
1:55have a central layer of context for all
1:57your factories. There is a lot of cool
1:59stuff to get into here.
2:00And let me tell you, it was a hard
2:02requirement because of the effort I put
2:04into this harness for me to not have to
2:06change it when I incorporate a platform
2:08like Port. And so, the entire
2:10architecture that I showed in one of my
2:11last videos that I'll link to right
2:13here, I'm still using it exactly. I'm
2:15just using Port to add the wrapper
2:17around it for four core things that
2:19we're going to cover today. We'll talk
2:21about the agent and skill registry, the
2:23context lake, the standards we can apply
2:26to every single factory, and how we have
2:28the control to match each factory to
2:30those standards as they evolve. And so,
2:33the biggest takeaway for you as we go
2:34through everything here is just learning
2:36from Port. What does it take to take a
2:38software factory to an enterprise level,
2:40and what are the different components?
2:41Because let me tell you, Port knocks it
2:43out of the park. There is a reason why
2:45I'm partnering with them, and I'm so
2:47interested in incorporating my software
2:48factory with Port. They're really doing
2:51it the right way, and I've even had some
2:52conversations learning from them how I
2:54can incorporate my software factory with
2:56their platform, and now bringing that to
2:58you so you can take inspiration from
2:59everything here. And so, if you haven't
3:01seen it yet, or just as a refresher, I
3:03want to start by going over my
3:04underlying software factory, the core
3:06engineering loop. And then this will set
3:08the stage for Port and the components
3:09that I have on top of it. And so, let me
3:12be very clear. If you are just getting
3:14started incorporating a software factory
3:16or building one, I encourage you to keep
3:18it simple and stick to the core loop.
3:20But everybody is going to have to scale
3:22at some point, and that is when it's
3:23going to be really important to have
3:25things like an agent and skill registry
3:27and a context lake. And so, we'll get to
3:28that, but let's start with the
3:29underlying loop here. So, the way that I
3:31always start with my software factory is
3:33I begin by planning the next release.
3:36And so, for Greenfield, this means
3:38planning the proof of concept for my
3:39application. For Brownfield, it means
3:41planning the next epic, the next set of
3:43tasks, whatever you want to call it.
3:45Generally, something that can be split
3:46into 10 to 20 tickets. And so, we're
3:49very much a part part process with the
3:50agent here creating something like a PRD
3:53document. It's really up to you what you
3:55create. The point is you have some kind
3:57of context to go into the core loop of
3:59the factory. Because now the agent is
4:01going to take that planning and figure
4:03out exactly how to split it into tasks
4:05and even do dependency mapping so it can
4:07figure out what loops it can run in
4:08parallel versus other work that has to
4:10wait for different tickets to be
4:12finished. And so then with every single
4:14issue or ticket that it creates from the
4:16split, it's going to send each one
4:17through the core engineering loop. Then
4:19every time the core loop runs, and I
4:21usually just have a cron job to trigger
4:23it on a schedule basis, it's going to
4:24start by picking the next issue to
4:26implement based on priority order and
4:28the dependency mapping. And then it's
4:30going to take that and go through the
4:31full plan, implement, and validate
4:33cycle. So it's going to plan around the
4:35bug or new feature, whatever the issue
4:37is describing. It's going to implement
4:39based on the plan and then go through
4:41very comprehensive testing. So it's
4:42going to validate the application
4:44exactly as a user would, right? Like
4:46what is the user journey we can go
4:47through to test this new feature. And
4:50then it's going to spin up a separate
4:51agent to perform a code review on the
4:53pull request and only once all the
4:56checks are green with our CI and the
4:58code review from the other agent, then
5:00we can merge. And typically, we're not
5:02just going to merge straight to
5:03production. I don't really believe in
5:04that for software factories. It's more
5:06get it on a branch that's going to be
5:08deployed to staging. And so we take the
5:10entire release, we run a bunch of loops
5:12to go through all the tickets, and we
5:14get everything set up in a staging
5:16environment. So now we at the end can go
5:18through and validate things in batch. We
5:20don't have to sit there and code review
5:22and test every individual issue. We can
5:24test everything at once. And then if
5:25there's anything that's off with any of
5:27the features or whatever we wanted to
5:28build, we just send that in as new
5:31issues to the factory. So it can
5:32automatically pick up any of our
5:34feedback, implement that, and then we
5:36can iterate. Typically just a couple of
5:37times, and then once everything's
5:38looking good in the staging environment,
5:40we ship to production. So there you go.
5:42That is a couple of minutes, just a very
5:44high-level of my underlying engineering
5:46loop. I covered a lot more in that video
5:48I linked to earlier, but now that you
5:50understand this, we can talk about the
5:52problems that we encounter when we
5:54really start to scale the factory and
5:56run multiple. And so right now there are
5:58four core components of Port that I want
5:59to cover with you just to show you what
6:01it truly takes to scale a software
6:03factory. And they help you with a lot
6:05more than just this as well. There are
6:07so many considerations that go into
6:09scaling a software factory. Like another
6:11one is thinking about ownership. Because
6:13when we have our factory running with
6:15all these different agents, a lot of
6:16times they're borrowing our credentials
6:18for things like our GitHub account. And
6:20so when something breaks in production
6:21and we trace it to a commit from a
6:23certain account, we don't really know,
6:24is it the engineer or is it their
6:26agents? And so that's why Port also
6:28helps us by assigning identities to all
6:30the agents running in our factories. And
6:32speaking of our agents, that brings us
6:34to the first one here with our registry.
6:37Every agent and every skill has an owner
6:39and we're going to sort of version
6:40control everything within Port so that
6:43we can pull it into our factories as
6:45needed. So Port calls themselves the
6:47harness around your software factory for
6:50orchestration, context, governance, and
6:52measurement. And this in my mind is the
6:54first big component for this. So let me
6:56show you in Port what this looks like.
6:58There's a lot of inspiration we can take
6:59here for how we manage our skills and
7:02agents for our factory. And so what the
7:04skill registry gives me is a central
7:06place to store every single skill that
7:08I'm using in any of my factories. And
7:10it'll help me identify the number of
7:12distinct versions that I have. It's bad
7:15for this number to be high generally
7:16because it means that my skills have
7:18drifted across my different factories.
7:20As much as I can, I want to maintain one
7:22version of every skill and then
7:24distribute it from the registry.
7:26And by the way, the reason the skill
7:28registry is so important is because
7:29every single step within my underlying
7:32engineering loop is driven by a skill. I
7:34don't just have a couple of lines for a
7:36prompt. I have an elaborate skill that's
7:38defining my exact process for everything
7:41here. The only reason I trust the
7:42software factory is because I'm spending
7:44a lot of time engineering the harness
7:46and the skills and agents are a very big
7:49part of that. So, we have the skill
7:50registry and going to their demo
7:52environment, we can also see a better
7:53example what it looks like to assign
7:55ownership and the publish status, even
7:57building out our skill life cycle for
7:59how we can rerun certification or making
8:01updates to our skills, rejecting
8:03publishes, deprecating them over time,
8:05right? Like managing the life cycle of
8:07our skills for our factories is also
8:08very important. And we're going to be
8:09doing a very similar thing for the
8:11agents we're running throughout as well.
8:13So, the skills are simply our workflows,
8:15but agents are roles within our factory.
8:18Like we have the planner that helps us
8:20build that release up front, the backlog
8:22splitter that takes that plan and splits
8:23it into tickets. And for every single
8:25one of these agents, we have the
8:26platform we're using for it. Like some
8:28parts of our factory we're going to
8:29drive with Claude code, other parts we
8:31could drive with Codex or Pi. We have
8:33the model defined for each one, the
8:35permissions for what it's able to touch
8:37or what it can't do, the approval
8:39status. And so, we have this registry of
8:41everything that's driving the entire
8:43underlying harness. And so, I still get
8:45to define the agents and skills for
8:46exactly how I want to run my factory.
8:48Portless helps me manage it all and make
8:50it very easy for me to reuse these
8:52things, pull from the registry for every
8:54factory that I'm running. So, this tab
8:56is my actual agent registry, but going
8:58to their demo environment again, you can
9:00see what it looks like when we have more
9:01runtime platforms. Like running part of
9:03our factory in GCP Vertex AI or AWS
9:06Bedrock, a lot of what enterprises are
9:08doing. And just like we have the skills
9:10life cycle, we also have the agent life
9:11cycle for how we're creating new agents,
9:13promoting them, updating their
9:15permissions. You want to have an owner
9:17within the organization that's managing
9:19these things for the different
9:20factories. Now, the next really
9:22important component to cover here is the
9:24context lake. And this is probably my
9:26favorite part of Port, even though I
9:27love it all. Because here's the thing,
9:29our underlying engineering loop, what we
9:31have consisting of our agents and skills
9:34that we pull from the registry, this is
9:36really just our process. It has nothing
9:38to do with business context, and we need
9:41a lot of that if we really want to set
9:43our factories up for success. And so,
9:44obviously, connecting our factories to
9:47our GitHub repos and directly to CI, but
9:50also platforms like Jira and Slack, so
9:52it has business context and even
9:54knowledge of things like who owns the
9:56different applications and features that
9:58we're building. Now, going back to their
10:00demo environment really quickly, they
10:01have a good really good example of what
10:02it looks like when we have a rich
10:04context lake. Because as our factory is
10:07working with the higher level planning
10:08we do with it, and even as it's going
10:10through the individual tickets, there's
10:11a lot of information that's critical,
10:14where it can't just get that by looking
10:15at the code base. It needs extra
10:17business context from things like Slack
10:19or SharePoint or Jira. For example,
10:21knowing is a human required before we
10:24would maybe approve the merge, cuz some
10:25things you might want to be more
10:26autonomous, other things you need more
10:27of a review because the blast radius
10:29might be high. And again, determining
10:31this kind of thing, you can't always do
10:32it from just the code base, especially
10:34things like the service tier in the
10:35domain. Like, this is a
10:37marketing-related feature. Here's the
10:38owner that we'd want to send approval
10:40to. Having the factory not just work
10:42entirely in isolation, but figuring out
10:44the places where we need a human,
10:46especially for certain things, that is
10:48what Port helps us manage with the
10:49context lake.
10:51Now, of course, Port also has very rich
10:53information on the code base itself with
10:54your pull requests and the status of the
10:56factory and CI and all of that. But what
10:58I want to show you right now is more of
10:59the business context that we need to
11:01especially pull in when we're first
11:03doing the planning. So, when we're
11:05actually having the conversation with
11:06our agent establishing the plan before
11:08we send it into the factory, we're still
11:11going to be using Port because of the
11:12context lake. And so, the exact prompt
11:15that I have here is not really important
11:16to go into a detail. I'm essentially
11:18just planning what I want to create up
11:19front with my agent, and I'm using the
11:22Port MCP server to pull key information
11:25from the context lake. And here we're
11:27pulling things like the ownerships of
11:29the repositories, templates we can use
11:31as a starting point, the readiness of
11:33the repo. There is so much context that
11:35we're pulling here. And of course, you
11:36can tune how you're using the Port MCP
11:38server to figure out what you're
11:39pulling, what you're searching. And in
11:41Port, you can decide the platforms that
11:43you're connecting, like GitHub and Jira
11:44and Slack. The important thing here is
11:46after it makes all these calls, it's
11:48just in a better place to help us plan
11:50for the factory run itself and exactly
11:52what we're going to build. And so right
11:54here, like it figured out the owners,
11:55the factory team, the repos at bronze,
11:57that's the readiness of the repo to
11:59become like we'll talk about in a little
12:01bit. And so all this now goes into us
12:03having a conversation with our agent
12:05using Port's context to assist to create
12:08that PRD. That is what we plan, and then
12:10this is what we send into the factory.
12:12And by the way, none of this is
12:13theoretical. I've already built a lot of
12:15apps with my software factory and Port.
12:17Like this is the demo that I have for
12:18today, just one of many applications
12:20I've built. And so this is a team
12:22knowledge base, a central place for
12:24people to get answers across
12:25repositories. It's not the most
12:27elaborate application, but I built this
12:29over just two days using Port and my
12:31software factory, completely hands-off
12:34after I built that initial PRD. And so
12:37it's a rag application under the hood
12:39where I can ask questions to the central
12:40knowledge base, it'll search through
12:42everything, and you'll even have source
12:44citation. And it looks pretty good
12:46overall. It's not the fanciest thing
12:48like I said, but this is pretty
12:49impressive for just letting it rip for a
12:51couple of days. And if we look in Port,
12:53we have full management of everything.
12:55And so all the pull requests that I
12:56created, there's 20 in total for this
12:59release that I created a PRD for. We can
13:01see all of them in Port with their
13:03status, and we can click into this and
13:05get a lot more detail. And we also have
13:07the scorecard. This is the readiness of
13:10our repo, the things that we need to
13:11have in place so it's truly ready to
13:13become an autonomous factory. And this
13:15is the standard that we can apply to
13:17every software factory. So essentially
13:19what we have here is a readiness check.
13:21Is our repository ready to become a
13:24software factory? Are we ready for that
13:26level of autonomy? Now, Port can't check
13:29everything because we do bring our own
13:31engineering loop, but it is able to
13:32check the high-level things. Like, do we
13:34have an owner defined for the
13:35repository? That's very important like a
13:37part of our context like like we're
13:38talking about earlier. Does it have a
13:40readme, a PR template, does it have a CI
13:42workflow, and is CI passing on main? If
13:44all these things are checked, then we
13:47are gold and we are ready to implement
13:49the factory into this repository. And
13:51so, for all the other repos that I have
13:53that are not gold, this is usually how
13:54we'll start where we have a sort of
13:56laundry list of things that we have to
13:58address in our repository before we are
14:01ready to just turn on the autonomy.
14:04Now, not only does this set of standards
14:05give us a checklist to work against as
14:08we initially set up our factory, but
14:10also every single time we have a pull
14:11request merged, Port is going to grade
14:13that merge against the standards that we
14:16have here. So, for example, when CI is
14:18failing, we know we have to do something
14:20about it. It's reported in Port and it's
14:22even going to go through a workflow
14:24where we can approve addressing this
14:26problem, creating an issue in the
14:28factory, just like all the other work we
14:30do to address the problem in the
14:31codebase itself. And so, I actually have
14:33a demo prepared to show you what this
14:35looks like.
14:36All right, so I've recorded this ahead
14:37of time to very quickly show you both
14:39the scorecards and controls working
14:41together in Port. These are the last two
14:43big components I want to show you. And
14:45so, what we have here to start is a
14:47workflow. We can build workflows in Port
14:49to help us manage our context like our
14:51factories, everything. And so, I'm going
14:53to go ahead and play this here and just
14:54kind of talk through everything that's
14:55happening. And so, this is our workflow
14:57to fix our scorecard whenever there is
15:00an issue. And so, what I'm doing here is
15:02I'm purposely making CI fail in my
15:05workflow. And we'll see that Port is
15:07automatically going to detect that and
15:09that's going to trigger the workflow so
15:11it proposes to me. I have to approve it,
15:13but it's going to propose creating an
15:15issue to address CI failing in the
15:17repository. Now, obviously this is a
15:19little bit forced because you can see
15:20that I'm making a commit to purposely
15:22break it, but the point is your CI
15:25starts failing all of the time in a real
15:27code base because you always have tests
15:29that are flaky or things that break when
15:31you're making changes. And so we want
15:33Port to identify that. That's what we're
15:34showing right here. And so now we have a
15:36back-end failure in our CI. And so going
15:39back to Port, look at that. It went from
15:41gold to silver. The readiness is down
15:43because we have CI failing. And so now
15:45we can see an execution of this
15:47workflow. It is now waiting for our
15:49approval. It detected the failure and
15:51now it's saying, "Hey, I want to create
15:52a GitHub issue so that we have the
15:54factory automatically pick it up and fix
15:56the CI." So we have sort of this
15:58self-improvement using Port to guide it.
16:00Okay, so we approve it, which means now
16:02we're going to go and automatically
16:04create the issue in GitHub. I'll pop
16:06over there so you'll see that we have
16:08the new issue, which means the next lap
16:10is going to pick this up and then fix
16:12the CI automatically. And then I'll kind
16:13of skip ahead here to show what that
16:14looks like. So we get to the point where
16:16the lap has addressed it, it's merged,
16:18our CI is green, and then going back to
16:21Port, we go back from silver to gold
16:23because our CI is passing. And so it's
16:26autonomous with human in the loop at
16:28least for that fast approval, but
16:29besides us quickly approving things,
16:32everything is self-improving. So it's
16:33able to detect when CI is failing. So
16:36especially if your CI is elaborate
16:38enough where you're confident green CI
16:40means the application is in a healthy
16:42state, this is a really important
16:44addition that Port gives us to make our
16:46repository self-improving.
16:49So there you go. With that, we've
16:50covered all four of the core parts of
16:51Port that I wanted to hit on today,
16:53helping us evolve our software
16:55factories. And the most important thing
16:57with everything here is the human in the
16:59loop. I really respect and resonate with
17:01how Port has built the human in the loop
17:03throughout the entire harness. Because
17:05when I first started building my
17:07software factory, it was more of what I
17:09call a dark factory where there's no
17:11human in the loop at all. Not doing the
17:13planning up front with the agent, no
17:14staging environment, it's going straight
17:16to production. It's not going to ask for
17:18our approval when it adjusts things
17:20within CI, right? Like it could adjust
17:22its own harness. It could literally do
17:23anything and we have no visibility. And
17:26I purposely moved away from that. I
17:28still want to make the process as
17:30autonomous as possible, but it has to be
17:32within reason. There needs to be certain
17:34things where we have human in the loop.
17:36And so Port's philosophy with their
17:37larger harness is we build in the
17:40owners. We have the context like we do
17:42the planning. We have the approval when
17:44we are working with the standards
17:46because we need human in the loop and
17:48then we just remove that when we see
17:50fit. But for the most part, we aren't
17:52just going to be completely lights out
17:54like I used to do with the dark factory.
17:56So just another golden nugget to end
17:58things off for you here. And so I hope
17:59that this video was useful for you
18:00overall giving you inspiration and ideas
18:03for truly scaling your software
18:05factories. And Port just has a great
18:07platform that's an option for you to do
18:09so. And like I said, I did work with
18:11Port on this video. And Port is free for
18:13up to 15 seats. No credit card required.
18:15You can go ahead and try it right on
18:17your repositories. You can try it for
18:19free with a link that I'll have in the
18:20description. And thanks to Port for
18:22working with me on this. And so with
18:24that, if you found these ideas useful
18:26for you, you're interested to follow my
18:27content on agentic engineering and
18:30building out my software factory, I
18:31would really appreciate a like and a
18:33subscribe. And with that, I will see you
18:35in the next video.

About this transcript

This page contains the full transcript of Run Your AI Software Factory Like a Real Engineering Org (Here's How) by Cole Medin, generated from the public captions YouTube serves with the video. The transcript has 4,175 words across 589 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.