YT YouTube2Text

YouTube2Text → Transcripts → I Built A Claude DevOps Agent To Run My Software Factory

I Built A Claude DevOps Agent To Run My Software Factory — Transcript

by Owain Lewis · 3,772 words · 533 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00Today we're talking about how you can
0:01use AI agents to manage and [music]
0:02operate your software in production. To
0:04me, this is one of the most interesting
0:05and exciting use cases for autonomous AI
0:08agents because a single [music] outage
0:09can cost a business upwards of $100,000.
0:12The great thing about using AI agents
0:14for this use case is they're pretty much
0:15superhuman when it comes to searching
0:16through large amounts of data to find
0:19[music] specific problems with your
0:20software. So, in this video, I'm going
0:21to show you how you can build an
0:22autonomous AI system that manages and
0:24monitors your production [music] systems
0:26using the Claude agents SDK. I'll walk
0:28you through the architecture and give
0:29you a demo so that you can see how it
0:31works in practice. If you're new to the
0:32channel, my name is Owen. I've been a
0:33software developer for the past 20 years
0:35and I build autonomous AI systems like
0:36this every single day. All of the
0:38resources for the video I'll link for
0:39free in the description below. So,
0:41[music] let's get into it. So, just to
0:42show you how important or how serious
0:44the problem is when it comes to software
0:46outages, if you think of a company like
0:48Amazon, which is kind of the extreme
0:50case, [music] if they're down for an
0:52hour, it cost them $72.8 $8 million per
0:55hour, which kind of actually makes sense
0:57if you think how many people depend on
0:58Amazon, how many transactions and sales
1:00they make during that time period. If
1:02their software systems are not
1:03available, not only do they lose
1:05revenue, but they also have an impact to
1:07their reputation. So, if Amazon is down,
1:10customers get frustrated. And for a
1:12company like Amazon, it's okay. But for
1:13a smaller business, if you think like
1:15the Claude API, for example, if that's
1:17down too often, you start to lose trust
1:19in that company and you're more likely
1:21to go to a competitor that you trust to
1:23be reliable. So reliability and uptime
1:26is such a critical thing for any any
1:28software business. And you can see some
1:30other numbers here which are really
1:31interesting. So this is kind of a bit
1:33crazy. So $66,000
1:36to $23,000 per minute of downtime. So
1:40that's pretty kind of crazy in terms of
1:42how much impact this has on a company
1:43like Amazon. For a smaller company,
1:45obviously the impact is less, but it's
1:47still pretty significant. The average
1:48business is going to be majorly impacted
1:51if their software systems are not
1:52available for a large or long period of
1:55time. So the idea here behind an AI S or
1:58site reliability engineer is that you
2:00have an AI agent that is a frontline
2:02responder to any incident or production
2:05error that happens with your software
2:06system. So say if something breaks in
2:08production, the agent will then
2:10immediately start investigating that
2:11issue, figure out what's going wrong.
2:14It'll look through the logs, the
2:15metrics. It'll look at are there any
2:17recent deployments or changes to the
2:19system. It will use all of that evidence
2:21then to come to some kind of conclusion
2:23and it can give you a recommendation of
2:24what action to take. And as we start
2:27deploying more and more code with AI
2:28agents, this kind of system becomes much
2:30much more important. Because if you
2:32think about how much more code we're
2:33releasing to production, a lot of it is
2:36generated by AI. So we need these kind
2:38of safety systems in production. If
2:40there are any mistakes or regressions or
2:42bugs or problems with our system, we
2:44need AI agents to help us move faster to
2:47resolve those issues. So one interesting
2:49framework that I really like when it
2:50comes to AI agents and autonomous
2:52systems is this ODA loop. So basically
2:54what we're doing is we're just observing
2:56what's happening. So we're we're paying
2:57attention to the environment. We're
2:59orientating ourselves by looking at all
3:02of the information and figuring out
3:03where we are, then making a decision
3:05about what to do and then taking action.
3:07So, this is how these autonomous AI
3:09agent systems work. We look for signals
3:11in the environment. So, when something's
3:13broken, there's an alert firing. We then
3:16use our tools to find the information we
3:18need. We interpret that evidence. We
3:20make a judgment call and decide what to
3:22do. And then we we take action. So,
3:24either notifying a human or fixing
3:26something ourselves. Basically, those
3:28are typically the the two options you've
3:29got. So, there's a couple of really good
3:31articles that I recommend checking out
3:33on this topic. So, the anthropic team
3:34talk about this a lot. Using AI agents
3:37or using Claude to act as an on call
3:39first responder. So, this is a great
3:41example where a team can ask the agent
3:43questions. The agent is constantly
3:46monitoring what's going on in production
3:48and the anthropic team use this quite a
3:50lot. There's also another good article
3:52here from Google. So all of the major
3:54companies right now are leaning into
3:56this idea of having an AI agent act as
3:58like a first line of defense when it
4:01comes to operations. So you can see here
4:03there's a couple of interesting ideas
4:05here. I think the main one that's
4:06interesting here is the incident
4:07management one. So basically whenever an
4:10incident happens an AI agent can
4:12coordinate what's going on. It can do
4:13things like handoff. It can create
4:15documents and incident postmortems. It
4:17can analyze an incident and then it can
4:19also do an investigation as well. So you
4:22can see here the AI agent is going to do
4:24some kind of investigation. Again, it's
4:26just looking at logs, looking at
4:27monitoring, tracing, and things like
4:28that, and then working out what the
4:30agent thinks is wrong with the system
4:32and then giving you a recommendation for
4:34how to fix it. This was a really
4:36interesting repo from Anthropic. So if
4:38you want to check this out, they have a
4:40a few resources here that I think are
4:42really interesting. They have things
4:43like skills that you can check out here.
4:45So this is the Anthropic on call kit. So
4:48you can see here we have a triage prompt
4:50and a skill. So you can take inspiration
4:52from this. So this is basically what
4:54we're going to be doing in the video
4:55today. So we have investigate an alert
4:57or incident in the channel. So this is a
4:59skill that allows the agent to figure
5:01out how to diagnose and figure out
5:03what's wrong with a software system. And
5:06you can kind of take this as
5:07inspiration, but obviously you're going
5:08to want to customize this for your own
5:10business, your own applications, and you
5:12need to wire in your own tooling as
5:14well. All right. So this is the
5:15architecture of what we're going to be
5:16building. So the idea behind this system
5:18is you have an alert. So when something
5:20breaks in production, we generate an
5:22alert or some kind of metric or trigger,
5:24we'll fire an alert. So say if your API
5:26is responding slowly or if there are
5:28some errors in the system, that will
5:30then trigger an alert and we send that
5:32alert into a queue and then we send that
5:34down to an AI agent. So in this case,
5:36we've got the Claude agents SDK
5:38listening to the queue and then
5:40responding by investigating. So the
5:42agent is going to use the tools it has
5:44available to search through the logs,
5:46look at the metrics, look at the
5:47deployment history, try and analyze
5:49what's going on in the system and then
5:51it's going to send those findings to a
5:53web console. So this allows us as the
5:55operator to see what's actually going on
5:57and then the agent is basically going to
5:59help us recover the system. either it's
6:01going to take action autonomously by
6:03fixing something like rolling back your
6:05application or it's going to ask you
6:07it's going to give you a recommendation
6:09of what to do and then it's going to
6:10help you take action to to fix that. So
6:12you really want to think about whether
6:13you want human in the loop or not. I
6:15think for these kind of use cases you
6:17generally want a human in the loop
6:18because although AI agents are really
6:20really powerful they do often jump to
6:22the wrong conclusions. If they see some
6:25kind of data or some kind of signal they
6:26maybe overindex on that signal too much.
6:29So this is actually one of the most
6:30challenging domains for building AI
6:32agents in general because figuring out
6:34why a system is broken is actually quite
6:36complicated. You need to look at lots of
6:38different signals. You need to use quite
6:40a lot of intelligence and judgment. So I
6:42think it's best to combine human
6:44judgment with like scraping capabilities
6:46of an agent. I think that's where you
6:48get the most leverage from these
6:49systems. So we can do quite a lot with
6:51this agent. So this agent can
6:52autonomously listen to an event and then
6:54take action. So if something is broken
6:56in the middle of the night, the agent
6:58will just kind of fix the issue for you,
7:00which is one way to do this. The other
7:02way to do it is have the agent as more
7:03like an assistant. So you can ask the
7:05agent questions like why is the checkout
7:07failing? Search through the logs, find
7:09me evidence, and this is going to speed
7:11up diagnosis. So looking through this
7:13information as a human takes a really
7:15long time. It's incredibly timeconuming
7:17to remember where you need to look,
7:19which dashboard to check, stuff like
7:21that. So an AI agent can get you the
7:22information very very quickly which will
7:24speed up the response time but it can
7:27also do things like watching your
7:28deployment. So when you deploy your
7:29software you can have an AI agent that
7:31then watches that roll out and it can
7:33monitor for several minutes. So we can
7:35say watch the next deployment for 5
7:37minutes look for like errors and latency
7:40increases investigate any regressions
7:42and then if there's a problem the agent
7:44then can ask you if it can roll back the
7:46change basically. So you can use AI
7:48agents to monitor production but you can
7:50also use AI agents to monitor your
7:51rollouts and deployments as well which
7:53is really really powerful. And then if
7:55we want to roll back a change we can
7:56basically just ask the agent to take
7:58action and do that. So you can make
8:00these systems as powerful as you want.
8:02You can give the agent the capabilities
8:03to do almost anything but you just have
8:05to make that judgment call about how
8:07much control and autonomy you want to
8:09actually give these agent systems. So
8:11one interesting thing in this article
8:13from the anthropic team is this triage
8:15loop. So the idea here is we have this
8:17on call agent. So the clawed on call
8:19agent and then we can send in
8:21information to this agent in multiple
8:23different ways. So we can have different
8:24alert channels. So we can have things
8:26like production alerts. We can also have
8:28humans that report issues. So maybe in a
8:30slack channel you want to ask an agent
8:32to go and investigate something like
8:34maybe the application is responding more
8:36slowly or a customer is getting
8:37frustrated and they report an issue. So
8:39you can use AI agents to then triage
8:41those issues and the agent can make one
8:44of two decisions. either it can page an
8:45on call or escalate to a human to kind
8:48of come up and fix it or it can save
8:50that information, write that to a
8:51lessons.mmd. I really like this idea. So
8:54the idea here is that over time the
8:56agent can recognize certain patterns
8:59like if you have the same outage or the
9:00same root cause multiple times, which is
9:02really common. Typically when you get
9:04outages as a software company, it's
9:06normally the same things happening over
9:08and over again. So by having the agent
9:10write to a lessons.mmd file, the agent
9:12can then have some kind of persistent
9:14memory, which is going to improve the
9:16ability for the agent to actually
9:17diagnose things in production. So all of
9:20the weird things that your company has,
9:22all of the tribal knowledge about past
9:24incidents, you can store those in a
9:26lessons.mmd file and then the agent gets
9:28better over time. Okay, so this is the
9:31starting state for our demo. So we're
9:32going to start with no alerts firing.
9:34Everything is healthy in production
9:36right now. And you can see this is our
9:38on call agent UI. So I built a custom
9:41web UI to kind of demonstrate these
9:43ideas. [music] So we can chat to the
9:45agent and you can see here some previous
9:47issues that were resolved. So right now
9:49everything is good in production. So
9:51what we're going to do is deploy a bad
9:53change [music] and then watch the agent
9:54monitor that change and then diagnose
9:57what was going wrong. So we're going to
9:58deploy a bad change into production. So,
10:01we're going to push out a bad version of
10:02our application with errors in it and
10:05then see if the AI agent can autodetect
10:07and then auto recover [music] based on
10:09what it finds. So, there's currently an
10:11active incident that's happening in
10:13production. So, the AI agent has already
10:15started to triage this and it's already
10:17working on it. And if we go into the
10:19metrics, you can see a pretty obvious
10:21[music] spike in errors here from
10:23alerts. And then if you scroll down into
10:25the logs, you can see there's some kind
10:27of payment configuration error here that
10:29is causing problems. And so we've got a
10:31bunch of traffic and a large number of
10:34500 errors in production. So something
10:36has broken and the agent has now started
10:38to investigate. So if you click onto
10:40this active incident, you can see the
10:42encore agent has already triaged it and
10:44it's suggested the next step as rolling
10:46back the change. It's already gone
10:48through the logs. It's pulled out all of
10:50the relevant information. The agent is
10:52able to pull out data that can help you
10:54with your diagnosis. And this is really
10:56really useful because finding this
10:58information as a human would take you a
11:00really long time. Going through all of
11:02the logs, finding the specific error
11:04messages very very timeconuming. You can
11:06imagine this was happening at 3:00 a.m.
11:08when you're tired. So the agent is
11:10confident roll back is the right fix.
11:12Errors began when traffic moved from
11:14this version. Nothing else has changed
11:16since. So it's pretty clear that we need
11:18to roll back the change. So we can just
11:19ask the agent to do that. Roll back the
11:22change.
11:23And now it should offer us an approval.
11:26So this is where you need some kind of
11:27human in the loop. So if you're really
11:30confident, in some cases you might want
11:31the agent to act autonomously. But when
11:33it comes to doing things like software
11:35roll backs or taking action in
11:37production, it's really really useful to
11:39have some kind of approval mechanism. So
11:41you can see here we're trying to roll
11:42back from this version to the healthy
11:44version that was running previously. And
11:47we can just go ahead and approve that.
11:48And just now looking at the graphs, you
11:50can see pretty pretty clearly that the
11:51error rate has now dropped. So it looks
11:53like the roll back is complete. So the
11:55alarm is still open. It will take a
11:56while for it to recover. But you can see
11:58here from this pretty dramatic drop in
12:00error rates that we seem to have gone
12:01back to a healthy state, which looks
12:03good. So for this next demo, we're going
12:05to take a look at an example where we
12:06have maybe customers are reporting
12:08higher latency. So we've got a version
12:09of our application that is slower. So
12:12the latency or the response times on our
12:14application have gone up. So this means
12:15things are basically slower for
12:17customers, but nothing is actually
12:18broken. The application is still
12:20working. it's just very very slow to
12:22respond. So what we can do is we can
12:23start a chat with a agent and then we
12:25can sort of investigate this issue. So
12:27we can ask a question like how many
12:29requests did we serve in the last 24
12:30hours, break them down by response codes
12:33and give me the timestamps and any
12:36limitations. So we're just going to do a
12:38bit of a triage to see if we can figure
12:39out why the application is currently
12:41running slow. We can also do something
12:43like checking out the timeline of
12:45today's deployments. So we can look at
12:47whether an incident is correlated with a
12:50recent deployment. So you can see here
12:52the on call agents figured out the
12:53checkout has been slow since the last
12:55deployment. And you can see here every
12:58request is about 3,000 milliseconds.
13:01So what we can do is do a roll back. So
13:03same as before, but you can imagine you
13:05might want to take some different action
13:06if this was a real production
13:08application like maybe scaling your API
13:10or something like that. Okay. Okay, so
13:12if you're interested, this is the
13:13codebase for the agent and there are
13:15multiple different files here. So the
13:17main one here is the runner.py file.
13:19This is the actual AI agent. So this is
13:21using the clawed agents SDK and the
13:23clawed agents SDK allows you to build a
13:26clawed codel like agent. It's basically
13:28using the clawed code foundations, but
13:31it allows you to add your own tools and
13:32customizations and write your agents as
13:35code. And you can see here we have a
13:37bunch of different tools that are
13:38available to the agent. So the agent can
13:40read the logs, it can read metrics, it
13:42can do things like checking out
13:43deployment events as well. So really you
13:46want to think carefully about the tools
13:47that you give your agent. You could
13:49build a tool for example to roll back a
13:51change. You could build in any kind of
13:53tool to help the agent mitigate or
13:55resolve an incident. So really the
13:57quality of your agents often comes down
13:59to the quality of the tools you actually
14:01give the agents. The good thing about
14:03writing this in code is you can be very
14:05careful. You can ensure that the agents
14:07only have specific permissions. So you
14:08can do a lot of checking to make sure
14:11that the operations the agents are doing
14:12are safe. So this is the highle
14:15architecture of what we're going to be
14:16building. So this is actually pretty
14:17straightforward. The diagram is probably
14:19a little small to read, but the general
14:20idea we get a notification or an alert
14:23happens inside our system. So something
14:24breaks, we trigger an alert. Most cloud
14:28platforms give you the ability to
14:29generate alerts when something breaks.
14:31We then send that alert into a queue and
14:33then we have an AI agent that will
14:34triage start investigating the issue.
14:37The agent will then come to a conclusion
14:38and then it will write its findings back
14:40into a database. And we also have a web
14:42UI as well. So we can actually go and
14:44see what's going on. We can talk to the
14:46agent. We can interact with it back and
14:48forth. So this is going to be really
14:49useful when it comes to diagnosing
14:51issues. Another really interesting idea
14:54that I wanted for this particular S sur
14:55agent was the ability for the agent to
14:57watch or monitor a rollout. So in a
14:59previous video I was talking about the
15:01software factory ideas from OpenAI. They
15:04use a similar similar system to this
15:05where they basically have an agent
15:07monitor a roll out. So as you deploy
15:09your software, you can ask the agent to
15:10basically monitor it. So you can just
15:12jump into the console and say something
15:13like, "Watch the next deployment for 5
15:15minutes." The agent is then going to
15:17monitor that and pay attention. If
15:19something breaks, you can then talk to
15:21the agent or the agent can take action
15:22to fix it. So I think this is a really
15:25useful thing to have. If you build
15:27software, certainly if you're building
15:28software at scale, having an agent that
15:30can monitor your rollouts is kind of a
15:32nice thing to have.
15:33So I think these systems are really
15:34important and as we start using more and
15:36more AI generated code as we start
15:38moving faster we're deploying more
15:40changes more frequently having AI
15:42systems like this is really really
15:43important if you're a business that
15:45[music] sells software or if you operate
15:47software with real customers having
15:49something like this is going to help you
15:51[music] out massively. So humans are
15:53good at making decisions but finding
15:55[music] data and finding information at
15:574:00 a.m. when you're tired is actually
15:59quite stressful and incredibly
16:01difficult. So if you can give an AI
16:03agent to your engineers to help them
16:04diagnose things more quickly to [music]
16:06find information and suggest next next
16:09actions, it's going to really massively
16:11reduce the time to resolve a particular
16:14incident. This is a great example of an
16:16agent that is providing massive value,
16:18[music] but it's also showcasing some of
16:20the limitations when it comes to AI
16:22agents in general. They can often jump
16:24to conclusions [music] and so you have
16:25to be very careful about relying too
16:27much on whatever they tell you. [music]
16:29So often they'll find the information
16:31you need, but maybe they'll misinterpret
16:33the data or they'll come to a false
16:35conclusion. So you still need to be
16:36really careful. And it kind of just
16:38shows that you still need human in the
16:40loop when it comes to really [music]
16:41critical systems like this. But overall,
16:43I think this is such a great use case
16:45and it's one of my favorite examples of
16:47autonomous AI systems. If you have any
16:49questions about the video, let me know
16:51in the comments below. Thanks again for
16:52watching and I'll see [music] you in the
16:53next one. Take out.

About this transcript

This page contains the full transcript of I Built A Claude DevOps Agent To Run My Software Factory by Owain Lewis, generated from the public captions YouTube serves with the video. The transcript has 3,772 words across 533 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.