YT YouTube2Text

YouTube2Text → Transcripts → Let Your AI Agent Do ITS OWN Harness Engineering (Here's How)

Let Your AI Agent Do ITS OWN Harness Engineering (Here's How) — Transcript

by Cole Medin · 3,571 words · 507 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00So, on my YouTube channel this year,
0:01I've mostly been focusing on AI agents
0:04for coding. But, what a lot of people
0:06don't realize, maybe even including
0:07yourself, is that a lot of these
0:09strategies that go into making our
0:11coding agents more reliable over time,
0:13also translate over very well to other
0:16kinds of AI agents that we deploy to
0:18production for automation and business
0:20use cases. And one of those big ideas is
0:23harness engineering, making everything
0:26around the model, like the rules and the
0:28workflows, better over time. Now, of
0:30course, you wire agent to get better
0:32over time. But, what I'm referring to
0:33specifically here with harness
0:35engineering is tracking all the mistakes
0:37that your agent is making in production,
0:40and then going back retroactively,
0:41having that conversation with your agent
0:43to figure out what should we make better
0:45in your rules and skills so these
0:47mistakes don't happen again. I teach
0:49that all the time with harness
0:50engineering for coding agents, and
0:52really it's the same thing with your
0:54production agents. And the thing is this
0:55translates even better than you would
0:57think because these days production
0:59agents for the most part are built just
1:01like coding agents, where we manage the
1:03entire harness with a lot of markdown.
1:05And this keeps things very flexible. The
1:08quote-unquote old way of doing things
1:09with agents was a lot more of the
1:11plumbing directly in the code, like
1:12creating our Python and TypeScript
1:14functions for our tools, for example.
1:16But, now we have our skills as markdown,
1:18our global rules, like our instructions
1:20on MD, our memories. This should look
1:22very familiar. It's a very similar sort
1:24of harness setup to a coding agent. And
1:26this translation works so well that all
1:28the components of agentic engineering in
1:30our harness, they really have a
1:31one-to-one mapping with our agents in
1:33production. Like we have our rules,
1:36which for a traditional agent is the
1:37exact same thing as a system prompt. We
1:40have our sub-agents, that's the same
1:41thing as multi-agent workflows where
1:43we're doing any kind of delegation. We
1:45have MCP tools for our connections.
1:48Skills are our workflows, replacing a
1:50lot of our traditional tools. And then
1:52we have hooks for our guardrails. If you
1:54want to monitor or block things for each
1:57tool call or the start or end of
1:58conversations, we have our guardrails
2:00for that. And so today, I'm not only
2:02going to show you an example of building
2:04out this harness, but also improving it
2:06over time with our two core loops. We
2:08have the main memory of the agent, and
2:10then also improving it retroactively by
2:12looking at the traces as we are
2:14monitoring the agent in production
2:16talking to real users. That is the
2:18biggest goldmine of information that we
2:20can't ignore. Most people do. And so
2:22we'll get into all of this, but also I
2:23just want to show you at a high level
2:25how do we build an agent that takes on
2:27the coding agent harness. Because the
2:29thing is, as we create our markdown
2:31documents for our skills and our rules
2:33and everything, we need some kind of
2:34system that actually bundles it and
2:36connects it together when we deploy the
2:38agent to production. Now, for this
2:40platform, you can build it yourself if
2:42you want to manage your own deployment
2:43and harness engineering pipelines.
2:45That's totally fine. But a lot of the
2:47time it's more practical to use a
2:48platform that manages this for you. And
2:51so that's why for all these demos, I'm
2:52going to be using LangChain's managed
2:54deep agents, which sets up all this for
2:56us in the LangSmith. I've seen this work
2:59time and time again in production. And
3:01as we go through everything, you'll see
3:02how this is very universal. All the
3:04strategies apply no matter how you're
3:06building these agents. And for a quick
3:08shoutout, thanks to LangChain for
3:09working with me on this video. It is
3:11about time that I've collaborated with
3:12them because I'm using their platforms
3:14like managed deep agents all of the
3:16time. And for a super quick throwback
3:18here, this is one of the very first
3:19videos I put on my YouTube channel over
3:22two years ago when I first started my
3:24content. I was building AI agents coding
3:27them by hand with LangChain as my
3:29framework. And so it's always had a
3:31special place in my heart. I continue to
3:32use them because LangChain has always
3:34stayed on the cutting edge, especially
3:36now with managed deep agents helping you
3:38deploy your agents in the sort of coding
3:40agent fashion. And then the other thing
3:42that they have is the tracing and
3:44metrics to help us really get into the
3:46harness engineering figuring out where
3:47our agent is going wrong, and even
3:50giving us suggestions that we can
3:51automatically apply. So we'll talk about
3:53this once we get into the different
3:54improvement loops for our harness as
3:56well.
3:57Okay, cool. So I want to quickly show
3:58you what it looks like to build these
4:00kinds of agents now with LangChain's
4:02managed deep agents, and then we'll get
4:03into the learning loops, and I'll even
4:05give you a live demo. So, kind of like
4:07what I was saying earlier, back when we
4:09used to build our agents a year or two
4:10ago, it was a lot of that plumbing in
4:13the code. I mean, we had to write so
4:16much code to build out our guardrails
4:17and our tools and hook in our system
4:19prompt and everything. But now take a
4:21look at this agent.py file. It is 30
4:24lines of code. That is all it takes to
4:26create the definition for our agent, and
4:28this includes guardrails, sub-agents,
4:31all the tools that we have, the skills
4:32and the instructions, because all this
4:34is simply defined as markdown documents
4:37now. So, with some of the agent
4:38definition here, we are writing into
4:40code. There's a little bit of that
4:41plumbing still, but for the most part it
4:43works like coding agents, where things
4:45are picked up automatically. So, we have
4:47our instructions.md. As long as it's at
4:49the root of the agent folder, kind of
4:51like our claw.md or agent.md with the
4:54coding agent, then it's going to be
4:55picked up automatically. We have our
4:56skills in the skills folder, just like
4:58we have our, you know, dot claw or dot
5:00codex folder, whatever it is. And so,
5:03all these things when we package it up
5:05together to run it in development
5:06locally or deploy to production,
5:08LangChain handles stitching everything
5:10together behind the scenes. So, it's
5:12very easy for us to build and manage our
5:13agents, and it's all very flexible. So,
5:15if we're not happy with how the agent is
5:17performing, then we simply just change
5:19these markdown documents exactly like we
5:21do for the system evolution, as I call
5:23it, for our coding agents. There's also
5:25a lot more behind the scenes with the
5:27evals and learnings, making our harness
5:28better over time. So, let's get into
5:30that now. And I'll even give you a live
5:32demo here with the first loop, which is
5:34the memory for the agent. So, no matter
5:37the agent you have deployed to
5:38production, you're going to want a
5:40system like this. LangChain does it
5:41really well. There's a lot of
5:42inspiration we can take from this setup,
5:44and I'll show you with the live demo as
5:46well. We essentially just need a way for
5:48our agent to learn the lessons as it's
5:51running in production. And that does
5:52mean escalating some things to a human.
5:55So, whenever someone is talking to our
5:57agent and our agent is struggling, we
5:59want it to be able to recognize that and
6:01escalate the uncertainty. And so, we
6:03have some kind of inbox where we're
6:04managing the different things the agent
6:06is uncertain with. We give it the right
6:08answer and we make sure the agent writes
6:10that down as a lesson. So we have some
6:12kind of context hub as LangChain calls
6:15it where we're storing these memories.
6:16It can always retrieve that later so it
6:18can recognize like, "Oh yeah, this
6:19situation I handled before. This is how
6:22the human team corrected me. So I'm
6:24going to give the right answer this
6:25time." And this is so important because
6:28especially when we're first deploying
6:29our agent out to real users, we don't
6:31really know the gaps in its knowledge
6:33and process. And so LLMs, yes, they
6:36hallucinate, but they do a surprisingly
6:38good job at surfacing uncertainties. So
6:40we can correct these things and have
6:42these lessons build up over time. And of
6:44course, we have an efficient retrieval
6:46strategy here. So it's not like we're
6:47just dumping in all these lessons into
6:49context like our main instructions. Now
6:51of course, we can use these lessons to
6:53improve our instructions. We'll get into
6:55that with the second loop. But right
6:57here, it's all about building up a
6:59knowledge base of how we're handling the
7:01key situations with customers and users
7:03because generally there's a lot of
7:05commonalities between how people are
7:07using our agent. And so this knowledge
7:09base that we start to build up in what
7:10LangChain calls the context hub helps
7:13significantly. And as the agent gets
7:15better and better, there's also just
7:16less that is escalated over time. So
7:18it's not like this continues to become a
7:20huge burden for the humans managing the
7:22agents. So here we have the production
7:25bundle for my agent here in managed deep
7:27agents. And so within the memory folder,
7:29there's a couple of things that we have
7:30here. First of all, we have all of the
7:32cases, things where the agent was
7:34uncertain and it escalated to me in the
7:36team inbox that I'll show you in a
7:38little bit. And so I make that
7:39correction and then it's going to
7:40remember those things both per user and
7:43it's going to build up this sort of
7:44internal agents.md, the core rules that
7:47are all learned from the lessons that
7:49I've taught it with the escalations. All
7:51right, so let me show you this in action
7:53now. So this is a custom dashboard I
7:55built with a demo version of a real
7:58agent that I'm working on, the Dynamis
8:00member help desk. And so this here is
8:02proof. I've been running this and
8:03testing it before I made this video that
8:05we We able to improve the agent pretty
8:07autonomously over time. I'm just going
8:09to give you one example of this here.
8:11So, I kept the data set simple. I kept
8:13everything simple here just for the
8:14demo. And so, I have a help desk here
8:17where I've mocked different members of
8:18the community asking different questions
8:20about Arcan, my course content, whatever
8:22it is. And so, this is Nina. I'm going
8:24to act like I am a Nina here with the
8:26question to my support agent. So, "Hi
8:28Amina, I committed a fix my plan.md
8:31command locally and I ran the Arcan
8:33workflow, but the run used my old
8:34command text. I know that seems oddly
8:36specific, but this is a real question
8:39that I received about Arcan a couple of
8:41weeks ago." And so, all the things here
8:42that I've tested with is actually based
8:44on real questions that I've gotten. And
8:45so, the help desk agent is going to
8:47answer in a second here, but as you'll
8:49see, it isn't going to be confident in
8:51its answer. So, as you saw, it took a
8:53little bit to start thinking just cuz I
8:55had to compile the agent locally, but it
8:57only took about 10 seconds for it to do
8:59everything after, including searching
9:01the memories for this specific person,
9:04loading the skill for the diagnose
9:05setup, and then also updating the
9:07memories for this person. So, we have
9:09individual user memories that I'll show
9:11you more in a little bit as well.
9:13But, here's the most important thing.
9:15The help desk said, "I don't have enough
9:17yet to say for sure." So, it's still
9:19going to try to help. We don't want the
9:21agent to just block and wait for a human
9:23entirely, but it surfaced the
9:25uncertainty. And then take a look at my
9:27team inbox now. It was four right before
9:30I had this conversation. Now it's moved
9:31up to five because there's something
9:33that was just escalated to me. So,
9:35here's the question from the person. And
9:37then here's the answer from the help
9:39desk. And then I have to confirm the
9:41fix. And so, I can either correct the
9:43agent here or say that it did it right.
9:45Either way, it's going to update its
9:47memories. It's going to learn a lesson
9:49so it's confident the next time. So, for
9:51this case, I have the solution already.
9:52Arcan creates the work trees from origin
9:54main. So, the solution is to run get
9:56push and rerun. So, I'm going to send
9:57this into the agent, host the confirmed
10:00fix. And so, now it's going to learn a
10:02lesson from that. So, I'll come back
10:04once we have it. And boom, just 10
10:05seconds later, we have the confirmation
10:07that we have a new shared memory. We
10:09have learned a lesson. So, it updated
10:11that core agents.md file in the memory
10:14folder like I showed you earlier, and
10:15then also the private memory for Nina.
10:18This is the important one though,
10:19because now going forward, if someone
10:20else has a similar issue, it's going to
10:23be a lot more confident in its answers.
10:25So, let me show you that, too. So, back
10:27over in my chats here, I'm going to act
10:28as a different person now. So, I'm Omar.
10:30My Arkon run finished green and open a
10:32PR, but the diff is missing three
10:34commits I made this morning. Is this a
10:36bug or what? And for this, it's not
10:38quite the same as Nina's problem, but it
10:40is similar enough where it's going to be
10:41able to learn from that lesson. So, I
10:43also want to show you something that's
10:45not so cheesy where it's like literally
10:46the same exact problem, just to show
10:48that the agent is able to extrapolate
10:50based on memories. And so, there we go.
10:52This time it didn't say it was
10:54uncertain. It said likely not a bug.
10:55Here's exactly how Arkon works, right?
10:57This is the lesson that we just taught
10:59it, and that let it to giving a very
11:01solid answer now. If I go into the
11:03memory tab that's hooked directly into
11:05LangChain, I can take a look at that
11:06cases folder right here, and sure
11:09enough, we've got our new memory. Arkon
11:11builds the run work tree, so the members
11:12fix was get pushed and then rerun Arkon
11:14workflow run. There we go. This is the
11:16exact same thing that we're looking at
11:17here within the cases, right within
11:19managed deep agents. And then also, if
11:21we go into the context hub, we can see
11:23the per user memories as well. Like
11:25these are the memories for just Nina.
11:27And so, we also have that separation
11:29that's guaranteed with the guardrails in
11:31the agent, so we have per user memories
11:33that never have any overlap. And it's
11:34not like users can view other users'
11:36memories. Cool. So, the last thing I
11:38want to show you now is the second loop.
11:40We've talked a lot about the memory. And
11:43this is an important part to our context
11:45hub, but it really isn't the main part
11:47of our harness. The per case lessons,
11:50it's a good resource for the agent to
11:52search through when it needs, but we
11:54have the higher level harness, which is
11:56our core workflows and instructions. And
11:59we want to update these over time as we
12:01are learning from our lessons and the
12:03other traces that we have with our
12:05production data. And so, that's what I
12:07want to show you now. This is the other
12:08part of managed deep agents. This is the
12:10engine. Using LangSmith under the hood
12:13for all the trace data, but then using
12:15the engine to collect that and start to
12:17make suggestions for how we can improve
12:20our agent based on the failures that
12:22we're seeing with real data. So, this is
12:24a real example of the failure that came
12:26from my production runs. And so, we have
12:29the description at the top here getting
12:30quite detailed. If I'm interested, we
12:32have the evidence with the LangSmith
12:34trace. I can go into the exact
12:36conversation and see what the model was
12:38doing when this issue happened. And then
12:40my favorite part is at the bottom we
12:42have the proposed fix. All of this
12:44running behind the scenes with LangSmith
12:46engine. And so, it proposes, "Here's
12:48what we should change to the
12:49instructions.md."
12:51This is just like system evolution with
12:53our coding agents where it's surveying
12:55the entire harness, our rules and
12:57guardrails and instructions and
12:59everything. It's figuring out, "What
13:00should we adjust so this kind of issue
13:02doesn't happen again?" So, here is our
13:04proposed fix. It's nice and concise cuz
13:07I also don't want to just like bloat my
13:08rules by shoving a bunch of new context
13:10here. And then I can click apply and
13:13it'll immediately apply this to my
13:15production agent. So, I'm approving
13:17things and having it fix itself in real
13:19time, but only based on the fixes that I
13:22am overseeing, right? I don't really
13:23want my agent to autonomously improve
13:25itself because then I lose control of my
13:28agent and I don't really know the
13:29direction it's heading. That's really
13:31the big theme with everything here. We
13:33always want the agent to be surfacing
13:35and applying fixes automatically, but we
13:37still need to be in a loop approving
13:39things. So, if we go to the context sub,
13:41all of the memories that we've been
13:43creating for our agent for individuals,
13:44the main lessons that we're learning,
13:46we're always a part of every single one
13:48of these. And going into our skills and
13:50instructions, everything that we change
13:51here, again, we approve before anything
13:54is changed. Just like when we're working
13:56with our coding agents, generally we'll
13:57create pull requests to update our rules
14:00or our skills. We're doing the same kind
14:02of thing here, but just directly in
14:04LangSmith with managed deep agents as
14:06our platform to do so. And so, no matter
14:08how you end up doing this sort of
14:09harness engineering for your production
14:11agents, you need a platform where you
14:13have the trace data and you have the
14:16dashboard for you to approve things as
14:18your agent is working and surfacing
14:19thing. And continuing with that pull
14:21requests for a coding agent analogy, we
14:23literally have the commits right here in
14:25managed deep agents. And so, I can go
14:27through each one of these and see what
14:29exactly the different parts of my
14:30harness look like here. I can promote
14:32different versions, roll things back if
14:34I need. So, it's also not the end of the
14:35world if we make our agent worse by
14:38applying the fixes that it's proposing.
14:40We have full control of our system here.
14:42It's also incredibly easy to deploy this
14:44anywhere. Like I've even deployed it to
14:46my Slack environment, so I can talk to
14:48the managed deep agent right here.
14:50Now, of course, I have worked with much
14:52larger data sets before, but for just
14:54for this demonstration, I had a set of
14:5614 questions where the initial agent
14:58that I created with the first version of
15:00rules and skills and everything, it only
15:01got four out of the 14 questions right.
15:04But, after I went through the learning
15:05process of applying lessons, using the
15:07LangSmith traces with the engine to
15:09apply its suggested improvements, I got
15:11to 13 out of 14 right. So, LLMs aren't
15:15always going to be perfect. It's not
15:16going to be 100% of the time. You can
15:18get really close though, and we're using
15:20both of these loops to get there. So,
15:22that is how you let your agent do its
15:24own harness engineering with you still
15:26in the loop approving everything. And
15:28so, I hope this video showed you the
15:29power of LangChain's managed deep
15:31agents, but just also generally the
15:33high-level ideas here for how we want to
15:35engineer our agents that we're deploying
15:37to production. Just taking all these
15:39ideas from coding agents, so what makes
15:41them work so well. And the biggest
15:43lesson here is harness engineering is
15:44not a one-time job, and that's what
15:46managed deep agents helps with. It's
15:48giving your agent a bespoke harness
15:50where you can use any model, and then
15:51layering on top the LangSmith engine, so
15:53you can use LangSmith to keep improving
15:55your harness. All the while, you
15:57maintaining full control. And so, I'll
16:00have a link in the description to try
16:01out LangSmith. You get $50 in credits if
16:03you use the code Cole M. And thanks to
16:06LangChain for working with me on this.
16:07And so, if you appreciate this video,
16:09you're looking forward to more things on
16:11agentic engineering and harness
16:12engineering, I would really appreciate a
16:14like and a subscribe. And with that, I
16:16will see you in the next video.

About this transcript

This page contains the full transcript of Let Your AI Agent Do ITS OWN Harness Engineering (Here's How) by Cole Medin, generated from the public captions YouTube serves with the video. The transcript has 3,571 words across 507 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.