YT YouTube2Text

YouTube2Text → Transcripts → Всё, что нужно знать об AI Agent Harness за 19 минут | Loop, Eval, Tracing, RAG

Всё, что нужно знать об AI Agent Harness за 19 минут | Loop, Eval, Tracing, RAG — Transcript

by Рома Филонов | AI, больно · 4,936 words · 751 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00In this video, I will quickly cover all
0:02the essential terms you need to know
0:04about AI, agents, and harnesses.
0:06Over the last year or so, a huge number
0:08of them have piled up: loop engineering
0:10, harnesses, LLM calling, LM-OBS, and
0:12all these terms—they are quite
0:13superficial, but there are so many of
0:15them.
0:16In these 20 minutes, I will break down
0:17each one with examples and show you that
0:19there's nothing complicated about it. We
0:21will go from the very basics of how
0:23context and LLM queries are structured,
0:25to how they can be wrapped in a loop—
0:27essentially creating your own agent
0:29system. This is exactly how CodeX and
0:32Claude work.
0:33They just keep running in a loop,
0:34deciding for themselves what additional
0:36actions need to be taken to solve your
0:38task.
0:38Beyond that, we will look at how
0:40different types of memory are structured
0:41.
0:42Automatic memory, where we write
0:43everything to a Markdown file, and
0:45smarter neural memory, where we can
0:47place everything into a vector database
0:48.
0:49For example, like how the additional
0:51utility library Honcho does it for CodeX
0:53or Claude. We will see how to interact
0:55with it. And at the end, we will get
0:57acquainted with some concepts for
0:58production, like how to track the
1:00performance of agents in a live
1:01production environment. Well then, let's
1:03get started. At the heart of any agent
1:05system or harness—it doesn't matter what
1:07that is yet—lies a basic atomic
1:09operation. That is a request to an LLM.
1:11The LLM can be local, or a request via
1:13an API. Now we will look at how to do
1:15this using Python. We usually open, for
1:17example, a chat with GPT and write our
1:19text there. But that is not all the text
1:21that the LLM sees.
1:23In addition to that, there is also a
1:25system prompt, written either by us or
1:26by the developers, which contains some
1:28extra instructions for the model's
1:30behavior. When we are chatting with it,
1:32the entire past history of that chat is
1:34essentially always fed into it as well.
1:36All of this combined is called the
1:38model's context, and it is limited. For
1:40models, it's roughly around a million
1:41tokens. That is the limit of what it can
1:44hold in its conversation. To go beyond
1:46this limit, you will have to perform
1:47some additional manipulations. After the
1:49response is received, we can apply some
1:51further validations to it. This depends
1:53from project to project. This is called
1:55guardrails. For example, we can
1:56automatically check that our model's
1:58output doesn't contain any 18+ content
2:00or prohibited material. For instance,
2:02GPT has such filters when it generates
2:05images. After it has generated one, a
2:07check automatically runs to ensure there
2:09is no pornography or anything prohibited
2:12. In that case, the generation simply
2:14stops and is not shown to you. Let's
2:15look at how to build the simplest GPT
2:18chat in Python. We will need the
2:20following. We'll need the OpenAI library
2:23to make requests, and an internal API
2:25key. I already have one. I will read it
2:28from a file.
2:30Next, we'll just run a simple while true
2:32loop, where our code waits for us to
2:35type a message.
2:36Right here in Python, it will receive it
2:38.
2:39If we type the word "exit", our while
2:41true loop will automatically break and
2:42the dialogue ends.
2:45If we write something else, we add that
2:47text to our conversation history.
2:49And this conversation history is sent to
2:52our LLM model via API. For example, I'll
2:54be using the GPT 3.5 model here.
2:56Actually, we can write any other one
2:57here. We can go ahead and change the
3:00text here to GPT 4.
3:02That's it, let's just run this script.
3:04It's running now, and it's waiting for
3:06me to type a message. Let's try asking
3:08it how much free space and memory I have
3:11on my computer.
3:13Right, let's wait. The request has been
3:15sent. And now the text is being written
3:17. The model is writing it to us directly
3:19. It's telling us that it doesn't know
3:21how much memory I have on my computer.
3:24So we'll need to perform some extra
3:26manipulations so it can find that out.
3:28Of course, when we just send it text,
3:31the model can only accept and output
3:33text. It has no information on how to
3:35interact with my computer. Therefore,
3:37it's a separate task to provide it with
3:39that information. A model that can use
3:42tools is called an agent. Now let's see
3:45how to build one. The only change we
3:47need to make to our code is to define a
3:49special tool right here. Its format is
3:52always roughly the same. Well, you don't
3:54need to dwell on what's written here.
3:56I'll just pass text to the model
3:58explaining how it can check how much
4:00memory I have.
4:01Like, how much free space is on my
4:03computer. Here we have the same while
4:05true loop, where I just pass info to the
4:08model that it can also check my free
4:11memory.
4:12And if it gets a question and realizes
4:14that, aha, it needs to check the memory
4:17, it can just call a Python function to
4:19check the volume itself.
4:21Let's run the code and see what it does
4:23. There, it's running. Let's ask how
4:26much memory I have on my computer. Send
4:30. There. And here in the logs, we can
4:33see it writing that it used a tool to
4:36check that I have 24 GB of RAM total,
4:39and how much is free.
4:41And then it wrote its final answer. So,
4:43first it just used a Python script to
4:46find the specific numbers. These numbers
4:48were fed to it as text, providing that
4:50info, and it wrote the final response.
4:52That's it, the end. This is an example
4:54of how a harness or any agentic system
4:56is built. This is what AI engineers do
4:59in their jobs and projects. This is
5:02exactly how Codebase, Claude, and anyone
5:04else is built. It's just a ` while True`
5:07loop where our model gets a lot of text
5:10on how it can interact with other tools
5:13. Accessing a calendar, using an API for
5:15another service, or interacting with a
5:17computer, for example, to edit our files
5:19—that's what Codebase and Claude do.
5:21They just have many functions for
5:23opening a file, changing a specific line
5:25you need, or reading from it. All of
5:27this, in essence, is a harness. You can
5:29find more of these interview questions
5:31for an AI engineer on my interview
5:32preparation website.
5:34It contains real interviews with real AI
5:36questions from Russian and international
5:38companies.
5:40So, to sum it up, a harness is just a
5:42huge, huge code wrapper around model
5:44requests.
5:46We looked at what a single request to a
5:47model looks like. It's just text. We
5:49send text, we get text. And the whole
5:52accumulation of code around it is called
5:54a harness. And of course, the most
5:55well-known examples that everyone knows
5:57are Codebase and Claude. Now there's
5:58Devin, harness, Open Dev, there are a
6:00million of them. It's just a codebase
6:02that wraps around any LLM and allows it
6:04, in this case, for instance, to be used
6:07for coding tasks, so that even regular
6:09people could interact with Excel, the
6:11browser, or anything else. But these are
6:14examples of super-powerful, very
6:15powerful harness systems that can do
6:17almost anything. They are
6:18unrealistically huge and complex. They
6:20are essentially a Swiss Army knife. They
6:22have absolutely everything. But there
6:23are plenty of examples on the market
6:25where we need to create more local,
6:27targeted solutions. For example, we
6:29could build a separate harness system
6:30that interacts with a browser. There.
6:32And an example of such a system is the
6:35Perplexity browser agent. So there are
6:37also developers sitting there, working
6:39on creating these kinds of code wrappers
6:41so that the LLM can interact more
6:42effectively with browsers and websites.
6:45We, for example, could be some kind of
6:47banking application that wants the user
6:50to be able to log in and not have to
6:52hunt for buttons on how to transfer
6:54money to someone, but instead just type
6:56: "Please transfer 5,000 rubles to Vasya
6:58."
6:59And that's it, the model itself will do
7:01everything. Well, not the model, but
7:02rather our harness system.
7:05I mean, if you're a regular developer
7:07using tools like Cursor for basic coding
7:09or task execution, you're already using
7:12a highly advanced harness system on a
7:14daily basis.
7:16There's really no need to develop your
7:18own harness system. That phrase, in
7:20principle, sounds very strange. These
7:22are more likely commercial projects that
7:23AI engineers work on.
7:25There might be a separate company that
7:27wants to implement more constraints and
7:29wrappers around the use of an LLM.
7:31For example, I already brought up the
7:32bank scenario. We want to fully automate
7:34bank operations and the interaction
7:35between the LLM and its various systems
7:37. And it is important to understand, by
7:39looking at a chart like this, that the
7:40quality of the final solution depends on
7:42more than just the model. Whether we use
7:44Claude Opus or GPT-6 Astra, it really
7:46doesn't matter. What also matters is the
7:48specific code wrapper we are using.
7:50Harness, for example. Here they take the
7:53same model and measure it against
7:55special benchmarks. Most often, these
7:57are benchmarks for coding tasks. They
7:59are very easy to measure. We provide a
8:00task, we know what the correct answers
8:02should be, and we can very easily
8:03measure the quality of the agent's
8:05performance. And here, please, is an
8:06example.
8:07If we used, for instance, an older model
8:10like GPT-5.2 Codex and simply changed
8:12the harness system around it, we
8:14immediately gained a 13%increase in
8:17task-solving quality.
8:1913%is a lot; that is a very significant
8:21boost. We can also look at the results
8:23on another benchmark.
8:25Here we have the Qwen 37 Max model
8:27competing on various tasks, and we just
8:29switch it from one harness to another,
8:32and we see that simply swapping our
8:34harness adds to its final quality.
8:37What does this tell us? It tells us that
8:39comparing models purely in isolation is
8:41unfair. We need to compare the final
8:44wrappers that they interact with.
8:45Because we can give a task to a model
8:48and, for example, in one harness,
8:50meaning one code wrapper, it will
8:52perform an additional 100 checks.
8:54Checked the computer's status, checked
8:55something in the browser, or used
8:57something else. But if it interacts with
8:59a harness that lacks such functionality
9:01and doesn't initiate this check loop,
9:03the work quality will be lower. Let's
9:06quickly get familiar with the concept of
9:08a model's context. We understand that
9:10any model, any LLM, takes tokens as
9:12input. These are like words. More often
9:14, they are sub-word units. And models
9:16have a limit on how many tokens they can
9:19use. Meaning, the chat conversation you
9:21have with it isn't actually infinite.
9:22It's limited. The limit is roughly
9:24around a million tokens. That's it,
9:26technically it just cannot continue the
9:28same conversation with you any further.
9:30To keep it going indefinitely, you need
9:32to use extra tricks. For example,
9:34conversation summarization. You often
9:36see this in Claude or Codey when you've
9:38been chatting with it for a long time.
9:40It suddenly writes: "Wait, I am starting
9:42the summarization process of our
9:44dialogue. This will take a couple of
9:45minutes." So, what does the overall
9:47filling of this context look like? For
9:49instance, when you just open a chat with
9:51a model, you type in your text request,
9:52like "how much space do I have on my
9:54computer," but besides that, other texts
9:56you don't see get in there. This
9:58includes the system prompt, the skill
9:59descriptions; it all goes into the
10:01context.
10:02And for a small model, a very large
10:04context is problematic because small
10:06models, like local ones, will start
10:08making mistakes, confusing skills,
10:10forgetting what they have, and there
10:12will be constant issues. That is why
10:14this kind of static context usage is
10:16very dangerous. For example, if you are
10:18working on a large project where you
10:20have hundreds of skills, the model will
10:21simply go crazy. It won't be able to
10:23process that much. You will fill up all
10:25its free space, and it will start
10:26forgetting about skills or using them
10:28incorrectly. In short, there will be a
10:29lot of problems. That's why we will move
10:31on to explore advanced methods of how
10:34you can dynamically pull in specific
10:36knowledge. Well, that's about it.
10:38Furthermore, when you start working with
10:40a coding agent or Codey, it constantly
10:42runs in a loop—like reading code, then
10:44deciding on its own to trigger a tool,
10:46for example, reading a file, and so on.
10:49It keeps doing this in a loop, over and
10:50over, until it reaches the end and stops
10:52. There you go. So, what is the actual
10:54complexity here? Developing such agentic
10:56systems. It might seem quite primitive.
10:58We just run a loop, and the model
11:00decides what to execute itself. But no,
11:02there are many nuances there. Let's
11:03consider the case with context
11:05summarization. Anthropic encountered
11:07this.
11:08The problem was as follows: this
11:10summarization is set up such that, for
11:12example, 800,000 context tokens are
11:14filled, and we tell the model, having
11:16seen it in the code: "Aha, that's it,
11:18we've hit a limit, please summarize all
11:20of this correspondence."
11:22But it is quite difficult for it to do.
11:24The model might just forget something or
11:26write something incorrectly because the
11:28more context it consumes, the harder it
11:30is for it to work. The whole task
11:32becomes heavier for it. And so at
11:33Anthropic, they discovered that things
11:35went very poorly if done that way dryly
11:37. The model lost a lot of details, and
11:39everything got much worse.
11:41Therefore, they had to develop an
11:43additional type of interface to also
11:45pass information into the new chat,
11:46specifically information from Git.
11:48That is, they consume and check in Git
11:50what the latest commits and changes were
11:53.
11:53They put all of this into separate files
11:55, into JSON and text files, where what
11:57was done and what needs to be done next
11:58is written down. And all this top info
12:00is passed to the new chat. And the new
12:02chat starts with this information
12:03already. Well, we also understand that
12:05all this summarization, plus the extra
12:07info, also cuts into the context in the
12:09next chat. And in fact, there is less
12:11and less of it available. The model
12:13might start performing worse. Therefore
12:15, it is important to start chats with an
12:17empty context more often. Meaning, don't
12:18be afraid to start new chats. You
12:20shouldn't stay in one chat or one thread
12:22for too long, because the more context
12:23it has consumed, the worse it will
12:25perform. Yeah. And an example, by the
12:27way, from Aider. Aider is an open-source
12:29product. You can go in and see how it's
12:31structured. This tool invocation that
12:33happens there is written in Rust. And,
12:36for example, you can go here and take a
12:38look. This will all be a link. You'll be
12:40able to go to the Telegram channel, look
12:42, basically click here in Excalidraw,
12:44land on this script, and see how huge
12:47this script of functions is—the one that
12:49knows how to modify files, like patching
12:51, taking one of our code files and
12:53changing some content in it. There is
12:56just a ton of this tool functionality
12:58there. So, it's a huge technical
13:00challenge to make such agentic systems.
13:02Quite complex. But let's go back to our
13:05previous diagram here. We are going to
13:08finish it off now. We’ve realized that,
13:11okay, we have some model context, say, a
13:14million tokens. It gets filled up with
13:16our messages and the model's tool calls
13:18. Everything it does, it also adds to
13:20this context. It constantly decreases,
13:22and decreases, and decreases. And we do
13:23all this simply in a while true loop.
13:25This is what is called an Agent Loop.
13:27That's it, the end. Actually, our task
13:29could be a single workflow or pipeline,
13:31for example, gathering the latest news
13:34from the past day.
13:35We can just write a text prompt, write a
13:38function that will go to Google, make a
13:40few queries, collect them, and have the
13:42LLM summarize them. That's all. This is
13:44essentially a single pipeline. It runs
13:46once, but we can, theoretically, put it
13:49in a loop so we collect context, then
13:51the LLM double-checks everything,
13:53verifies the dates to ensure we actually
13:55collected data for that day, and it can
13:57go through the loop again. So, this
13:59thing will, obviously, be much more
14:01accurate, but it will consume more
14:03tokens and run slower. This is how
14:05Claude and Codex work by default. Yes,
14:06if you throw a task at them, the model
14:08will start checking itself in a loop.
14:11But you can manually write your own
14:13script using special libraries like
14:15LangGraph that will process it just once
14:17and that's it.
14:18Meaning, you can configure it very
14:20precisely. There. And everywhere in
14:22these loops, we use something called
14:25tooling.
14:26Tooling is when we pass information into
14:28the model's context about the fact that
14:29we can call an external script.
14:31The model itself doesn't know how to
14:33interact with a computer. It will
14:34interact with the computer through tools
14:36. Tools are just functions. For instance
14:39, in Codex, we could see that this
14:41function is written in Rust. That's it,
14:43it will simply call it. Okay, that's
14:45clear. Here was an example where a user
14:47asks, "Where is order number 42?" And
14:49our model knows information about all
14:51the tools. It understands that it needs
14:53to call a function like "getOrder" and
14:55pass the order number 42 into it. That's
14:57it.
14:58We somehow interacted with the store's
15:00API, queried the database, set it all up
15:02ourselves, got the response, it went to
15:04the LLM, and the LLM wrote the final
15:06result.
15:08And that is everything it thinks about
15:09it. But here we have the next point:
15:11guardrails, the fact that we still need
15:13to secure all of this.
15:14We can use our code-level checks to
15:16verify that it didn’t make up an order,
15:19for example, that it established a
15:21secure database connection, and that it
15:23didn’t leak things like user IDs or
15:25confidential information.
15:27So there is endless room here for how to
15:29implement these checks. Checks can be
15:31algorithmic, using regex, or whatever
15:33else, as well as additional LLM calls.
15:36Meaning we just take the response and
15:38feed it into a separate model chat,
15:39asking it to verify if everything is
15:41okay. This is to generate the final,
15:42definitive response. Now, a big topic is
15:45how memory is configured for agents, for
15:47this whole harness. By default,
15:50everything basically works through a
15:52thing called skills. For example, we
15:54want to automatically pass information
15:57to the model about how we want to format
15:59merge requests or conduct code reviews
16:01in our project. We simply create text
16:03files, called skills, like agent.md, or
16:06whatever.
16:07There, we write down specific
16:09information for each skill it needs to
16:10know.
16:11But there is an obvious problem here.
16:12The more of these skills there are, the
16:14harder it is for it to navigate them.
16:16After all, each skill is a large text
16:18block that needs to be fed into the
16:20model. If it has to use a lot of such
16:23skills, we'll quickly eat up the entire
16:25context, and it will start performing
16:27worse. Therefore, this is not a super
16:29great solution. But there is a more
16:30precise approach, like using neural
16:32network search. Something like RAG. We
16:33can make it so that our model
16:35dynamically decides which skill it needs
16:37to use. Imagine we have 1,000 skills.
16:40It’s impossible to just read all those
16:411,000 skills. The context will get
16:43clogged. But we can create some
16:45capability so the model can
16:46automatically and quickly realize: "Aha
16:49, I need these." Specifically these
16:50exact skills. So they are the best fit
16:52for this topic. This is done through
16:54semantic or neural search. For this, we
16:56will need some kind of vector database.
16:58For example, this is already implemented
17:00through a tool like Hoonch. Yeah, its
17:02API will end up being paid, but there is
17:04a free tier at the start of usage. You
17:06can try using it. I always have it
17:09enabled by default, and it helps a lot
17:11because I work on many different
17:12projects in parallel, and it knows how
17:14to dynamically understand. "Aha, we are
17:16working on this right now." We need to
17:17pull info, for example, about how
17:19authorization works on the project. What
17:21does it do? In the process, within the
17:23chat, it can realize, "Oh, we're talking
17:25about something important, I should
17:26probably save this." It takes this text
17:28from our conversation and feeds it into
17:31a separate neural network, which
17:33generates a vector from this text.
17:36For instance, if there were 2,000 tokens
17:38of correspondence, it converts them into
17:41a vector of numbers, say 512 in length,
17:43and stores it in a vector database.
17:46That's it. So, it sits there somewhere
17:48in the vector DB. Then, in the
17:49background, it can summarize all these
17:51texts and draw conclusions using its own
17:54model on the server.
17:56Later, when we start working on a
17:58project, we initiate a chat.
18:00"Hey, Copilot, fix this authorization
18:02thing for me."
18:03It automatically goes into the vector
18:05database and compares these vectors.
18:07It finds the nearest vector. It compares
18:10them by cosine similarity. Essentially,
18:11it looks like this. We have many points
18:14in 512 dimensions. Our database consists
18:17of a multitude of these points. And our
18:19query comes in. Here is the input query
18:20. We convert it into a vector as well.
18:22And it sits right here.
18:24We see that this red point is closer to
18:26them, to this specific area.
18:29So, let's find the closest vector from
18:31here. This one turns out to be the
18:33closest. And we see that this vector
18:34contains important information about
18:36authorization. That's all. And this is
18:38exactly what we will pass into the
18:39context. We don't need to pass those
18:42other guys. These ones, let's say, we
18:43don't need either. We will pass only
18:45this most important and valuable info.
18:46Everything is ready. Thus, we save a lot
18:49of context. There we go. And you can
18:51come up with a ton of such features and
18:54additions. We can additionally store all
18:56these facts in another database—not
18:58necessarily a vector one, but a regular
19:00DB where we can apply extra filtering,
19:03like by date—take the latest important
19:05events from the past week, and feed only
19:07those into the context.
19:09There are many options here, but that’s
19:11the basic idea. That's it. Now we've
19:12learned how to further optimize the
19:14context and compress it. And it's
19:16important that it's better to solve
19:18separate tasks with separate chats. So,
19:20you shouldn't stuff all the information
19:21into a single chat. There is such a
19:22thing as effective context size. If the
19:24model's total context is, say, a million
19:26tokens, then the first 100,000 tokens
19:28are its effective context, where it has
19:30the highest accuracy. As you consume
19:32more, for example, 500,000 or 800,000,
19:34its quality may just keep dropping and
19:37dropping. Therefore, if you've hit, say
19:39, 100,000 in context, you can stop there
19:41and switch to the next chat. Just ask it
19:43to summarize the text, pass it to the
19:45next conversation, and start from there
19:47. Then, most likely, the accuracy will
19:48be better. You might notice that the
19:50model becomes less lazy and proactively
19:52takes more actions, like performing
19:54checks, validations, and so on.
19:56So. And our final point is how to do all
19:59this in production.
20:01These agentic systems are very difficult
20:03to validate in the first place, because
20:05we have so many individual operations. A
20:07user request arrives. We need, for
20:09example, to determine exactly what they
20:10want to do, which tools to call, and how
20:12many context tokens we will spend on
20:14these tools. Because the more tokens we
20:16spend, if we use models via API, like
20:19GPT or Claude, the more money we spend.
20:21That's why this whole thing needs to be
20:23optimized. There are special libraries
20:25that allow you to trace agent runs, that
20:27is, to see each of its steps.
20:29Which tool it called, what conclusion it
20:31drew from that tool. This is all done
20:32automatically, for example, via the
20:34LangSmith library.
20:36We set up agents basically on something
20:38like LangGraph, and we can automatically
20:40log and collect all that info.
20:43So. And based on this info, we do our
20:45observability, understanding that we
20:47spent this many tokens here, and that
20:49many on that call.
20:51For example, calling this MCP spent,
20:53well, an incredible number of tokens.
20:55Because of that, we slowed down and
20:56didn't really grow in quality. And when
20:58we know each of these problems precisely
21:00, we can solve them specifically. We
21:01know the request time for each of these
21:03things. We know, for example, how many
21:05times the model might have entered a
21:07loop itself. For example, it called a
21:08tool incorrectly and led itself into a
21:09loop. Then into a loop again. So we
21:11measure specifically our downtime and
21:13where we are making mistakes. So. And
21:15finally, we also need to be able to
21:17measure all this quality, to do
21:18so-called evals. So. Yes. What I
21:20described before is also part of Evals,
21:22but we might also need something like an
21:24LLM-as-a-judge here. For example, we
21:26were solving the task of product
21:27recommendations for a user. Here is an
21:29example of such an agent in Yandex Lavka
21:31. You can go in and type yourself: "
21:33Please, please find me some Coca-Cola
21:35and chips." Well, there can be a great
21:37many such requests there. It might be
21:39some kind of vague, unclear question,
21:40and you can't perform any automatic
21:42validation or calculate our accuracy.
21:44That is why we will need a neural
21:45network as a validator. We will collect
21:47, for example, user stories, like: the
21:49user wrote this request and received
21:51this response. But how do we know if it
21:52was a good response or not? We can use
21:54an LLM, an LLM judge, that will validate
21:57all of this. Practically all the
21:58solutions that exist are validated in,
22:01well, exactly this way.
22:03And even all the benchmarks for LLMs
22:05that are used for comparison, like GPT
22:07or Claude, are most often benchmarks in
22:09math, coding, or tests where you have to
22:11choose the correct answer, because it is
22:13very easy to validate them there, so you
22:15don't need an extra LLM.
22:17We simply know if the answer is correct
22:19or not.
22:20Therefore, it's hard to use, for example
22:21, texts where translation accuracy is
22:23evaluated—say, translation from English
22:24to Russian—as benchmarks, because there
22:26are discrepancies; we might say, "Well,
22:28this translation is correct, and this
22:29one is correct too."
22:31Meaning, there is no 100%correct answer
22:32here. But in math, coding, and tests,
22:34there is a 100%correct answer. That's
22:36why it's easier to measure their quality
22:37there. And before we roll all this out
22:39to production, we can automatically run
22:41a very large number of different
22:43scenarios in our sandbox.
22:45We just collect a separate library of
22:46requests and we know, more or less, what
22:48the response should be, what tool calls
22:49should occur in the process, and where
22:51it shouldn't go.
22:52This is to ensure the LLM doesn't
22:53accidentally go and make an unnecessary
22:55database query.
22:56Before a release, we essentially update
22:58our agent harness system and perform
23:00many of these runs. We look at what it
23:02did, whether it accessed the right tools
23:04, and whether it interpreted their
23:05output correctly. And we can measure our
23:07model's metrics in advance after this
23:09trace and understand whether this agent
23:11can be rolled out to production or not.
23:13For instance, if it's doing some
23:14nonsense in there. Well, in reality,
23:15that's practically all there is to it.
23:17That’s the end. These are all the basic
23:19terms for working with an agent and a
23:21harness. There is nothing complicated
23:22about it. Everything is, well, super
23:24obvious. All you need to do is write the
23:27functions yourself. Python already has
23:29built-in tools for this from LangChain
23:30and LangGraph that allow you to document
23:32these functions and pass them into the
23:34neural network's context. Essentially,
23:36what remains is this kind of targeted
23:38technical work, cleaning up the mess.
23:41For example, measuring how many tokens
23:43we've spent or setting up validation.
23:45That is one of the most difficult tasks
23:46. I mean, it’s not hard to have Codeium
23:48or Claude Code build you some kind of
23:50agent. The hard part is validating it
23:52afterwards and collecting data to
23:53evaluate that it’s working well. All of
23:55this takes a lot of time and human
23:57resources, because a neural network
23:59often doesn't know what it's doing. You
24:01need some kind of external check.
24:02Actually, this is roughly what an AI
24:04engineer does at work. You can find more
24:07subject-specific questions on what the
24:09AI engineer selection process looks like
24:10in my profile, as well as on my website
24:12, which has a database of questions on
24:14ML, product analytics, Python, and all
24:16that. And you can find a collection of
24:18videos on how to build your own agent
24:20with LangGraph and master basic concepts
24:22in LLMs and AI engineering on my
24:24Telegram channel.

About this transcript

This page contains the full transcript of Всё, что нужно знать об AI Agent Harness за 19 минут | Loop, Eval, Tracing, RAG by Рома Филонов | AI, больно, generated from the public captions YouTube serves with the video. The transcript has 4,936 words across 751 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.