YT YouTube2Text

YouTube2Text → Transcripts → How to Debug, Evaluate, and Ship Reliable AI Agents with LangSmith

How to Debug, Evaluate, and Ship Reliable AI Agents with LangSmith — Transcript

by LangChain · 10,462 words · 1,430 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:02All right. Um, so we'll go ahead and get
0:05started. Um, today we're going to be
0:06talking primarily about how to um,
0:10essentially debug, evaluate, and ship
0:11reliable AI agents. Um, and we're going
0:13to talk about it in the context of
0:15Langmith. Um, so a lot of what we see
0:18today um, is built into our Langmith
0:21product in terms of the way to ship
0:23reliable and production agents. So I'm
0:25excited to jump into all of those
0:27topics. I'm I'm Victor. I'm one of our
0:29deployed engineers here at Langchain. So
0:31really excited to get into all of this
0:33and if you have any questions afterwards
0:35um please feel free to reach out to me
0:37or reach out to our team and we can
0:38definitely um help you along your agent
0:41building journey. Cool. Um so go ahead
0:44and jump right into it. So um let me
0:49There we go.
0:51Cool. [clears throat] So um the AI
0:54imperative is here. Um this is something
0:55we talk about a lot is you know every
0:57company is trying to figure out how to
0:59use AI. Um we're seeing generally that
1:02the challenge of AI is less of like a if
1:05we're going to do AI and more of when
1:07and the speed and reliability at which
1:09you can do so is very very important. Um
1:12so that's kind of the high level of of
1:14what we're seeing with AI with different
1:16companies. um essentially wanting to
1:18generate more revenue per employee um
1:20improving their products and experiences
1:22and also um increasing the efficiency of
1:24your internal teams. Those are kind of
1:25like the three paradigms that we see
1:27people wanting to use AI for. Um a quick
1:30little poll here. I'm just curious where
1:32everyone might be in their AI journey,
1:35whether they're still figuring it out.
1:36Um actively piloting some AI initiatives
1:39currently building agents already have
1:41agents in production. Um would just be
1:44curious to hear where you're all at
1:45today. Cool.
2:00Um, so you should be able to respond to
2:02that panel. Um, let me check on my side
2:04really quickly.
2:17Let me pull this up.
2:21Awesome. Thank you. Cool. Um, yeah. So,
2:23that's exactly where um, you know, we we
2:25see most people kind of start coming to
2:27Langmith is is when Okay, cool. I just
2:30got the results. So, it looks like the
2:31majority are still looking to scale
2:33agents in production. Um, and we're
2:35seeing that there are a lot of different
2:37people in this kind of phase. um and
2:39still kind of also piloting different AI
2:41initiatives and trying to figure out
2:42where to use agents. Um this is exactly
2:45where LinkSmith can come into play and
2:46help you build reliable agents. Um so
2:50building with AI is a little bit
2:52different than our traditional software
2:54engineering workflow, right? Um
2:56traditionally if you're building in
2:57software, you would have very
2:58deterministic code that you could see it
3:00whether it compiles or not. Um and
3:02that's pretty pretty easy to determine
3:05um if your code is going to work or not.
3:07with LLMs, we started to see more
3:09constrained apps, right? So, we have
3:11like one LLM call, like a one-time call
3:14with with just that context of that
3:16particular LLM run. Um, and that has
3:18been probably the most common way that
3:20we see people get started with LMS is
3:22just like some some generation or some
3:25sort of like filling out of a of a
3:26document or something like that on
3:28behalf of the u user. So, those are kind
3:31of fuzzier. Um, they're still kind of
3:33easy to see if they're working or not.
3:35you know, you'll see like a a an error
3:37if it doesn't run properly, things like
3:38that. Um, as we get into agents, this is
3:41where the really interesting um portions
3:45of evaluating and building with AI and
3:48where things really start to differ from
3:50traditional software with agents. Um,
3:52the majority of our logic will now live
3:54in our agents reasoning. So, um, if an
3:57error happens, if things like that
3:59happen, we can't really go into a stack
4:01trace and see what happened. um or we
4:02can't look at our code and and try and
4:04figure out where in our logic there's
4:06something wrong. We need to kind of be
4:08able to dig into what's going on with
4:10our agents. So, it's really important to
4:12be able to have the observability to go
4:14in and start to evaluate our agents. And
4:16this is one of the core things of of
4:18Langmith and one of the core things that
4:20we've seen building a lot of agents with
4:22different enterprises is your agent uh
4:24your agents and app logic no longer
4:26lives fully in code. there's kind of
4:28this like uh different space within the
4:30agents own logic that these things
4:32happen.
4:34Cool. So on to um Langmith itself. So
4:37Langmith allows us to basically turn
4:40these traces that we get from our agents
4:43into um a systematic improvement or what
4:46we refer to as a flywheel. Um and this
4:48is done using our agent engineering
4:50platform. So the idea is is that we
4:52start to observe our agents in
4:54production using our tracing and
4:55monitoring features. we can then start
4:57to evaluate them and curate data sets
5:00based on um that production data. From
5:03there, we can then redeploy our agent or
5:05deploy it for the first time using
5:06deployments um and then kind of continue
5:08on in this flywheel. So, we're going to
5:10go through all of these in high levels.
5:12So, I'm excited to kind of jump into
5:13them. So, um let's go ahead and do that.
5:19Great. So, [clears throat]
5:21here we are in Langmith. We're in a
5:23little uh demo workspace that we have
5:25here. And LinkSmith is a really good way
5:28to be able to understand what your
5:30agents are doing. So on the far left,
5:33you'll see tracing, monitoring, data
5:34sets, etc. What we'll do first is um you
5:38know, see how we can set up tracing. So
5:40if you click here on this tracing quick
5:42start, you'll see that um with Langchain
5:44and Langraphph, our open source
5:46frameworks, it's really easy to get
5:47started with tracing. Um, all you need
5:49to do is set these environment variables
5:51here and everything's ready to go.
5:53Additionally, we have integrations with
5:55all of these different popular
5:56frameworks. So, if I click here in our
5:58docs, you'll see we have, you know, all
6:00of the most popular agent frameworks out
6:02there um for easy setup and easy
6:05tracing. Additionally, you can manually
6:07instrument um your code as well in
6:10Python and TypeScript. And we also if I
6:13go back to our integrations um we do
6:16have the ability to trace with open
6:18telemetry as well. So this is another
6:19popular option that we see.
6:22Cool. So that's how we can set up
6:24tracing on an existing agent. Um one of
6:27the agents that I really like to demo is
6:29called chat lang chain. So anyone can go
6:31to this today. Chat lang chain is fully
6:34open to the public. Um and I can select
6:36a different model to use. Let's just use
6:38uh a fast model here. And I can say how
6:40do I use lang to improve my agents.
6:46Cool. So um chat lang chain is built
6:49actually um with lang chain and lang
6:51graph. It is all plugged into lang. So
6:53all of the tracing and observability
6:54that we'll see today. And then lastly we
6:56have the ability to deploy on lang. So
6:59it is actually also deployed on
7:00linksmith. So we can see here that it's
7:02giving us a nice trace on the agent's
7:05performance and that looks like it did a
7:07great job. So this is just a demo of
7:09what this agent does. Now to kind of get
7:11into the logic of what just happened is
7:13I can jump into our tracing projects
7:15here. I can filter down on just chat
7:18lang chain as an application and we can
7:20see just the resources associated with
7:22chat lang chain and then from there I
7:24can open up our actual project.
7:28Great. So you'll see in the last day or
7:30so. So we're we're filtering by the last
7:31day. I can see the last 6,000 runs here.
7:34Um in the past day I can also like set a
7:37different time period. So I can go to
7:38last week and I can look at traces from
7:40last week if I wanted to and start to
7:42understand how our agent is performing
7:43then. Um and we can pop open any one of
7:46these traces. So I could pop this open
7:49and we can see exactly how this agent
7:51works. So on the far left side in the
7:53tracing view um we'll be able to see the
7:55different steps that our agent has taken
7:57and you'll see here that you know the
7:58first step is we jump into this
8:00guardrail here. It then goes to this
8:02guardrail LLM call and anytime we make
8:04an LM call we can see the system prompt.
8:07We can see the tools I had access to. So
8:09we can see this is a pretty basic
8:10guardrail. It just basic just just
8:12checks like is this question related to
8:14lane chain or langraph and this allows
8:15us to gate our actual agent from you
8:17know malicious queries or prompt
8:19injection things like that. So this is
8:21actually built into the architecture of
8:22our agent and then from there it
8:24responds with whether the decision is
8:26allowed or rejected. It then continues
8:28into this model tool calling loop. So
8:30this is a simple like reason and action
8:32style agent right? So it has access to a
8:34few different um has access to a few
8:37different tools. So if I jump into this
8:39LM call, we can see it can search our
8:41docs, it can search support articles,
8:43check links, etc. We have a nice system
8:45prompt here and our agent decided to
8:47call two tools and then it finally
8:50responded back to the user. So we can
8:51see that full context gets sent to our
8:54LLM here and it finally responds. If I
8:57click on output here, we can see the
8:59update and right here is the full
9:01response. So this is really nice to be
9:03able to debug our agents. We can also
9:05jump into a waterfall view and we can
9:07see um related to latency uh kind of how
9:10all of these steps took place and it's
9:12really nice to be able to see all of
9:14these happen. Um and we can see for
9:16example these tools were executed in
9:17parallel and we can see that that
9:19drastically reduced the latency of our
9:21agent. So that's really nice. Um one
9:24thing that we have inside of Langmith as
9:27well is Pauly. So on the far right side
9:29we'll see Paulie is here. I can ask
9:32Pauly any question. So Polly can help
9:34you filter things. It can answer
9:35questions you have related to Langmith.
9:37Um it can show you recent errors, things
9:39like that. But I can ask Pauly to filter
9:41down
9:43on runs from the past two days.
9:51Poly will always have the context of the
9:54pages that you're on. So it's able to
9:56see what page you're on and it's able to
9:57just show you. For example, I asked it
9:59to show me the runs and the runs would
10:00show us all the intermediate steps. I
10:02can switch it back to traces and we can
10:04see all the traces from the past two
10:07days. So this is a great way to interact
10:09with our agents.
10:11Cool. So [clears throat] one other thing
10:13that we can do is view our traces in a
10:15thread style view.
10:17So threads um you can think of as like
10:19full conversations. So if I continue
10:21this conversation here and say like hi,
10:23you can see that we're going to kind of
10:25continue on to a second turn of the
10:26conversation.
10:28It can be really useful to see how your
10:29agent performs over multiple turns of a
10:32conversation. So what we can do in this
10:34thread view is pop any one of these open
10:36and we can see sort of the back and
10:38forth between our user and our agent and
10:40we can see all of these different steps.
10:42I can switch to a trace view again and I
10:45can go to like step nine or sorry turn
10:47nine. So this is like the ninth back and
10:49forth that our user and our agent is at.
10:51And maybe our agent as that context
10:52window grows starts to struggle. So we
10:54can find traces and threads that are
10:56happening way down the conversation
10:58stream and see how it's performing as
11:00the context grows. So that can be very
11:03important as well.
11:05Awesome. So um we covered how we can get
11:07traces into Langmith and filter on them
11:10and and debug um exactly what's going on
11:12our agent using poly and being able to
11:14look at different things. We've seen the
11:15thread view. Um next, how can we like
11:18start to automate this and see how we
11:20can evalu?
11:23This is what we refer to as online
11:25evaluators. We also have offline
11:26evaluators which we'll get to when we
11:28get to data sets and experiments and how
11:29we can test and improve our agents. Um
11:32but essentially these online evaluators
11:34will allow us to observe our production
11:35agents and calculate how they're
11:37performing. So one of the things that we
11:40can do here is um so I see a question
11:43here about the option to eval sub agents
11:45which are using tools and skills on
11:47their own decisions and demand. So yeah,
11:49we can absolutely eval sub agents and
11:51full endto-end agents. There's there's a
11:53variety of different eval strategies
11:55that you can explore inside of Langmith.
11:58Um, one of the ways that we can set
12:00these up is on the top right. I can
12:02click new. Under new, you'll see this
12:04new evaluator button. And then you'll
12:06see we have a variety of different
12:07options for eval um for our tracing
12:10project. I can either do LLM as a judge,
12:12I can um create one from scratch, or I
12:16can create an LM as a judge from one of
12:17these pre-builts. I can also use a
12:20custom code evaluator um for more
12:21deterministic type uh queries. And then
12:23we also have the ability to evaluate a
12:25full thread. So that is really helpful
12:27for being able to determine how our
12:29agents performing over full conversation
12:31and is a pretty unique feature to
12:33Langmith. So to show you how to set one
12:35of these up, I'll show you one that's
12:36already created. So, for example, we
12:38have this kind of more simplistic
12:39helpfulness one that I think does a good
12:41job of demonstrating this. Um, when we
12:45set up our eval, we'll set up a name for
12:47it. We can set up a sampling rate, you
12:50know, from one to 100% of the traces.
12:52I'm going to leave it at 10%. Um, and
12:54this just helps control cost. So,
12:56what'll happen is as these traces flow
12:57into Linksmith, we can set a sampling
12:59rate for, you know, a random sampling
13:01rate to send these traces off to be
13:02evaluated.
13:04From there, we can filter down on
13:06different runs. So, the default for your
13:08evaluators is going to evaluate the
13:10entire run. Um, you asked about
13:12evaluating sub aents and things like
13:14that. I could remove this and I could go
13:16into a run name and I could say I just
13:18want to evaluate my guardrail, for
13:19example. We could do that. And now we're
13:22only going to be filtering down on the
13:24runs for our guardrail. You could do the
13:26exact same with a sub aent um and be
13:28able to evaluate just those sub aents as
13:30well. Um, but I'll leave this on is
13:32trace, which means is it an endto-end
13:34trace.
13:36Um, from there, what we can do is
13:38actually set up this eval. So, this is
13:39an LM as a judge. So, I do need to bring
13:41an LLM into the into the picture. Um, so
13:44in Langmith, we can set an API key um
13:47and use any one of our providers that
13:49we'd like to use. So, we have all the
13:51popular model providers here. We have
13:53cloud providers here as well, including
13:55Azure, Google, and Bedrock. And then we
13:57also can bring in custom models via
13:59OpenAI compliant endpoints. So we just
14:01need to make sure that they're OpenAI
14:02compatible and we're able to bring in
14:04any custom models as well. So um I'm
14:06going to leave this on GBT5 Nano for
14:08now.
14:10From there we can set up a system
14:11prompt. So we can say you're an expert
14:13um evaluator assessing how helpful a
14:16chat's response can be. So this is
14:18basically going to score on a 1 to 10
14:20how helpful our agent was for our user.
14:23From there we will pass in the original
14:25user's message. So all we're doing here
14:27is injecting in a variable and and
14:29finding that original user's message
14:31which you can see highlighted here. And
14:33then we'll grab the last message from
14:34our AI or from our agent. So we'll send
14:36those off to be evaluated. We can then
14:38set a feedback kind of configuration. So
14:41we can say on a scale of 1 to 10. I
14:43could also say boolean or I could even
14:45make it categorical if I wanted it to
14:47dump it into different categories as
14:48well.
14:51Great. So once that is set up, all of
14:53these traces will start going through.
14:56And the other thing that we can do is we
14:59can also set up a multi-turn evaluator.
15:02So the way that this will work is kind
15:04of same thing. We'll hit that add new.
15:06And I'll show you one that's already set
15:07up. So this one we're going to be trying
15:10to see if our agent actually
15:11accomplished the task that our user
15:13initially set out to do. And the way
15:15that we'll do that is again we can set a
15:16sampling rate. But what's a little bit
15:18different about these thread evaluators
15:20is we can set a thread idle time here.
15:22So this will be how long we'll wait
15:23until the last user message to mark a
15:26thread as completed. So for chat lang
15:28chain we have it set to an hour. I can
15:30also set it to a day. I can set it to 7
15:31days however you might have won.
15:34From there I can go ahead and you know
15:37set up the same exact way that we set up
15:38the other elements as a judge a system
15:40prompt a human message. But what's
15:42really nice and and kind of like a
15:44really nice DevX here is that we can
15:46basically select which messages we want
15:48to pass to this evaluator. So I can grab
15:51all the messages and that would include
15:52all of the intermediary tool messages,
15:55sub aent calls, etc. Um, but that might
15:57be a little bit noisy and a little bit
15:58too much context to send an LM as a
16:00judge. So I can actually just grab out
16:02the human and AI pairs, which will be
16:04the human messages, the final AI
16:07response, the human message, the final
16:08AI response. So, it's just going to be
16:09the back and forth between the user and
16:11the agent. And that'll kind of map to
16:12what you see here in the UI, right? Like
16:14what the user experiences here.
16:16So, we can go ahead and do that. And
16:18then we can set a boolean of whether
16:20this user's issue was resolved over the
16:22course of the conversation. So, this can
16:24be a great way to evaluate how your
16:25agent's performing. When you think about
16:27how agents work in the wild, a lot of
16:28times they'll they'll generate responses
16:30that look semantically correct, but the
16:32user might immediately after be like,
16:33"No, that's wrong." Or, "No, I tried
16:35that and that didn't actually work." Um,
16:36so it's good to see over the course of a
16:38conversation if the agent is actually
16:39able to accomplish the goal.
16:43Awesome. So that is online evals um in
16:46the context of LLM as a judge and thread
16:48level evals. You can also do custom
16:49code. Um I'll kind of skip over that
16:52today just to make sure we cover all of
16:53the different topics we're looking to
16:54cover. But essentially we could just
16:56write custom code that will run in a
16:57sandbox. Um it can't make external
16:59requests. So it's just more for like
17:01deterministic things that you're trying
17:02to evaluate. So we have one that checks
17:04for example if lang chain products are
17:06mentioned using code and it just looks
17:08for strings. Um cool. So when these
17:10evaluators run they're actually going to
17:12write feedback either on threads or on
17:14the runs. So if it's a normal element as
17:15judge it's going to write to our runs.
17:17We can actually filter down and find
17:19traces that this happened. So here we
17:22are. We can go into this LM as a judge
17:24or sorry we can go into this tracing
17:26project and you can see I have a
17:27pre-made view for helpful traces. And
17:30all this does here is it's going to
17:31filter down on feedback scores where the
17:34key for helpfulness is greater than or
17:36equal to eight. So now we have all of
17:38these traces that got high helpfulness
17:40scores. And I can pop any one of these
17:42open and we can see, you know, this one
17:44got a score of nine. I can open this up
17:47further and we can see the reasoning as
17:48to why it got a nine. So our our LM as a
17:51judge will also include reasoning as
17:52well. So that's really helpful. Um, so
17:55that's how we can kind of find all of
17:56these traces and start to see what's
17:58going on here.
18:00Great. Um, I'll talk in a little bit how
18:02we can use automations to actually
18:04automate this process of of grabbing
18:05these traces and sending them to more
18:07interesting places. But, uh, before I
18:09jump into that, um, I do want to show
18:11you insights really quickly. And
18:13insights are going to be how we can
18:14start to understand how our agents are
18:17performing. Once you start getting, you
18:18know, hundreds or or thousands of traces
18:20and you're trying to um essentially
18:23ensure that your agent is going to be
18:25performing as you want it to and you can
18:27find these edge cases. You can find
18:29different ways that users are using your
18:31agents. As anyone that has worked in
18:32software or product knows, people tend
18:34to use your your agents or your products
18:36in ways that you don't intend. So, it's
18:37really helpful to be able to find those
18:39um in real time. So, we have a few
18:41different examples of insights reports
18:43here that we've ran. To create a new
18:45one, you just go up here in the top
18:46right and we'll click new insights
18:48report. And you can do this manually
18:51where you'll kind of go through and set
18:52all these different things. Um, we also
18:54have an auto setup where you just start
18:56chatting with um, our insights agent and
18:58it will basically configure this
19:00insights job for you. But essentially
19:02what's going to happen is our agent is
19:04going to go through and cluster and
19:06categorize all of these traces depending
19:08on what you're trying to find out. So,
19:09if you're trying to find failure modes,
19:11it's going to find all of the traces
19:12that your agent made uh mistakes and
19:14start to cluster those together into
19:16different categories. So, you can see
19:18what types of questions are leading to
19:19the most errors. Um, if you're doing
19:21like a broad usage pattern one, it'll
19:23show you all the different ways that
19:24people are using your agent. Um, so
19:26that's kind of up to you how you want to
19:27categorize things or have your agent
19:29analyze them, but you can also include
19:31attributes alongside your insights that
19:33will help you um parse through all of
19:35this data. So, it'll basically just like
19:36an online evaluator, it'll show you
19:38these different metrics related to these
19:40clusters or these categories. And that
19:42might sound a little confusing. Um, but
19:44let me switch over into one of these
19:46that has already ran so we can kind of
19:48see what this looks like. So, you can
19:49see we have these kind of three that
19:51run. Um, you can set a schedule if you
19:53want. You'll be able to see the
19:55distribution over time. So, it's going
19:57to go ahead and start categorizing
19:59things for us. So we can see here that
20:01um we are basically trying to identify
20:04how people use chat lang chain and I
20:06wanted to put these top level categories
20:08for it to cluster them into. So um we
20:10asked it to categorize it into lang
20:12chain, langraph, lang and deep agents. I
20:15can also um not put any categories and
20:18it will automatically create those
20:19clusters. So if I go to like Q&A topics
20:21for example, it just came up with these
20:23categories on its own as it was
20:25summarizing and grouping together these
20:27these different um categories. Uh so
20:30that's that's essentially how the deep
20:32or sorry how the insights agent will
20:33work. So if I go back to this usage
20:35pattern one, you see that it starts to
20:36give us insights on all of this data
20:38that it's analyzed and summarized. So
20:39this is like over thousands of traces.
20:41It's gone through and seen how people
20:42are using um chat link chain and it's
20:45saying you know people that are chatting
20:46with chat link chain are building
20:48production agents not prototypes. Rag
20:50still dominates around 30% of the the
20:52usage in terms of how people are asking
20:54chat link chain questions. Um and people
20:57are having some issues with some growing
20:58pains with the ecosystem. So that's
20:59helpful to know. Um and we can see the
21:01distribution between two different runs.
21:03So this happened on March 1st. This
21:05happened on March 8th. We can see kind
21:07of how things are changing over time and
21:08how people are asking different
21:09questions. Um and if I jump into one of
21:12these clusters here, so I can go into
21:15here. We can see what questions people
21:17are asking about link chain versus link
21:18graph versus deep agents. So these are
21:20all our open source packages and then
21:23also in langmmith. So what are people
21:24asking about linksmith and I could dive
21:26into any one of these uh subcategories
21:28here. We can see summaries of these
21:30different traces. I can also look at
21:31these individual traces one by one as
21:33well. So very very helpful for
21:36understanding your agents performance.
21:37And one little anecdote here of how
21:39we've used this um actually at Langshade
21:42is for chat lang chain when we released
21:44our deep agent package which is our um
21:46open- source agent harness uh very
21:48similar to like claude code or agents
21:50like manis um we were really excited
21:52about it we released it we blasted on
21:54socials and then people went to chat
21:55lang chain to ask how to build with deep
21:57agents um but we had failed to update
21:59that guardrail that you saw earlier so
22:01everyone was asking about deep agents
22:02and they were getting blocked by our
22:03guardrail so we saw a big big um kind of
22:06issue or error analysis or failure mode
22:08under deep agents and we were able to
22:10identify that very quickly and find out
22:12that oh it was just our guardrail that
22:14was off. Everything else is fine. We
22:16really quickly update our guardrail
22:17prompt and everything kind of continued
22:19without a hitch. So really helpful to
22:21identify those edge cases that you may
22:23not consider or may forget to update
22:24when you're building your agents.
22:28Great. So that is our that is our um
22:32insights agent. So I talked earlier
22:35about how when we have feedback that's
22:37written to traces, how we can filter
22:38down on these here. The really nice
22:41thing about Langmith is, you know, when
22:43you have all of this data being
22:44collected, it's it's not really the best
22:46thing in the world to come in and filter
22:48on these and then grab them and then
22:49maybe add them to a data set or
22:51something like that. So what we can do
22:52in Langsmith is we can set up
22:54automations.
22:56So in this plus button on the top right,
22:57same place as the evals and the insights
23:00report, we can set up a new evaluator.
23:02So these new evaluators will allow you
23:04to automate that process for you. So for
23:07example, the helpful traces, if I open
23:09this up, I have some filters here where
23:12I'm filtering down where feedback for
23:14helpfulness is greater than or equal to
23:168. Um, so this is identifying, you know,
23:18traces that are getting either an 8 out
23:20of 10, 9 out of 10, or a 10 out of 10 on
23:22the helpfulness score evaluator. And I'm
23:24able to essentially filter down on those
23:27particular traces, set a sampling rate,
23:29and then I can do something with it. So
23:30I can automatically send that into an
23:32annotation queue which I'll show you in
23:33a moment. Um which is essentially a
23:35place for subject matter experts,
23:37product managers, etc. to to come in and
23:39review traces for you. We can also just
23:41add this directly to a data set if we
23:43wanted to. Um we could trigger web hooks
23:45to send it off to a different um
23:46different environment.
23:48So that's really helpful to be able to
23:50set up these. Um, the other thing that
23:52we can do is in chat lang chain here I
23:55do have the ability to collect feedback
23:57and this is a feature of lang
23:59deployments is with our built-in agent
24:01server you can get like thumbs up thumbs
24:03down um and collect feedback directly on
24:05the traces. So if I left a thumbs down
24:08um it might be a good indication that we
24:10should go and review that trace. So I
24:11can set up another automation that every
24:13time a user leaves a thumbs down, which
24:15would be a zero, I can actually send
24:17every one of those traces to a different
24:19annotation queue so that we can uncover
24:21why users are getting frustrated with
24:22our agent. So that's how we can set up
24:25automations. Um typically these will
24:27send to an annotation queue. So in these
24:29annotation cues, um this is an example
24:31one that we have here that is for that
24:33helpfulness score. So if it gets greater
24:34than or equal to 8 and we'll be able to
24:36see everything here on this um
24:38annotation queue for subject matter
24:40experts and PMs and people alike we can
24:43set up instructions for them. We can ask
24:45them to leave feedback on the traces. So
24:47whether it was correct or not. They can
24:48also leave notes to each other. So I
24:50left a note to myself in this case. Um
24:52but you can see those here.
24:55Additionally, what's really nice about
24:56this annotation cue is as a reviewer, I
25:00might have way more context of what this
25:01agent should say in in reality, like the
25:04golden ground truth of what it should
25:05say here. Um, and those can be very very
25:08valuable when you go to test your agent
25:10later on. Um, even if your agent made a
25:12mistake, it's helpful to correct that
25:13mistake and save that as part of your
25:15data set to give it a hill to climb as
25:17you start to try and improve your agent.
25:18Um, and we'll talk about that in a
25:20moment as well. But what I can do is I
25:22can start to remove these. So, let's
25:24just remove these intermediary messages
25:26because, you know, I don't want to cloud
25:27the final output um with all of these
25:29different tool calls and intermediary
25:31messages. So, I can remove some of
25:33those. Now, we arrive at the final
25:35message. So, we have the original human
25:36message from chat lang chain and the
25:38final AI message that our agent made. I
25:40can hit edit here and I can start to
25:42clean this up. So, maybe I don't want
25:44chat lang chain to actually ever provide
25:46code. I just want it to link to the
25:47appropriate docs when relevant. So, I
25:49can go ahead and grab all of this code.
25:52I can delete it [clears throat] and I
25:54can go ahead and hit save. And now we
25:56just have a nice response that is just
25:57going to link uh kind of broadly explain
25:59the topics and then link to the relevant
26:01docs. So once that's done, I can go
26:03ahead and hit add to data set and we can
26:05add that to a data set for further
26:06testing. So I have this chat lane chain
26:08golden data set and I can go ahead and
26:10add it to that. Once that gets added,
26:12we'll see it in this data sets and
26:14experiments. So right here I have this
26:16chat chain golden data set and you'll
26:19see here we have a bunch of examples
26:21that are stored. We have around 10 in
26:22this demo data set. I typically
26:24recommend anywhere from like 50 to 100
26:27um once you're kind of actually trying
26:29to improve your agent and get some good
26:31data on it. And you'll want to have um
26:33you'll want to have data in your data
26:35set that's on both ends of the spectrum.
26:36So, kind of examples that are, you know,
26:39tried and true that your agent should
26:40always be able to answer and then
26:42questions that are kind of difficult,
26:43um, but you want your agent to be able
26:45to answer in the future. And you'll want
26:47to have those reference outputs mapped
26:48to exactly what you want your agent to
26:50say in the wild in production. So, we'll
26:52curate that data set of reference inputs
26:54and outputs and then we can start doing
26:56experiments. So, experiments are going
26:58to be offline eval.
27:01And there's a few different ways that we
27:02can do it. on the top right. We can
27:04either run these in playground and
27:05that's great for testing like a single
27:07step of your agent. We can also run them
27:09in lang studio which is you know
27:11available if you're using link chain and
27:13langraph. You can run a local server
27:15very very easily from our CLI and you
27:17can go ahead and test your agent in
27:19studio. The final is going to be running
27:21it via the SDK. So if I click run an SDK
27:25you'll see we have a small little guide
27:26here. Um basically you just need the
27:28langu client. You'll point to the data
27:30set by its name. Define whatever
27:33evaluators you want to run. So, similar
27:35to the ones that we set up in the UI,
27:36you can do these in code. Um, and you
27:38can actually set up very robust
27:40evaluators in code since you'll have
27:42access to, you know, any libraries you
27:43might want, things like that. And then
27:45from there, we'd use this evaluate
27:46function to evaluate and log all of
27:49these different um test cases against
27:52our new agent. So you'll use evaluate.
27:55You'll pass in your target function
27:56which would be your agent or your LLM
27:58application. Point to the data set by
28:00its name to find any of the evaluators
28:02and let that run.
28:05So the other thing that we can do um
28:07before I show you like an experiment is
28:08you know I showed you how you could set
28:09up an evaluator in code when you run
28:11that evaluate function. You can also set
28:13them up in your UI. So I have this code
28:15checker one that I created and it's
28:17essentially the same process that we saw
28:18with the evaluators on the tracing
28:20projects. um you just define it in the
28:22UI. Um and what'll happen is once I have
28:25this defined here with my you know input
28:27output and reference output mappings
28:30what'll happen now is when I run these
28:32experiments here. So we see we ran a few
28:34different experiments it'll actually run
28:36that code checker no matter what. So you
28:38can have these like standardized
28:39evaluators on a data set that will run
28:40no matter what environment you run them
28:42in whether it's studio or playground or
28:44the SDK. So it can be a really nice way
28:46to kind of ensure that these certain
28:47evaluators run no matter what
28:49environment your team's running them in.
28:52Great. Um, so we see here that we wanted
28:55to test out chat lang chain against this
28:56golden data set and we wanted to
28:59basically try out four different models.
29:01We tried out haiku 45, GPT5 mini, GBT 52
29:05and sonnet 46 and I can kind of scroll
29:08over to the side here and we can see
29:10that we included metadata as to the
29:12provider we use, the model we use, etc.
29:14So it can be really easy to filter down
29:15on this later. So, it's always nice to
29:17include metadata so that you can kind of
29:19understand what different knobs you
29:21turned with your agents. And these
29:22experiments are going to be a great way
29:23to test out different um architectures,
29:26different prompts, different models, um
29:28anything that might be a change that you
29:30want to make in your agent and ensure
29:31that it actually improves your agent and
29:33doesn't cause any regressions. And one
29:35of the ways that we can do that is by
29:36setting a baseline, which is like a
29:37newer feature in Linksmith. So, we can
29:39set a baseline here. If I unset it,
29:41you'll see they all get kind of get
29:43mapped. But let's say I'm using um let's
29:45say I'm using Haiku 45 in production. I
29:48could go ahead and set that as the
29:49baseline. And then every time I run a
29:51new experiment, they'll all get compared
29:53to this baseline. So we can see, you
29:55know, every one of these more powerful
29:56models did a bit better than Haiku. Um
29:59but we can see like the latency went up
30:02um and the cost went up and things like
30:03that. So we can see these kind of side
30:05by side in this dashboard. But what I
30:07can also do is grab Haiku and I can grab
30:10let's say, you know, Sonnet looks the
30:11best to me. So I can go ahead and grab
30:13those two and I can go ahead and hit
30:15compare.
30:19So what's really great about this is,
30:20you know, I I'm looking at my traces.
30:22I'm trying to identify which one is best
30:23or sorry, looking at my experiments. I
30:25can actually go side by side and make a
30:27very nice comparison here. So we can
30:28see, you know, side by side which ones
30:31are doing better based on these
30:32different evaluators I set. So I set one
30:34related to accuracy, a code checker, um
30:36including docs if the if the docs that
30:38are included are relevant to the answer,
30:40faithfulness, which is essentially like
30:42hallucination. We can see latency
30:44metrics, we can see cost metrics, token
30:46count, etc. So we can see that um across
30:49the board it seems like Sonnet is a lot
30:51better um in terms of the way that it
30:53performed on those test cases. Um but
30:55the cost is nearly double. So this is
30:57one of those things that you might need
30:58to consider when you're building agents
30:59is like is it worth the improvement
31:01versus the cost and things of that
31:02nature. Um so that's essentially how we
31:04can use experiments. Um one thing I'll
31:06shout out here too is is Paulie is with
31:08you here as well. Um I can create a new
31:11thread here and I can ask it to ask uh
31:14what was the best experiment
31:18in your eyes.
31:21So, it's going to go ahead and look at
31:22our current data set and see if it's
31:24able to um basically evaluate that um
31:27and find the the best experiment. So, it
31:28says based on the experiments, Cloud 46
31:30looks like the strongest performer
31:32overall. GBDT5 mini came close but lag
31:34behind. So, that's really helpful. Uh
31:36Pauly could also suggest prompt
31:37improvements and things like that. Um
31:40but to show you that prompt improvement,
31:41what we can do is go into our prompt
31:43playground. So, I like to talk about the
31:46prompt playground in the context of a
31:48trace. Um, so if I go back into our
31:51tracing project here for chat lang
31:53chain,
31:57[clears throat and cough]
31:58we'll be able to see all of these traces
32:00flowing in. And again, I can filter
32:02down, you know, like last week or
32:03something like that. Um, and find these
32:06different relevant traces.
32:09And then in here, we'll be able to see
32:10any one of these LLM calls here. So we
32:12can see, you know, one, two, three
32:13different LM calls. Whenever I click on
32:16any run type that is the type of LLM, we
32:18can actually pull these directly into
32:20our prompt playground. And the reason
32:22why you might want to do this is you
32:23might notice a particular trace that
32:24your agent did poorly on. Maybe you
32:26filter down on one that it did poorly
32:27on. Um or maybe for this guardrail it,
32:29you know, it didn't allow something that
32:30it should have. We can come in here now
32:33and we can actually pull this prompt in
32:34with the context of the actual trace
32:36that we were looking at. So this pulled
32:38in the actual human message here where
32:40someone asked about custom middleware
32:42and it looks like it did well. It
32:43decided to allow I could also remove
32:45this and I could say what is the weather
32:48in SF.
32:51I could then you know just like those um
32:53online evals I can choose which model I
32:56want to use. So I could switch to
32:57entropic for example switch to a smaller
33:00model like haiku hit apply and I can
33:03give this a try again. So we see now
33:06because we have this output schema
33:07defined as decision um and it can either
33:10be approved or blocked. It's always
33:11going to either respond in approved or
33:13blocked. Um, and we see here that, you
33:15know, weather isn't being allowed, which
33:17is which is best. I I think that's a
33:18good idea that weather isn't allowed.
33:20One thing I can do is I can ask Poly to
33:22actually improve this prompt. So, I can
33:24go over here. I can click new thread and
33:26I can just say improve this prompt using
33:28best practices.
33:30What Poly will do here is it's going to
33:32use all of the best practices that we've
33:34seen from building a variety of
33:35different agents and working with
33:36different, you know, enterprises,
33:37startups, etc., and it's going to apply
33:40some of those best practices. So, um,
33:42one of those is being is most likely
33:44going to be using XML tags. So,
33:46typically, uh, XML tags are going to
33:48help your performance a lot. Um, the
33:50other is maybe some few shot examples,
33:52some positive examples of what it should
33:54or shouldn't. Um, and that'll really
33:56help improve the prompt over time. So,
33:58it might take Poly a bit to do this, so
34:00we'll see if it goes right away. Um,
34:03usually it takes like around a minute or
34:04so to go through the whole prompt and
34:06make all the updates. But, there we go.
34:08So it added some XML tags here, added
34:10some explicit context, some enhanced
34:12clarity, some concrete examples. Um, and
34:15this can really help us when we build
34:17our agents.
34:19Cool. Um, the next thing that we can do
34:21is, you know, for whether it got
34:22blocked, um, I can also like add in a
34:25dynamic variable. So I could say, uh,
34:27runtime allowed
34:31topics and we could say, um, let's say
34:35weather. And now it should go ahead and
34:37allow it. So that allows us to like
34:38dynamically inject in things that we
34:40want. So typically you could inject in
34:42things like memory or things like that
34:43in your actual agent. So it can be very
34:45[clears throat] helpful to to do that.
34:47Um from there you can go ahead and hit
34:48save and that'll save the prompt into
34:50the prompt hub. Prompt hub is where you
34:52get everything saved and versioned. So
34:54this works as like a repo under the
34:55hood. Um we'll see full commit
34:57histories. We can then pull in our
34:59prompts into our applications using our
35:01Langmith client um in both Python and
35:03TypeScript. And we would just use this
35:05pull prompt method and define it by its
35:07name and its commit ID here. So very
35:10very helpful to be able to pull these
35:11into our application.
35:14All right. Um so that is a high level of
35:17Langmith um observability eval um that
35:20sort of thing. There's so much more to
35:22Langmith as well including our
35:23deployments product which makes it
35:25really easy to deploy an agent in just a
35:27few clicks here. You just point to a
35:28repo, you set some environment variables
35:30and that is it. you're ready to go and
35:32deployed your your lang chain or lang
35:34graph agents. We also have agent builder
35:37here which is a no code way to build
35:39your agents. Um and it is a very very
35:42powerful tool. So that isn't in the
35:43scope today but I like to mention them
35:45since I think they're both fantastic
35:46products. Um so that is pretty much it
35:49for my presentation. Um I'll go ahead
35:52and shift over to a Q&A now if there are
35:55any questions. Um so I'll go through the
35:58chat now. Let me get this pulled up.
36:02and we can start to go. All right. So, I
36:05see some questions. Um,
36:08how can we use uh can Paul use whatever
36:11LM we want? Can I have an a bedrock
36:13configured to it? Um, that's a really
36:14good question. So, in here, if I go into
36:16our settings and I scroll down to
36:19integrations, you'll see model
36:20configurations. Um, and this is a new
36:22feature that shipped recently. Um, as
36:24admins, we can configure which models
36:26are available to our users and
36:28especially for things like poly
36:29insights, etc. Um, and yes, we're able
36:32to use, you know, any one of these cloud
36:33providers. You can set those cloud
36:35provider models here and then set that
36:37as the default model for Poly, etc. So,
36:39you have full ability to, um, do all
36:42this configuration here and set it for
36:43everyone in your workspace as well.
36:46All right. Um, all right. So, let's see.
36:51Can we use poly uh model thread? Yep. Uh
36:56use some enterprise SAS application.
37:00Sorry, let me make this a little bit
37:01bigger. It's kind of small on my screen.
37:04Okay, we use some enterprise SAS
37:06applications. They have started offering
37:07agentic capabilities within their
37:09platform. Will Langmith be able to
37:11integrate with and monitor such agents?
37:13Um yes. I I think uh majority of of
37:16these platforms that are offering agents
37:18are exposing them in some way for us to
37:20trace and monitor them. Um I would love
37:23to hear which particular agents you're
37:24you're talking about, but I know we have
37:26integrations for example with cloud
37:27code, cloud agents SDK, um you know, any
37:31of those those popular ones. Uh Bedrock
37:33agents, things like that um can trace
37:36directly. Uh if there's any agents that
37:38that you're using that you you don't see
37:40an integration for in our docs page or
37:41anything like that, please reach out. Um
37:43I'm happy to help and see if there's any
37:45way that we could get you um set up with
37:47tracing there. Um let's see. Langmith be
37:50able to integrate with and monitor such
37:52agent. Oh, sorry that was a cut off. Uh
37:55my problem is there's no way to know
37:56ahead of time which sub agents my agents
37:58will choose. Is there a way to give a
37:59dynamic eval? Um yeah, so this is
38:02something that I probably do off like on
38:04my data sets and experiments and my
38:06offline emails. So um we have a nice
38:08guide on this. Um, so I can find it and
38:12send it through.
38:14Um, one moment. Um, but essentially you
38:17can do what's called a trajectory
38:18evaluation. And these trajectory
38:20evaluations are going to be how we can
38:22decide uh for different inputs if our
38:25agent's going to choose the right sub
38:26agent, if that sub aent is going to use
38:28the right tools, etc. So like those
38:30those can be very very important when
38:32you're building out your agents is is
38:34not only like looking at end toend tests
38:36or you know what they're producing but
38:39actually just monitoring the trajectory.
38:41So you can actually just run evals on is
38:43it calling the right agents in the
38:44correct order things of that nature. So
38:46there are the docs here on trajectory
38:50evals.
38:52Cool.
38:54All right. Um,
38:57how do you evaluate your LLM as a judge?
38:59Evaluators are performing as as
39:01expected. Uh, this is a great one
39:03actually. We have a feature for this.
39:05So, um, this is common issue that we see
39:07people have. You can go into your
39:09evaluators here. I can click new
39:11evaluator either on a tracing project or
39:14in an offline uh, you know, data set and
39:16experiment. But if I wanted to create a
39:18new evaluator and kind of make sure that
39:20it's aligned to how I want it, we can
39:22actually create it from labeled data.
39:23And this will send us down what we refer
39:25to as the align eval path. And basically
39:27what'll happen is you'll select some
39:29runs to annotate yourself. You will then
39:32get sent into like a unique annotation
39:34queue to annotate that data. So you know
39:36you select like five to 10 runs or or 50
39:38runs whatever you want. You then as a
39:40user will you know for let's say I'm
39:42doing uh helpfulness right or I'll just
39:45say helpful.
39:48And what I'll do is as I'm going through
39:49I can say like this is helpful yes or no
39:52for these like 10 traces. And then I can
39:54start crafting a prompt for an LMS as a
39:55judge. And we'll basically get to see if
39:58our LM as a judge is aligned with our
39:59preferences as a human or what we would
40:01say as a human. So it's pretty pretty
40:03important to um do that as well.
40:07Great. Um how do I detect and measure
40:08hallucinations? Um typically you would
40:10want to in your ground truth have like
40:11the documents or data set or the
40:14documents that your agent is retrieving
40:15or should reference. Um, and you can
40:18check with your LMS judge if the final
40:20response is grounded in the information
40:21from those documents.
40:24Um, cool. How do we budget costs
40:26overall? Can we set thresholds? Um, yes.
40:28So, you can set thresholds in the
40:29settings. Um, it's pretty pretty
40:31straightforward there. Is there a simple
40:33way to evaluate tool call success
40:35whether it was the right tool or in the
40:36right sequence? Um, I think that would
40:38go back to trajectory eval. Um, I would
40:40probably do that. Uh, we have some
40:42guides on that in the docs. Um, and some
40:44notebooks for that as well. Um, from
40:46Chase, where do we set the API key for
40:48Bedrock? I tried in the provider secrets
40:50and could not get it working. Um, you
40:53might have some more luck if you go
40:54directly into the playground and try and
40:56hit um,
40:58Bedrock here. It should ask you to do
41:00these different things and then on the
41:02top right there's like the secrets and
41:03you can add all these different secrets.
41:05Um, there's also like an IM ro so you
41:07can set an IM ro um, in your AWS account
41:09and that will basically get the
41:11permissions to like refresh the tokens.
41:13Um, so that should should work.
41:16Um okay I don't understand how data set
41:19is used in the links the ecosystem does
41:22the idea is to use it as a retrieval
41:24database to insert gold examples for a
41:25few examples. Um no so I think uh I
41:28think a good way of thinking about this
41:29is like part of our agent engineering
41:31flywheel. So the idea is you know we
41:34have these traces flowing in right so
41:36these are like production traces um and
41:39we don't really have a way to compare
41:42how our agent should perform versus how
41:44it actually did perform. So the way that
41:46we can do that is we can curate these
41:47data sets of golden examples. So we have
41:50you know all of these you know we have
41:51these 10 examples. So for this given
41:53input this is exactly what I want my
41:54agent to say in the wild likely for this
41:57one as well. So for this given input of
41:59how to you know what is the linksmith
42:01Python SDK equivalent to the runs
42:02endpoint. So this is kind of a
42:04complicated question. There's a lot of
42:05different ways that it could go. So this
42:07is the reference output of like once it
42:09does all of its research and responds
42:10back to the user. This is what I want it
42:12to say and how I want it to say it. So
42:14what we can do now is I can go and run
42:17my production agent against all of these
42:19examples and see if it's going to
42:21perform well. Um and we can get a
42:23baseline. So that's what we did here for
42:25this haiku 45 is we got a baseline that
42:28for a given input.
42:31So this is the golden input. We
42:33basically passed that into our
42:34production agent using haiku45 and we
42:36got this output here. And this output um
42:39it looks you know decent to me and it
42:42might look decent to an LLM as a judge
42:44but the benefit of having this reference
42:46output is now it knows exactly how it
42:48should have responded. So we can have
42:50different things like this code checker,
42:52this accuracy checker on a scale of 1 to
42:54five, uh answer relevance, things like
42:56that. And we can see okay compared to
42:58our reference output, how does this
43:00output compare? Um and this is what
43:02really helps us improve our agents and
43:03start iterating. So we can see okay um
43:06for example for accuracy it got a one um
43:08because it made some factual errors.
43:09It's it said that uh to use the langraph
43:12SDK instead of the lang SDK um and
43:15that's a pretty big like error that it
43:16made here. So it should have defaulted
43:18to using the linksmith SDK for querying
43:20runs versus creating runs. Um and that
43:23was kind of the issue there. So what the
43:25benefit is is how we can start to
43:27improve and start to create this this
43:29whole holistic strategy of going through
43:31and building out these agents um and
43:33improving them over time. So now I can
43:35do all of this offline testing outside
43:36of my production environment. I could
43:38figure out how I could improve my agent
43:39to do better on these test cases and
43:41then I could deploy a new version of
43:43this agent into production that will
43:45work a lot better. Um, I hope that was
43:47helpful, Rodrigo.
43:49Cool. Um, is there a simple way to
43:51evaluate tool call success? So, I think
43:53we just answered that a moment ago. Um,
43:56blacklist, can we spec out what our
43:57agents should not do? Um, good question.
43:59I I I would probably put that in
44:01prompting. Um, or you could also have
44:03hooks. So, in your actual agent
44:05architecture, and this is more on like
44:06the agent building side rather than the
44:08linksmith side. Um you could have hooks
44:10in that process in the agent tool
44:11calling loop that if it does a certain
44:12thing you could kind of kick it out or
44:14cancel the run or restart it or
44:15something like that. Um so a lot of
44:17people are doing that now with like
44:18cloud code to ensure that it doesn't uh
44:21do like destructive things. You can they
44:23just recently added hooks which we have
44:24in our deep agents package as well um
44:26where you can just kind of like hook in
44:27and and prevent any dangerous actions
44:29from happening. Um
44:33cool. Can you comment on a typical life
44:35cycle strategies for developing high
44:37quality evals plus graders prior to prod
44:39approaches to online evals and how to um
44:42how online evals inform refinement? Um
44:46yeah, so this is a a great question in
44:48terms of like the life cycle. Um
44:51[clears throat] I think quality evals
44:53and quality experiments like start with
44:55great traces. So even if you're not in
44:56prod um passing in a bunch of different
44:59inputs into your agent and seeing how
45:00it'll respond um is really helpful to
45:03then have your graders go through change
45:05that data annotate it in a way that it
45:07should be and then adding that to the
45:08data set for more for better
45:10experimentation. I think like starting
45:12with a example of what your agent has
45:14already produced and then editing that
45:16to where you want it to be is a bit
45:18easier than trying to generate like
45:19synthetic data which is what a lot of
45:21people start with. Um, so I typically do
45:23that even like for my own use cases when
45:25I'm building agents. I'll just like come
45:26up with, you know, 10 to 20 to 50 inputs
45:29and I'll just pass them into my graph.
45:31Um, and then or into my agent and then
45:34from there I will, you know, use the
45:35annotation Q, clean them up, put them at
45:37exactly how I want them and then add
45:38them to the data set cuz it's a little
45:40bit easier than trying to like go
45:42through and synthetically generate data
45:43or have your your subject matter experts
45:45generate them, things like that. Um, and
45:47then on online eval side, I think these
45:49are more like benchmarks rather than
45:50ground truths. So I think your your data
45:53sets and experiments are really where
45:54you do all of your like ground truth
45:56testing. Make sure your agent is at like
45:58a point that you think it's ready for
45:59production. And I view my online eval as
46:02more of a way to monitor our agents as
46:04they're in production. Um and one of the
46:06ways that we can do that is if I go over
46:07to this monitoring tab, we do get a
46:10pre-built dashboard with every
46:11deployment. So you'll see things related
46:13to like your trace metrics, um your LLM
46:15calls, like your latency, your cost,
46:17etc. Um different tool calls your agents
46:20are making. So you can see you know by
46:21tool count which which tools are getting
46:23called the most. You can see the latency
46:25by tool as well which can be helpful. Um
46:27and then the error rate as well. So we
46:29can see like for example today um we
46:31usually don't have any errors but we can
46:32see today we're having like a 03% chance
46:35or 03%
46:38of the the time um our search docs is
46:40hitting an error and that's an issue
46:41with uh you know we use Mintifi's doc
46:43server. So uh there might just be an
46:45issue on the Mintify side that's causing
46:46that. Um, we can see run types and then
46:49we can also see these feedback scores.
46:51So, you can see here like our
46:52helpfulness score. We can see it over
46:53this past seven days it's been, you
46:55know, hovering around six or 7%. I know
46:57if this like drops or spikes down,
46:59there's something there's some kind of
47:00issue with our agent. Um, same thing
47:03with like task completion. We can see
47:04that it's doing a good job of
47:05accomplishing user tasks. If there's
47:07like a big dip or a big drop, there
47:08might be something wrong. So, yeah, it
47:10can be really helpful to just like
47:11monitor over time.
47:15Um, cool. Um, how are annotation cues
47:18surfaced to SMMES inside of LinkSmith or
47:20can we surface them in another UI and
47:22integrate? Um, yeah. So, uh, you can,
47:24um, but typically you can go ahead and
47:27start create a rule-based access um
47:29through or rather through our rule-based
47:31access control, you can create a role
47:33that is specifically for the annotation
47:35Q. So, you can have data reviewers come
47:37in. Um, there's no like seat charges in
47:40Langmith. It's all usage based ch like
47:42billing. So, you don't have to worry
47:43about like having too many seats
47:44allocated or something. Um, but you can
47:46have users that are just specifically
47:48um, able to access like this annotation
47:50queue in this particular workspace and
47:52they're able to come in and access it.
47:54Um, the other thing that you can do
47:56which, um, I wish I actually had set up
47:58um, that's actually a good idea to get
48:00this set up on this demo workspace is
48:01you can enable a custom output rendering
48:03and that'll allow you to basically
48:05iframe in your own front end. So if you
48:07have like very unique traces where you
48:09want to allow uh reviewers to like see
48:11them in a particular format, you can
48:13enable this like custom output rendering
48:15which will allow you to see that in a
48:17very unique way. So um I unfortunately
48:19don't have a demo of that right now. But
48:20yeah,
48:25cool. Um so it looks like I'm being told
48:28that is it. Um thank you for joining. Uh
48:30I really hope this was helpful. Feel
48:32free to reach out to me um on LinkedIn
48:34or or you know contact afterwards. Um, I
48:37think our team is going to kind of send
48:38you to a different screen to to do those
48:40things. But, um, I enjoyed this time
48:42today. Thank you for all the great
48:43questions and, uh, yeah, feel free to
48:44reach out anytime. Thanks.

About this transcript

This page contains the full transcript of How to Debug, Evaluate, and Ship Reliable AI Agents with LangSmith by LangChain, generated from the public captions YouTube serves with the video. The transcript has 10,462 words across 1,430 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.