n8n Evaluation quickstart — Transcript
Full transcript
- 0:00Evaluation is a great way for getting
- 0:02confidence in your AI workflow. It's
- 0:04essentially a form of testing and it's
- 0:06the technique that helps you move from
- 0:07an impressive proof of concept to a
- 0:09solid production ready workflow. In
- 0:12fact, it's pretty much essential to do
- 0:14anything reliable with AI. And if you
- 0:16don't believe me, take these guys word
- 0:18for it. Anthropic calls it the crucial
- 0:20secret behind good prompting. And Vel
- 0:22goes as far as to say your prompt is
- 0:24broken if it doesn't have evaluations.
- 0:27It's important because LLMs can't be
- 0:29reasoned about like code can. So they
- 0:31have to be measured instead. Evaluation
- 0:34provides a way of doing that
- 0:35measurement. In evaluation, you run a
- 0:37test data set through your workflow to
- 0:39see how it performs on a range of
- 0:41different inputs. So let's look at an
- 0:44example. Here I am building a workflow
- 0:47for classifying support tickets. It
- 0:50takes an incoming ticket in the format
- 0:52of a subject and a body and using AI it
- 0:56calculates a category and a priority.
- 1:00I've got it working end to end and now I
- 1:02want to get a sense of how reliable it
- 1:04is. This Google sheet has test data that
- 1:07I can use for that. A handful of support
- 1:10questions ready for classification and
- 1:11I've also added in what the correct
- 1:13answers are by hand. By running them
- 1:17through my workflow, I can get a sense
- 1:20of where things are going wrong and get
- 1:22ideas for improvements. So, I've added a
- 1:24trigger here that will read in the rows
- 1:26from the data set and a set outputs node
- 1:29that will write the results back to that
- 1:31data set. Uh, and you can see that it's
- 1:34running multiple times in a row, once
- 1:36for each row in my data set. And we can
- 1:39see the results coming in here. Uh, the
- 1:42red cells are where the AI got it wrong.
- 1:45And I can look at these to get a sense
- 1:46of where I need to improve things. After
- 1:49I've made my changes, I can try again
- 1:51and I can see what's changed. Once my
- 1:54data sets once my data set starts
- 1:57getting bigger, I'll need a way of
- 1:58measuring performance, not just
- 2:00eyeballing it. And that's where metrics
- 2:03come in. Metrics are essentially a score
- 2:06that measures the quality of the
- 2:07workflow. Metrics can be code-based or
- 2:09can use AI to judge performance. So here
- 2:13I've got a very simple metric. I am just
- 2:15comparing the expected priority for my
- 2:17Google sheet with the actual priority
- 2:20that's been output by the workflow and
- 2:23doing the same for the category. But you
- 2:25can also calculate lots of metrics like
- 2:27correctness or document rag relevance
- 2:29with this feature. I can use the
- 2:32evaluations tab to run my evaluation.
- 2:36And once it's finished, I can see a roll
- 2:38up of my metrics over all my inputs as
- 2:42well as results for individual rows. I
- 2:45can also see how those metrics have
- 2:47evolved over time as I have worked on my
- 2:50workflow. So in summary, you should use
- 2:53evaluation when building your workflow
- 2:55to get an idea of how it performs over a
- 2:57handful of different cases, when
- 2:59deploying to production to check that
- 3:01the edge cases you know about are
- 3:03covered, and when making changes to
- 3:05check that you haven't accidentally
- 3:07broken
- 3:08anything. It's a feature we're very
- 3:10excited about, and I hope you find it
- 3:13useful.
About this transcript
This page contains the full transcript of n8n Evaluation quickstart by n8n, generated from the public captions YouTube serves with the video. The transcript has 576 words across 82 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.