8 Hour AI Agents Course in 30 Minutes (Deep Learning AI) — Transcript
Full transcript
- 0:00I took Andrew Ning's agentic AI course
- 0:02for you. So, here's the cliffnotes
- 0:04version to save you the eight hours.
- 0:06I've been really looking forward to this
- 0:07one, but it is not enough for you just
- 0:09listen to me talk about stuff. So, at
- 0:11the end of this video, I have included a
- 0:12little assessment because research shows
- 0:14that immediately reviewing information
- 0:16after you learn it is the best way to
- 0:18retain that information. Now, without
- 0:19further ado, let's go. A portion of this
- 0:22video is sponsored by HubSpot Media.
- 0:26Let's first go over the structure of the
- 0:28course. The Agentic AI course has five
- 0:30modules. Module one, introduction to
- 0:32Agentic Workflows, covers the
- 0:33foundations of Agentic AI and is what
- 0:36the rest of the course is built on top
- 0:37of. Module two is a deep dive into the
- 0:40reflection pattern, a common pattern for
- 0:42agentic workflows. Module three covers
- 0:44tool use, another common agentic
- 0:46pattern. The course also covers lots of
- 0:48examples and ways to improve the
- 0:49results. Module four is called practical
- 0:52tips for building agentic AI. And this
- 0:54module is by far the most important
- 0:56module. I have not seen this information
- 0:59especially on evaluations being covered
- 1:00to this depth anywhere else. It's what
- 1:03everybody talks about but nobody has
- 1:04actually really covered it well and this
- 1:06course does it well. And finally module
- 1:08five is patterns for highly autonomous
- 1:10agents where Andrew covers more
- 1:12experimental agentic workflows a taste
- 1:14for the future.
- 1:17All right let's jump into module one the
- 1:19fundamentals starting off by defining
- 1:21agentic AI. An agentic AI workflow is
- 1:24defined as a process where an LLM based
- 1:26app executes multiple steps to complete
- 1:28a task. For example, if you want your
- 1:30agentic AI workflow to help you write an
- 1:33essay on tea ceremonies.
- 1:36First, the LM will need to come up with
- 1:37an essay outline about tea ceremonies.
- 1:40Then, it will need to think to itself,
- 1:42do I need any web research? It decides
- 1:44that it does, and it will then do
- 1:45research on tea ceremonies across
- 1:47different locations and across different
- 1:49times. Next, it would then write a first
- 1:51draft, but that's not good enough. It
- 1:53would then consider what parts would
- 1:54need more revision and more research.
- 1:57Maybe it would even request a human
- 1:58review at this point if it's really
- 2:00quite unsure about tea ceremonies. Then
- 2:02finally, it would revise the draft and
- 2:04have the final draft ready. This is an
- 2:06example of an angentic AI workflow that
- 2:08is less autonomous since it's quite
- 2:10clear what the steps are to be taken in
- 2:12order to get the final result, but you
- 2:14can also have more autonomous agents,
- 2:16too.
- 2:18Using the same essay writing agentic
- 2:20workflow and a more autonomous workflow
- 2:22would look something like you similarly
- 2:24tell the LM to you know write an essay
- 2:27about tea ceremonies but instead of
- 2:29directing it to go write an outline and
- 2:31then do the web research and then etc
- 2:33etc you can just give the LM a set of
- 2:35tools like web search and news search
- 2:38and maybe archive search for more
- 2:40research papers too and you let the LM
- 2:42itself decide which tool it's going to
- 2:43use to gather the right information find
- 2:45the best sources compile it together
- 2:47then write it the essay draft. It will
- 2:49also have the autonomy of how it's going
- 2:50to reflect on how it is that you can
- 2:52improve things, whether it wants to call
- 2:54a human reviewer or not until the agent
- 2:56itself feels satisfied and produces the
- 2:58final result. It's the same input and
- 3:00the output that you want is the same as
- 3:01well. But you can see that one is very
- 3:02linear and the steps are well very well
- 3:04defined, less autonomous, and you have
- 3:06the much more autonomous version where
- 3:08the agent has access to tools and a lot
- 3:10more freedom to decide how it's going to
- 3:11get to that final essay.
- 3:14There's of course pros and cons to being
- 3:16less autonomous versus more autonomous.
- 3:18Like if you're less autonomous, the
- 3:19steps are predefined. They're clear and
- 3:21most of all, you have a lot more control
- 3:23over what is happening and the final
- 3:25result. On the other hand, if something
- 3:27is more autonomous, the agent gets to
- 3:29make a lot more decisions for itself and
- 3:31you can end up with really creative,
- 3:32really great results that are beyond
- 3:34what you imagined. But the downside is
- 3:35that you would then have less control
- 3:37and it could also potentially come up
- 3:38with results that may not be what you're
- 3:40looking for at all. In any case, with
- 3:42agentic AI workflows, there is this
- 3:44spectrum of autonomy that you, the
- 3:46person building the agentic workflow,
- 3:48gets to choose how autonomous you want
- 3:50it to be. Very explicitly though, Andrew
- 3:52in the course emphasizes that he rejects
- 3:53this notion of like a binary something
- 3:55is either an agent or it's not an agent,
- 3:57which is pretty much like what the
- 3:58people on X has been debating about for
- 4:00like the past 2 years. What is actually
- 4:02considered an agent? But Andrew is
- 4:04basically like, no guys, no guys, still
- 4:05just don't think about it that way,
- 4:06okay? Let's just call things like
- 4:07agentic AI and as long as it involves
- 4:09multiple steps, it's agentic AI and you
- 4:11can just be on the spectrum of least
- 4:13autonomous to more autonomous.
- 4:15Paraphrasing a little bit here. Anyways,
- 4:16what we all can agree on though is if
- 4:18you wrap a LM, a model in an agentic
- 4:21workflow, it is going to be much better
- 4:23than just calling it directly. This is a
- 4:25fact. It would also be often times
- 4:26faster and more modular, meaning that
- 4:28you can change things a lot more easily.
- 4:30So that is why agentic AI is so popular
- 4:32and very much worth building.
- 4:36Okay, let's now cover from a practical
- 4:38perspective. What are the building
- 4:39blocks of Agentic AI? Andrew's building
- 4:42blocks for Agentic AI is pretty
- 4:43simplistic actually. It involves just
- 4:45three components. Let's start off by
- 4:47addressing the first two which is models
- 4:49and tools. Models just refers to the AI
- 4:52model. That can be large language models
- 4:53or it can be like other types of AI
- 4:55models, multimodm models like video
- 4:56generation models, audio generation
- 4:58models, any type of model. And then
- 5:00there are tools which are basically
- 5:01functions and capabilities that you're
- 5:03giving your AI agent so that it's able
- 5:05to perform different types of task.
- 5:07Examples of this would include being
- 5:08able to use external software through
- 5:10their APIs. Stuff like web search,
- 5:13getting real-time data, being able to
- 5:14check emails, things like that.
- 5:16Information retrieval, so being able to
- 5:18access certain types of databases, and
- 5:20code execution, being able to write code
- 5:22to be able to do things like math, data
- 5:24analysis, building stuff. So all of
- 5:26these are tools. There is also a third
- 5:28component which I will talk about in a
- 5:30little bit because when you are first
- 5:32thinking about creating an agentic
- 5:33system, you're really just thinking
- 5:34about how do you take these building
- 5:36blocks of different models and different
- 5:37tools and chain them together and put
- 5:39them together in such a way to achieve
- 5:41the results that you want. For example,
- 5:44say you want to develop an agentic AI
- 5:46system with the goal to respond to a
- 5:48customer email for your company. This is
- 5:50an example of the type of email that you
- 5:52may get. Subject line here is wrong item
- 5:54ship. It's like, "Hi, I ordered a Blue
- 5:55Kitchen Pro blender. Here's the order
- 5:57number, but received a red toaster
- 5:59instead. I need a blender for my
- 6:00daughter's birthday party this weekend.
- 6:02Can you help?" Susan Jones. If you want
- 6:04to create an AI workflow that's able to
- 6:06accomplish this task, the easiest way to
- 6:07do this is actually think to yourself,
- 6:10how would I go about accomplishing this
- 6:11task? Like, how would a human do this?
- 6:13Okay, so the first step would be you
- 6:15need to extract key information,
- 6:16including the order number, the fact
- 6:18that it was supposed to be a Blue
- 6:19Kitchen Pro blender, but they got a red
- 6:21toaster instead, and the timeline. The
- 6:23daughter's birthday party is for the
- 6:25weekend. Then you would go and find the
- 6:27relevant customer record, which would
- 6:29likely include looking into an orders
- 6:31database to see what was actually
- 6:33recorded. You would then determine what
- 6:35possibly could have went wrong and then
- 6:36send a email reply back. Now, we need to
- 6:39translate these three steps into what an
- 6:41LM would do. Well, for the first one,
- 6:43extracting key information, the LLM is
- 6:45able to extract that information just by
- 6:47itself. But for the next step, which is
- 6:48finding a relevant customer record, the
- 6:50LM would actually need a tool that would
- 6:52allow it to look into the orders
- 6:54database. So, it would need to provide
- 6:56that with that tools for orders database
- 6:58query. And then finally, for writing and
- 7:00sending the response, the LM would be
- 7:02able to draft the response, but in order
- 7:04to actually send that email, it would
- 7:05need to have access to another tool that
- 7:07allows it to send the email. Make sense?
- 7:09This is how the building blocks are
- 7:11coming together here. You have the large
- 7:13language models and the different tools
- 7:15that it requires. Of course, the type of
- 7:16large language model you want to use on
- 7:18each of these steps could vary depending
- 7:20on what it's supposed to be doing
- 7:21exactly and how it is that you're
- 7:23actually going to build this type of
- 7:24tool also varies as well. But don't
- 7:26worry too much about this. We're going
- 7:27to go into a lot more detail later in
- 7:28the video. But here what I want to just
- 7:30make really clear is that you're
- 7:31basically taking um a goal and then
- 7:34translating that into a sequence of
- 7:36steps in which you will then use the
- 7:37building blocks of models and tools in
- 7:39order to accomplish those steps. So the
- 7:42ideal situation after your AI system
- 7:44goes through these steps is that it
- 7:46would come up with a wonderful email
- 7:47that says something like, "Thank you so
- 7:49much for emailing the company. After
- 7:51looking at your order number, we did
- 7:53notice that it does correspond to a blue
- 7:56blender. And we apologize that you
- 7:58received a red toaster instead. We will
- 8:00send you a blender with expedited
- 8:02shipping so that you can get it tomorrow
- 8:04and make it for your daughter's birthday
- 8:05on the weekend. And you can keep the red
- 8:07toaster as well. Then the customer,
- 8:09Susan Jones, will probably be very
- 8:11happy. But what happens if your agent
- 8:13doesn't give you this wonderful email
- 8:14and instead write something like, "Oh,
- 8:16thank you for shopping with us. As you
- 8:18can see, we are so much better than
- 8:19Rival Corp, our competitor, or something
- 8:21like sure, we'll issue a refund and not
- 8:23actually address the issue. That is why
- 8:25you need the third component of Andrew's
- 8:27framework, which is evaluations.
- 8:30Evaluations, as his name suggests, is
- 8:32all about evaluating the results of your
- 8:35agent. It's like the other half of
- 8:37building an AI agent. Not good enough to
- 8:39just build an agent. You actually got to
- 8:40make sure it works properly, right? And
- 8:42I am so glad that this course covers so
- 8:44much in depth about evals. I will be
- 8:46covering this in a lot more detail in
- 8:47module four, but very briefly, an
- 8:49evaluation in this case is making sure
- 8:51that your AI agent doesn't actually talk
- 8:54about your competitors because that's
- 8:56kind of like in poor taste. So, how do
- 8:58we do this? Well, first we can compile a
- 9:00list of all the competitors that we
- 9:01don't want to be mentioned like comp
- 9:03code and rival co or the other code.
- 9:05Then you can actually write a small
- 9:06snippet of code like this. If competitor
- 9:09is in response, then the num competitor
- 9:12mentions is plus or equal to one. What
- 9:14this basically does is check for the
- 9:15email responses and if any of these
- 9:17competitors are mentioned in the
- 9:18response then you would just increment
- 9:20the counter of number of competitor
- 9:22mentions to one more. Of course, ideally
- 9:25you want this counter the number of
- 9:26competitor mentions to be equal to zero.
- 9:29But you know if it does show up you
- 9:31would have a way now of tracking how
- 9:33many times it shows up and where it is
- 9:35showing up. This is a very objective way
- 9:37to determine if these competitors are
- 9:39actually still showing up. And you can
- 9:41use this counter as a way to then tweak
- 9:43your prompt, tweak your system, and do
- 9:45stuff to it to try to lower this number
- 9:47so that it eventually goes to zero.
- 9:49Don't worry about it if any of the
- 9:50detail seems a little bit hazy. I'll be
- 9:52going into a lot more detail later in
- 9:53the video. But what I just want you to
- 9:55understand here is that see by having
- 9:57like an evaluation like this where you
- 9:59can measure the occurrence of this bad
- 10:02behavior, you now have an objective way
- 10:04to improve your system to decrease the
- 10:07bad behavior. If you didn't have this
- 10:08EBAL, this way of measuring the bad
- 10:10behavior, then you can like kind of
- 10:12think that you are improving your
- 10:14system, but you don't actually
- 10:15objectively know if you're improving the
- 10:17system or not. That is why evaluations
- 10:19are so important. This is an example of
- 10:21a very objective codebased approach for
- 10:24evaluating a result. There's another one
- 10:26called LM as a judge in which you're
- 10:28actually using another large language
- 10:30model to rate the results and evaluate
- 10:32the results of your AI agent. Anyways,
- 10:34more detail in module 4. This course is
- 10:36an excellent introduction to Agentic AI,
- 10:38but when you're ready to start building
- 10:39your own AI agents with business impact,
- 10:41there are many other factors that you do
- 10:43need to consider, which is why I highly
- 10:45recommend that you check out the free
- 10:47master AI agents resource from HubSpot
- 10:49Media. It provides good practical
- 10:50frameworks and examples on how to get
- 10:52started with building useful AI agents
- 10:54with business impact. It also includes
- 10:56real use cases in marketing, sales, and
- 10:58operations that you can directly use to
- 10:59start implementing your own agents to be
- 11:01able to do things like content
- 11:02production optimization, process large
- 11:04amounts of data, do customized outreach,
- 11:06and much, much more. There's also a
- 11:08second download and resource called how
- 11:09to use AI agents that has an excellent
- 11:11checklist to guide your organization
- 11:13through each phase of implementing AI
- 11:14agents. This is so useful and it's
- 11:16something that we actually use in our
- 11:18company whenever we're launching a new
- 11:19agent. Really recommend you check out
- 11:20the resource linked over here also in
- 11:22description. Thank you so much Hopsoft
- 11:24Media for providing practical free
- 11:25resources on building AI agents and for
- 11:28sponsoring this portion of the video.
- 11:29Now back to the video. But first, let's
- 11:31move on to modules two and three which
- 11:33are covering what are known as agentic
- 11:34design patterns. And what are agentic
- 11:37design patterns you may ask? Well,
- 11:39remember we talked about earlier about
- 11:40sending that email for the blender and a
- 11:42toaster. Got to first decompose that
- 11:44task and then translate that into a
- 11:47combination of large language model and
- 11:49tool use. So you could just kind of like
- 11:51figure that out yourself with any type
- 11:54of task that you want your agentic AI to
- 11:56do. But luckily, we do have a lot of
- 11:59precedence now for specific types of
- 12:01task and specific types of design
- 12:03patterns like how it is that we're
- 12:04decomposing it and fitting together the
- 12:06pieces that has proven to shown really
- 12:08good results because more likely than
- 12:09not, whatever it is that you're trying
- 12:10to accomplish will converge to one of
- 12:12these design patterns. In this course,
- 12:14Andrew covers four design patterns. two
- 12:16that are less autonomous and two that
- 12:18are more experimental, more autonomous.
- 12:21Let's now talk about the first design
- 12:23pattern, which is reflection. The
- 12:25reflection design pattern is super
- 12:26intuitive. For example, if you tell a
- 12:28human, hey, go write this email, and
- 12:29they're like, you know, distracted or
- 12:31something and maybe they just like had
- 12:32lunch, so they just write some sloppy
- 12:34email. But if you go like, hey, reflect
- 12:36on this and improve the email, probably
- 12:37the result you're going to get is going
- 12:39to be a lot better. Exactly the same
- 12:40with AI. If you tell an AI to write an
- 12:42email, it will first do a first draft,
- 12:44which might not be that great. But if
- 12:46you tell it to go back and reflect on it
- 12:48and improve it, you will end up having
- 12:49much better results. The second step,
- 12:51asking to reflect on it and come up with
- 12:53another draft, is literally the
- 12:55reflection design pattern. Simple but
- 12:57powerful. And remember how it is that
- 12:58way you can actually go about tweaking
- 12:59this and improving the systematically is
- 13:01by having the whole like evaluation
- 13:02section. The course also provides like
- 13:04code examples of how to implement design
- 13:06patterns like the reflection pattern,
- 13:08which you can see over here, but I'm not
- 13:09going to go into too much detail about
- 13:10this. You can actually go and try out
- 13:12the implementation with code if you want
- 13:13to in the course. It's actually free to
- 13:16get this code. So, I'll link it in the
- 13:17description. By the way, don't be
- 13:18discouraged and like rage quit right now
- 13:20if you don't know how to code. Um, this
- 13:21is absolutely doable with no code as
- 13:22well. They just don't show it in the
- 13:24course. Like, we teach the full no code
- 13:25approach for doing this in our agents
- 13:27boot camp ourselves. I also do have a
- 13:28video over here that covers the basics
- 13:30of how to implement um agents using no
- 13:32code approach too.
- 13:35Okay, let's go on to the second design
- 13:36pattern called tool use. So, this part I
- 13:38did get like a little confused because
- 13:40we did talk about tools uh when we
- 13:42talked about the building blocks, right,
- 13:43of like models and tools and and
- 13:45evaluations, but I think Andrew included
- 13:47this as like a full-on design pattern
- 13:49because he wanted to emphasize like how
- 13:50much tool use can elevate the
- 13:52performance of an agent. So, we'll go
- 13:54through it. Firstly, to define tool use,
- 13:56you're letting your agent have access to
- 13:58tools that's able to implement and do
- 14:00certain things that it normally cannot
- 14:02do just as a model itself. For example,
- 14:04if you have an agentic AI workflow in
- 14:06which you want to be like booking
- 14:08meetings and stuff, you would need to
- 14:09give your agent access to a calendar
- 14:11booking tool. What tool use? You can
- 14:12make it like very strict telling your
- 14:14agent that you have to use this exact
- 14:16tool to do this exact thing or you can
- 14:18make it more autonomous and give your
- 14:20agents more like a selection of tools
- 14:22that it can choose from and they can get
- 14:23to choose which tool it wants to use to
- 14:25accomplish a task. But whatever tools it
- 14:27is that you choose to give your AI agent
- 14:29and how much you restrict how it's using
- 14:30the tools. The key here is that you do
- 14:32need to define the tools and you need to
- 14:34let your agent know that you gave them
- 14:36access to these tools and what the tools
- 14:38are in the system itself. Or else your
- 14:40agent would just be like what is this? I
- 14:42don't know what this is. Like for
- 14:44example, if you have a personal
- 14:45assistant agentic AI system and you tell
- 14:48it to find a free slot on Thursday in my
- 14:50calendar and make an appointment with
- 14:51Alice, you would need to give it tools
- 14:53such as make appointment, check calendar
- 14:55and delete appointment and tell it in
- 14:57the system prompt you have access to the
- 14:59tools make appointment which is for
- 15:01making an appointment booking. Check
- 15:03calendar which is for checking a
- 15:05calendar to check for availabilities.
- 15:06Delete appointment in order to delete
- 15:08existing appointments. then you can tell
- 15:10your agent to like you know find that
- 15:12time slot between you and Alice and it
- 15:14should be able to use the tools at his
- 15:16disposal in order to accomplish the
- 15:17task.
- 15:20So how it is that you can actually
- 15:21define these tools for your agents.
- 15:23There's actually quite a few different
- 15:24ways of doing this. The most simple
- 15:26approach if you're using code is to
- 15:27literally just like write a function for
- 15:29it and then give it access to your AI
- 15:31agent through some type of framework. Um
- 15:33the example that they showed in the
- 15:34course is called the AI suite, but
- 15:36there's like a lot of other frameworks
- 15:37that you can use as well. Not going to
- 15:39go into too much detail about this, but
- 15:41if you are interested in implementing
- 15:42this, there is an assignment in the
- 15:44course as well, which I will link in the
- 15:46description. So yeah, keep that in mind.
- 15:48Angie also explains that there's other
- 15:50ways for you to give your agent tools as
- 15:51well. For example, through something
- 15:53called MCP, which is the model context
- 15:55protocol. This is a protocol that was
- 15:57developed by Anthropic that standardizes
- 15:59the way that tools are being defined so
- 16:02that it's very easy for agents to have
- 16:04access to different tools without having
- 16:05to like write custom functions for all
- 16:07of them. MCPS are actually really cool.
- 16:08A lot of developers are actually
- 16:09building MCPs and like selling their
- 16:11MCPs and stuff like that as well. So, if
- 16:13you are interested in learning more
- 16:14about it, I do have a full video which
- 16:15I'll link over here that goes into the
- 16:17depths of MCP. But for now, let's move
- 16:20on to the much anticipated module 4,
- 16:26practical tips for building agentic AI,
- 16:28where we talk a lot now about
- 16:30evaluations, the missing piece of
- 16:32agentic AI development and what makes
- 16:34this course so valuable. To properly
- 16:36define it, evaluations or evals are
- 16:38agent evaluation mechanisms that refer
- 16:41to processes and criteria used to assess
- 16:43the performance of agents within a
- 16:44system. We already talked about why we
- 16:46need evals and why it is the third major
- 16:48building block because you can build an
- 16:49AI agent, but how do you even know how
- 16:51good it is and how do you know how to
- 16:52improve it if you don't measure the
- 16:54results of it? And that is where eval
- 16:56come in. For example, say you have an
- 16:57invoice extraction agent. So you give
- 16:59the system PDF invoices which then it
- 17:01converts to text. Then the LM extracts
- 17:04four required fields like the biller,
- 17:05biller address, amount, due, and due
- 17:07date. It has access to a tool called
- 17:10update database which will then use in
- 17:12order to update the database with a new
- 17:14invoice and the record is created.
- 17:16Simple linear agentic workflow here.
- 17:18Now, if we want to evaluate this agentic
- 17:20system, one approach of doing this is to
- 17:22just now take 10 to 20 different
- 17:24invoices and then feed it through the
- 17:26agent and then make a record of the
- 17:28results. You might notice that invoice
- 17:30one was fine. um it was able to extract
- 17:32the four required fields and everything
- 17:34is great but you found that invoice 2 it
- 17:36actually mixed up the dates between the
- 17:38due date and the invoice date. Uh
- 17:40invoice 3 was okay as well and then you
- 17:41keep on going and you find that invoice
- 17:4319 it also makes up the dates and
- 17:44invoice 20 it makes up the dates as
- 17:46well. This is actually the most
- 17:47practical way of doing this. So from
- 17:49collecting this data you notice that one
- 17:51of the tendencies of this agentic
- 17:53workflow is that it mixes up dates which
- 17:55is not good. We actually want to improve
- 17:57that. So we want to build an eval for
- 17:59this to set as a metric for us to
- 18:01improve. One thing to note here is that
- 18:02often times you don't actually know what
- 18:04eval that you should be building before
- 18:06you actually build out your system and
- 18:08start running examples through it. It's
- 18:10actually kind of hard to think through
- 18:11like what are the potential like places
- 18:13of failure for your system until you try
- 18:15it out. So that's why I do like general
- 18:17recommendation is just like build
- 18:18something quickly, quick and dirty and
- 18:20then just like start passing through
- 18:22different examples and collecting that
- 18:23data to decide what the eval should be.
- 18:26Say we want to create an eval now for
- 18:28date extraction because we notice that
- 18:30it gets a date extraction wrong a lot.
- 18:32How do we go about doing this? Well,
- 18:33step number one is to manually extract
- 18:35the due dates from the 10 to 20 invoices
- 18:37that you pass through. Then you want to
- 18:39specify the output format of the data in
- 18:41the prompt itself. So then you can
- 18:43extract the date from the LM response
- 18:45using code and finally compare the large
- 18:48language model results to the ground
- 18:50truth. Does the extracted data equal to
- 18:53the actual data? This is an example of
- 18:55evaluation that uses code if you
- 18:57actually want to implement evaluation
- 18:58using code yourself. By the way, also in
- 19:00the course there is the code for this
- 19:01that you can run yourself.
- 19:05Anyways, this is an example of a ebal
- 19:06that you can implement so that in the
- 19:08future as you're like tweaking your
- 19:09prompt and then changing things around,
- 19:11you can actually measure if your invoice
- 19:13agent system is becoming more accurate
- 19:16or less accurate at date extraction.
- 19:18Hopefully more accurate. There are a lot
- 19:20of other types of doing it. And the way
- 19:21that the course frames it is that you
- 19:23can think about evals as falling under
- 19:25two different axes. This date extraction
- 19:27eval is an example of an objective eval
- 19:30because the result of it is that the
- 19:32extracted date is either equal to the
- 19:34actual date or it's not equal to the
- 19:36actual date. Very objective either right
- 19:38or wrong. With this type of eval system,
- 19:40you can usually implement it pretty
- 19:42easily just through code itself. Another
- 19:44type of eval is a subjective eval. This
- 19:46is when the result isn't just like a yes
- 19:48or no binary kind of situation. An
- 19:50example of a subject eval is if you're
- 19:52trying to say assess the results of a
- 19:54research agent. The agent is supposed to
- 19:56research on a certain topic and then
- 19:58output like some sort of essay at the
- 19:59end. Say if you give it a prompt like
- 20:01research and write an essay about recent
- 20:03black hole science. There's no way to
- 20:05objectively evaluate how good the
- 20:07research and how good the essay ends up.
- 20:09You can't just like, you know, count the
- 20:11number of words or something like that,
- 20:12right? So, what you need to do to create
- 20:14this kind of email is first choose a few
- 20:16different topics that you're going to be
- 20:18using as the prompt. Like maybe
- 20:20something about buck coal, something
- 20:21about robot harvesting, renting versus
- 20:23buying a home in Seattle, etc., etc. So,
- 20:25for each of these prompts, you need to
- 20:27actually manually establish a ground
- 20:28truth annotation, which are like golden
- 20:31standard talking points that you know
- 20:33needs to be researched and needs to be
- 20:35part of the final essay as well. For
- 20:37example, with black holes, these could
- 20:38be stuff like event horizon or radio
- 20:40telescope. for robotic harvesting. It
- 20:42could be like robo pick and pinchers. To
- 20:44actually do this, you probably need to
- 20:45like find someone who actually knows
- 20:46these topics or you yourself know these
- 20:48topics as well. So, this part is manual.
- 20:50But then the second part is when you can
- 20:52use the LM as a judge. So, you have
- 20:53another type of LM to count how many of
- 20:56these topics actually appear in the
- 20:58research and the final results. The
- 21:00reason why you need to use an LLM to to
- 21:02count this and not just like um
- 21:04objectively use code to count this is
- 21:06because a certain term can appear in a
- 21:08lot of different iterations. Like the
- 21:09term event horizon could be like event
- 21:11horizon, but it could also be like
- 21:12uppercase, lower case, could be the
- 21:13horizon on the type of event at the
- 21:16horizon or like when the horizon is near
- 21:19something like that, right? There's like
- 21:20so many different types of iterations of
- 21:21this that you need an LLM to be able to
- 21:24capture all the different types of these
- 21:25iterations. Then you can get the LM to
- 21:27give you a final score for each prompt
- 21:29based upon how many of these terms were
- 21:31captured in the final result. This on
- 21:33screen now is an example of a prompt for
- 21:35the LM as a judge. That's the first axis
- 21:38of this 2x2 matrix uh for the different
- 21:40types of evals. And this one is
- 21:42objective versus subjective. The second
- 21:44axis is whether something has a perample
- 21:46ground truth or does not have a perample
- 21:49ground truth. Both the examples that we
- 21:50just talked about, the invoice example
- 21:52and the LM research and essay writing
- 21:55example are agenting systems that have a
- 21:57perample ground truth. This means that
- 21:59for every example like every input that
- 22:01we give there is a established ground
- 22:04truth for that example. like the invoice
- 22:06extraction has a specific date that is
- 22:08the correct result that is different for
- 22:10all the different examples and the
- 22:11topics for the research agent are
- 22:13different for each of the examples too.
- 22:15However, you can also have emails that
- 22:16have no perample ground truth meaning
- 22:19that there's a singular universal
- 22:20standard that is there for every single
- 22:23example. Say if you have an agentic
- 22:25system that is for marketing copy length
- 22:27um and you just want to make sure that
- 22:28the marketing copy length is always
- 22:30going to be less than 10 words. So like
- 22:32the number 10 is like the universal
- 22:33magical number. This is also an
- 22:35objective eval. So you can actually
- 22:36write code. So you can literally just
- 22:38write if the length of the text is less
- 22:40than or equal to 10, then that is good.
- 22:42If it's not, then it's bad. The final
- 22:43quadrant of this 2x2 matrix is when you
- 22:46have something that is subjective and
- 22:48does not have a per example ground
- 22:50truth. Say you have an agentic system
- 22:51that is supposed to create different
- 22:53graphs. The judging of the graph is
- 22:55going to be subjective. The result of
- 22:57the graph is going to be subjective. And
- 22:59the way that you judge it is going to be
- 23:00based off this universal criteria where
- 23:02you would ask the LM as a judge to grade
- 23:05the graph to see whether it has axis
- 23:08labels. Are the colors easy to see? Is
- 23:10there any overlapping text? Etc., etc.,
- 23:12etc. So, that would be like a universal
- 23:14grading criteria.
- 23:17Okay, great. That covers the four
- 23:19general types of of evals that you can
- 23:21make. Some of the tips that the course
- 23:23suggests when you're creating these
- 23:24evals, uh firstly is to just quick and
- 23:27dirty is okay to start. just create
- 23:29something and then just create some
- 23:30evals and just get started like that.
- 23:32You you can always continue to expand
- 23:33the set of evals. You want to try to use
- 23:35a lot of different types of examples to
- 23:37try to capture all the different types
- 23:39of errors that you can change into
- 23:41evals. And finally, you want to look for
- 23:43places where the performance of the
- 23:45agentic system is worse than a humans.
- 23:47These are the best areas for
- 23:48improvement. Great. That was the module
- 23:50on evals. Even if you implement just
- 23:52like a few of these different evals,
- 23:55you'll be setting up a system that will
- 23:56help you dramatically improve the
- 23:58results of your agents. I feel like
- 24:00evals are such an open secret and by
- 24:02implementing them, massive return on
- 24:03investment.
- 24:05All right, we're almost done. We are at
- 24:07module five, which is the patterns for
- 24:09highly autonomous agents. We're going to
- 24:11be covering two of the more experimental
- 24:12agents that are also more autonomous.
- 24:14The two patterns that we covered
- 24:15earlier, reflection and tool use, are
- 24:18less autonomous and they're more
- 24:19predictable. While for these
- 24:20experimental ones, planning and multi-
- 24:22aent systems, these are less
- 24:24predictable, less controllable, but if
- 24:26you get it right, the results can be
- 24:28really, really good. Let's start off
- 24:29with planning and jump straight to an
- 24:31example. Say you have a customer service
- 24:33agent in which it has an inventory
- 24:35database of different types of products
- 24:37in a store. You have like ID, name,
- 24:39description, price, and stock. An
- 24:41example of a customer query that the
- 24:42agent would receive would be like, do
- 24:44you have any round sunglasses in stock
- 24:46that are under $100? So you see this is
- 24:48actually a pretty difficult query
- 24:50because it involves the customer survey
- 24:52agent to have to go through the data set
- 24:54and first like find what are the round
- 24:56styles. Then they has to go pick out
- 24:58which of these are in stock and finally
- 25:00which of these prices are going to be
- 25:01under $100 to be able to come up with
- 25:03the final answer of yes we have our
- 25:05classic sunglasses which are classic
- 25:06round metal frame and cost $60. And of
- 25:08course this is just a single example of
- 25:10the kind of questions that customers
- 25:12will be asking. There's a broad range of
- 25:14questions that will be asked and that
- 25:15can be answered by using this database.
- 25:17However, there is no like set path for
- 25:20you to actually answer these questions.
- 25:22Each time you would have to figure out
- 25:23and plan out what is the way to actually
- 25:26answer these questions. This is where
- 25:27the planning design pattern comes in.
- 25:29Going back to that example of asking a
- 25:31question, do you have any round
- 25:32sunglasses in stock that are under $100?
- 25:34The planning design pattern will first
- 25:35have an LLM figure out what is the
- 25:38actual plan to answer this question
- 25:40given the set of tools that it has. Like
- 25:42it has tools like get item descriptions,
- 25:44check inventory, process item return etc
- 25:47etc. So it would use this information
- 25:49and the information of the database and
- 25:50come up with a plan like number one have
- 25:52to use get item description tool to find
- 25:54the round sunglasses. Second step is to
- 25:56use check inventory to see if results
- 25:58are in stock. Then third step is to use
- 26:00get item price to see if in stock
- 26:01results are less than $100. Then the
- 26:03agent will take that plan and execute
- 26:05each of these steps to get the final
- 26:06result. By the way, if you want to dig
- 26:08into this and actually implement these
- 26:09evals yourself, there is again
- 26:11assignments that you can try out in the
- 26:13course which I will link below. What is
- 26:15cool about this planning design pattern
- 26:16is that the plan is not predetermined.
- 26:19So the agenda system has to actually
- 26:20figure out the plan first and then go
- 26:22and execute all of it. Which means of
- 26:24course that the downside of this
- 26:25planning design pattern is that you
- 26:27don't actually know what the plan is. So
- 26:29it's actually very hard to troubleshoot
- 26:31and anticipate what kind of result
- 26:32you're going to get until it actually
- 26:33goes through the process itself. The
- 26:35upside of this is this increased amount
- 26:36of flexibility and sometimes like it is
- 26:38pretty magical to witness an agent be
- 26:41able to plan out super complex tasks and
- 26:43then execute it and get like crazy good
- 26:45results. This is the kind of magic that
- 26:47more autonomous systems have. Less
- 26:49control but more probability of really
- 26:51cool results.
- 26:54The other more experimental autonomous
- 26:56design pattern is a multi-agentic
- 26:58workflow. The idea of a multi- aent
- 27:00system is to take an agentic system and
- 27:03actually have it like be multiple
- 27:04different agents working together to
- 27:06produce the final result. The intuition
- 27:08for why this can be better is kind of
- 27:09like a company, right? If you have like
- 27:11one person trying to do all the
- 27:13different tasks in a company, um it's
- 27:15probably not going to do the best job
- 27:16and it's going to get overwhelmed
- 27:17because it has to do like so many
- 27:18different things and juggle so many
- 27:19different things at the same time. But
- 27:21if you have different people in that
- 27:23company that specialize in different
- 27:24things, each person would be able to
- 27:26focus better on their specific task. and
- 27:27when you put it together, it's going to
- 27:29produce a better result. That's why we
- 27:30usually have multiple people in a
- 27:32company and not just like the CEO trying
- 27:33to do everything themselves. So that's
- 27:34the same intuition for developing
- 27:36multi-agentic systems. For example, say
- 27:38you have a marketing team agentic system
- 27:40and instead of just having one agent,
- 27:42you actually divide that into three
- 27:44different sub aents. A researcher, a
- 27:46graphic designer, and a writer. So that
- 27:49when you receive a query like create a
- 27:50summer marketing campaign for
- 27:52sunglasses, your research agent will
- 27:54focus on conducting research and figure
- 27:56out that here are current sunglasses
- 27:57trends and competitor offerings. It
- 27:59would pass along that information to a
- 28:01graphic designer agent which would then
- 28:03come up with like five different data
- 28:05visualizations and five different
- 28:06artwork options for the report. It would
- 28:09then pass all of this to the final
- 28:10writer agent which will then compile
- 28:13everything together and write the final
- 28:14report. So this would be a marketing
- 28:16team agentic system that's able to
- 28:18generate report for a marketing campaign
- 28:20on a specific topic. There's actually
- 28:22quite a lot of research done on multi-
- 28:23aent systems and in our boot camp as
- 28:25well like we have a whole week that
- 28:26covers multi- aent systems because the
- 28:28results can be really really good like
- 28:30much much better uh than if you just use
- 28:32a single agent. This course doesn't
- 28:33cover too much into it though but I will
- 28:35leave a link in the description for an
- 28:36anthropic article that goes into a lot
- 28:38more detail about multi-agent systems. I
- 28:40also have a video that covers multi-
- 28:41aent systems as well which you can check
- 28:43out at this link over here if you prefer
- 28:45like a video approach. Anyways, for this
- 28:47course they also have codebased
- 28:48assignments where you can implement
- 28:50these more autonomous systems too. Oh my
- 28:52god. Okay, we have come to the end of
- 28:54this video of this crash course. I
- 28:56literally had let's see 93 pages of
- 28:59notes which I have condensed for you
- 29:01guys here. I overall I think this course
- 29:03really was quite good. It has a very
- 29:05good overview of all of the different
- 29:07parts um for building agentic AI. My
- 29:10critique though, if I may, Andrew does
- 29:12approach it from a very like researchy
- 29:14theoretical perspective, which makes
- 29:15sense given his background as like a
- 29:16research scientist, but I do feel like
- 29:19he didn't really cover like the actual
- 29:21process of deployment and what does that
- 29:23actually look like when you put it into
- 29:25the world and get users to start using
- 29:26your agents and stuff like that. That's
- 29:27like a whole other beast which we cover
- 29:29a lot in our boot camps because we're
- 29:30coming from like a more practical
- 29:32approach so that we're building these
- 29:33agents and actually deploying them and
- 29:35people are using them and you know
- 29:36you're hopefully making money from them
- 29:37as well. Another critique that I do have
- 29:39um is that the assignments here they're
- 29:41all like based upon code. If you didn't
- 29:43know any better you would have the
- 29:44misconception that you have to know how
- 29:46to code in order to implement these
- 29:47systems and that is not true. So I just
- 29:49want to like make that super clear for
- 29:51you guys too. You can do this with no
- 29:52code as well. I'll link in description
- 29:54some of the videos that I cover where
- 29:56you can implement these systems with no
- 29:57code completely.
- 29:59All right, as promised, here is the
- 30:01little assessment in which if you can
- 30:03answer these questions, then
- 30:04congratulations. You are going to be
- 30:06able to retain all the information we've
- 30:07just covered from this 8hour course.
- 30:09Please write your answers in the
- 30:11comments. All right, thank you so much
- 30:12for watching until the end of this
- 30:13video. I really hope that you learned a
- 30:15lot about Agentic AI and you're super
- 30:16excited to start building your own
- 30:18agents as well. And I will see you guys
- 30:20in next video or live stream.
About this transcript
This page contains the full transcript of 8 Hour AI Agents Course in 30 Minutes (Deep Learning AI) by Tina Huang, generated from the public captions YouTube serves with the video. The transcript has 6,840 words across 980 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.