YouTube2Text

8 Hour AI Agents Course in 30 Minutes (Deep Learning AI) — Transcript

by Tina Huang · 6,840 words · 980 segments · language en · Watch on YouTube

Full transcript

  1. 0:00I took Andrew Ning's agentic AI course
  2. 0:02for you. So, here's the cliffnotes
  3. 0:04version to save you the eight hours.
  4. 0:06I've been really looking forward to this
  5. 0:07one, but it is not enough for you just
  6. 0:09listen to me talk about stuff. So, at
  7. 0:11the end of this video, I have included a
  8. 0:12little assessment because research shows
  9. 0:14that immediately reviewing information
  10. 0:16after you learn it is the best way to
  11. 0:18retain that information. Now, without
  12. 0:19further ado, let's go. A portion of this
  13. 0:22video is sponsored by HubSpot Media.
  14. 0:26Let's first go over the structure of the
  15. 0:28course. The Agentic AI course has five
  16. 0:30modules. Module one, introduction to
  17. 0:32Agentic Workflows, covers the
  18. 0:33foundations of Agentic AI and is what
  19. 0:36the rest of the course is built on top
  20. 0:37of. Module two is a deep dive into the
  21. 0:40reflection pattern, a common pattern for
  22. 0:42agentic workflows. Module three covers
  23. 0:44tool use, another common agentic
  24. 0:46pattern. The course also covers lots of
  25. 0:48examples and ways to improve the
  26. 0:49results. Module four is called practical
  27. 0:52tips for building agentic AI. And this
  28. 0:54module is by far the most important
  29. 0:56module. I have not seen this information
  30. 0:59especially on evaluations being covered
  31. 1:00to this depth anywhere else. It's what
  32. 1:03everybody talks about but nobody has
  33. 1:04actually really covered it well and this
  34. 1:06course does it well. And finally module
  35. 1:08five is patterns for highly autonomous
  36. 1:10agents where Andrew covers more
  37. 1:12experimental agentic workflows a taste
  38. 1:14for the future.
  39. 1:17All right let's jump into module one the
  40. 1:19fundamentals starting off by defining
  41. 1:21agentic AI. An agentic AI workflow is
  42. 1:24defined as a process where an LLM based
  43. 1:26app executes multiple steps to complete
  44. 1:28a task. For example, if you want your
  45. 1:30agentic AI workflow to help you write an
  46. 1:33essay on tea ceremonies.
  47. 1:36First, the LM will need to come up with
  48. 1:37an essay outline about tea ceremonies.
  49. 1:40Then, it will need to think to itself,
  50. 1:42do I need any web research? It decides
  51. 1:44that it does, and it will then do
  52. 1:45research on tea ceremonies across
  53. 1:47different locations and across different
  54. 1:49times. Next, it would then write a first
  55. 1:51draft, but that's not good enough. It
  56. 1:53would then consider what parts would
  57. 1:54need more revision and more research.
  58. 1:57Maybe it would even request a human
  59. 1:58review at this point if it's really
  60. 2:00quite unsure about tea ceremonies. Then
  61. 2:02finally, it would revise the draft and
  62. 2:04have the final draft ready. This is an
  63. 2:06example of an angentic AI workflow that
  64. 2:08is less autonomous since it's quite
  65. 2:10clear what the steps are to be taken in
  66. 2:12order to get the final result, but you
  67. 2:14can also have more autonomous agents,
  68. 2:16too.
  69. 2:18Using the same essay writing agentic
  70. 2:20workflow and a more autonomous workflow
  71. 2:22would look something like you similarly
  72. 2:24tell the LM to you know write an essay
  73. 2:27about tea ceremonies but instead of
  74. 2:29directing it to go write an outline and
  75. 2:31then do the web research and then etc
  76. 2:33etc you can just give the LM a set of
  77. 2:35tools like web search and news search
  78. 2:38and maybe archive search for more
  79. 2:40research papers too and you let the LM
  80. 2:42itself decide which tool it's going to
  81. 2:43use to gather the right information find
  82. 2:45the best sources compile it together
  83. 2:47then write it the essay draft. It will
  84. 2:49also have the autonomy of how it's going
  85. 2:50to reflect on how it is that you can
  86. 2:52improve things, whether it wants to call
  87. 2:54a human reviewer or not until the agent
  88. 2:56itself feels satisfied and produces the
  89. 2:58final result. It's the same input and
  90. 3:00the output that you want is the same as
  91. 3:01well. But you can see that one is very
  92. 3:02linear and the steps are well very well
  93. 3:04defined, less autonomous, and you have
  94. 3:06the much more autonomous version where
  95. 3:08the agent has access to tools and a lot
  96. 3:10more freedom to decide how it's going to
  97. 3:11get to that final essay.
  98. 3:14There's of course pros and cons to being
  99. 3:16less autonomous versus more autonomous.
  100. 3:18Like if you're less autonomous, the
  101. 3:19steps are predefined. They're clear and
  102. 3:21most of all, you have a lot more control
  103. 3:23over what is happening and the final
  104. 3:25result. On the other hand, if something
  105. 3:27is more autonomous, the agent gets to
  106. 3:29make a lot more decisions for itself and
  107. 3:31you can end up with really creative,
  108. 3:32really great results that are beyond
  109. 3:34what you imagined. But the downside is
  110. 3:35that you would then have less control
  111. 3:37and it could also potentially come up
  112. 3:38with results that may not be what you're
  113. 3:40looking for at all. In any case, with
  114. 3:42agentic AI workflows, there is this
  115. 3:44spectrum of autonomy that you, the
  116. 3:46person building the agentic workflow,
  117. 3:48gets to choose how autonomous you want
  118. 3:50it to be. Very explicitly though, Andrew
  119. 3:52in the course emphasizes that he rejects
  120. 3:53this notion of like a binary something
  121. 3:55is either an agent or it's not an agent,
  122. 3:57which is pretty much like what the
  123. 3:58people on X has been debating about for
  124. 4:00like the past 2 years. What is actually
  125. 4:02considered an agent? But Andrew is
  126. 4:04basically like, no guys, no guys, still
  127. 4:05just don't think about it that way,
  128. 4:06okay? Let's just call things like
  129. 4:07agentic AI and as long as it involves
  130. 4:09multiple steps, it's agentic AI and you
  131. 4:11can just be on the spectrum of least
  132. 4:13autonomous to more autonomous.
  133. 4:15Paraphrasing a little bit here. Anyways,
  134. 4:16what we all can agree on though is if
  135. 4:18you wrap a LM, a model in an agentic
  136. 4:21workflow, it is going to be much better
  137. 4:23than just calling it directly. This is a
  138. 4:25fact. It would also be often times
  139. 4:26faster and more modular, meaning that
  140. 4:28you can change things a lot more easily.
  141. 4:30So that is why agentic AI is so popular
  142. 4:32and very much worth building.
  143. 4:36Okay, let's now cover from a practical
  144. 4:38perspective. What are the building
  145. 4:39blocks of Agentic AI? Andrew's building
  146. 4:42blocks for Agentic AI is pretty
  147. 4:43simplistic actually. It involves just
  148. 4:45three components. Let's start off by
  149. 4:47addressing the first two which is models
  150. 4:49and tools. Models just refers to the AI
  151. 4:52model. That can be large language models
  152. 4:53or it can be like other types of AI
  153. 4:55models, multimodm models like video
  154. 4:56generation models, audio generation
  155. 4:58models, any type of model. And then
  156. 5:00there are tools which are basically
  157. 5:01functions and capabilities that you're
  158. 5:03giving your AI agent so that it's able
  159. 5:05to perform different types of task.
  160. 5:07Examples of this would include being
  161. 5:08able to use external software through
  162. 5:10their APIs. Stuff like web search,
  163. 5:13getting real-time data, being able to
  164. 5:14check emails, things like that.
  165. 5:16Information retrieval, so being able to
  166. 5:18access certain types of databases, and
  167. 5:20code execution, being able to write code
  168. 5:22to be able to do things like math, data
  169. 5:24analysis, building stuff. So all of
  170. 5:26these are tools. There is also a third
  171. 5:28component which I will talk about in a
  172. 5:30little bit because when you are first
  173. 5:32thinking about creating an agentic
  174. 5:33system, you're really just thinking
  175. 5:34about how do you take these building
  176. 5:36blocks of different models and different
  177. 5:37tools and chain them together and put
  178. 5:39them together in such a way to achieve
  179. 5:41the results that you want. For example,
  180. 5:44say you want to develop an agentic AI
  181. 5:46system with the goal to respond to a
  182. 5:48customer email for your company. This is
  183. 5:50an example of the type of email that you
  184. 5:52may get. Subject line here is wrong item
  185. 5:54ship. It's like, "Hi, I ordered a Blue
  186. 5:55Kitchen Pro blender. Here's the order
  187. 5:57number, but received a red toaster
  188. 5:59instead. I need a blender for my
  189. 6:00daughter's birthday party this weekend.
  190. 6:02Can you help?" Susan Jones. If you want
  191. 6:04to create an AI workflow that's able to
  192. 6:06accomplish this task, the easiest way to
  193. 6:07do this is actually think to yourself,
  194. 6:10how would I go about accomplishing this
  195. 6:11task? Like, how would a human do this?
  196. 6:13Okay, so the first step would be you
  197. 6:15need to extract key information,
  198. 6:16including the order number, the fact
  199. 6:18that it was supposed to be a Blue
  200. 6:19Kitchen Pro blender, but they got a red
  201. 6:21toaster instead, and the timeline. The
  202. 6:23daughter's birthday party is for the
  203. 6:25weekend. Then you would go and find the
  204. 6:27relevant customer record, which would
  205. 6:29likely include looking into an orders
  206. 6:31database to see what was actually
  207. 6:33recorded. You would then determine what
  208. 6:35possibly could have went wrong and then
  209. 6:36send a email reply back. Now, we need to
  210. 6:39translate these three steps into what an
  211. 6:41LM would do. Well, for the first one,
  212. 6:43extracting key information, the LLM is
  213. 6:45able to extract that information just by
  214. 6:47itself. But for the next step, which is
  215. 6:48finding a relevant customer record, the
  216. 6:50LM would actually need a tool that would
  217. 6:52allow it to look into the orders
  218. 6:54database. So, it would need to provide
  219. 6:56that with that tools for orders database
  220. 6:58query. And then finally, for writing and
  221. 7:00sending the response, the LM would be
  222. 7:02able to draft the response, but in order
  223. 7:04to actually send that email, it would
  224. 7:05need to have access to another tool that
  225. 7:07allows it to send the email. Make sense?
  226. 7:09This is how the building blocks are
  227. 7:11coming together here. You have the large
  228. 7:13language models and the different tools
  229. 7:15that it requires. Of course, the type of
  230. 7:16large language model you want to use on
  231. 7:18each of these steps could vary depending
  232. 7:20on what it's supposed to be doing
  233. 7:21exactly and how it is that you're
  234. 7:23actually going to build this type of
  235. 7:24tool also varies as well. But don't
  236. 7:26worry too much about this. We're going
  237. 7:27to go into a lot more detail later in
  238. 7:28the video. But here what I want to just
  239. 7:30make really clear is that you're
  240. 7:31basically taking um a goal and then
  241. 7:34translating that into a sequence of
  242. 7:36steps in which you will then use the
  243. 7:37building blocks of models and tools in
  244. 7:39order to accomplish those steps. So the
  245. 7:42ideal situation after your AI system
  246. 7:44goes through these steps is that it
  247. 7:46would come up with a wonderful email
  248. 7:47that says something like, "Thank you so
  249. 7:49much for emailing the company. After
  250. 7:51looking at your order number, we did
  251. 7:53notice that it does correspond to a blue
  252. 7:56blender. And we apologize that you
  253. 7:58received a red toaster instead. We will
  254. 8:00send you a blender with expedited
  255. 8:02shipping so that you can get it tomorrow
  256. 8:04and make it for your daughter's birthday
  257. 8:05on the weekend. And you can keep the red
  258. 8:07toaster as well. Then the customer,
  259. 8:09Susan Jones, will probably be very
  260. 8:11happy. But what happens if your agent
  261. 8:13doesn't give you this wonderful email
  262. 8:14and instead write something like, "Oh,
  263. 8:16thank you for shopping with us. As you
  264. 8:18can see, we are so much better than
  265. 8:19Rival Corp, our competitor, or something
  266. 8:21like sure, we'll issue a refund and not
  267. 8:23actually address the issue. That is why
  268. 8:25you need the third component of Andrew's
  269. 8:27framework, which is evaluations.
  270. 8:30Evaluations, as his name suggests, is
  271. 8:32all about evaluating the results of your
  272. 8:35agent. It's like the other half of
  273. 8:37building an AI agent. Not good enough to
  274. 8:39just build an agent. You actually got to
  275. 8:40make sure it works properly, right? And
  276. 8:42I am so glad that this course covers so
  277. 8:44much in depth about evals. I will be
  278. 8:46covering this in a lot more detail in
  279. 8:47module four, but very briefly, an
  280. 8:49evaluation in this case is making sure
  281. 8:51that your AI agent doesn't actually talk
  282. 8:54about your competitors because that's
  283. 8:56kind of like in poor taste. So, how do
  284. 8:58we do this? Well, first we can compile a
  285. 9:00list of all the competitors that we
  286. 9:01don't want to be mentioned like comp
  287. 9:03code and rival co or the other code.
  288. 9:05Then you can actually write a small
  289. 9:06snippet of code like this. If competitor
  290. 9:09is in response, then the num competitor
  291. 9:12mentions is plus or equal to one. What
  292. 9:14this basically does is check for the
  293. 9:15email responses and if any of these
  294. 9:17competitors are mentioned in the
  295. 9:18response then you would just increment
  296. 9:20the counter of number of competitor
  297. 9:22mentions to one more. Of course, ideally
  298. 9:25you want this counter the number of
  299. 9:26competitor mentions to be equal to zero.
  300. 9:29But you know if it does show up you
  301. 9:31would have a way now of tracking how
  302. 9:33many times it shows up and where it is
  303. 9:35showing up. This is a very objective way
  304. 9:37to determine if these competitors are
  305. 9:39actually still showing up. And you can
  306. 9:41use this counter as a way to then tweak
  307. 9:43your prompt, tweak your system, and do
  308. 9:45stuff to it to try to lower this number
  309. 9:47so that it eventually goes to zero.
  310. 9:49Don't worry about it if any of the
  311. 9:50detail seems a little bit hazy. I'll be
  312. 9:52going into a lot more detail later in
  313. 9:53the video. But what I just want you to
  314. 9:55understand here is that see by having
  315. 9:57like an evaluation like this where you
  316. 9:59can measure the occurrence of this bad
  317. 10:02behavior, you now have an objective way
  318. 10:04to improve your system to decrease the
  319. 10:07bad behavior. If you didn't have this
  320. 10:08EBAL, this way of measuring the bad
  321. 10:10behavior, then you can like kind of
  322. 10:12think that you are improving your
  323. 10:14system, but you don't actually
  324. 10:15objectively know if you're improving the
  325. 10:17system or not. That is why evaluations
  326. 10:19are so important. This is an example of
  327. 10:21a very objective codebased approach for
  328. 10:24evaluating a result. There's another one
  329. 10:26called LM as a judge in which you're
  330. 10:28actually using another large language
  331. 10:30model to rate the results and evaluate
  332. 10:32the results of your AI agent. Anyways,
  333. 10:34more detail in module 4. This course is
  334. 10:36an excellent introduction to Agentic AI,
  335. 10:38but when you're ready to start building
  336. 10:39your own AI agents with business impact,
  337. 10:41there are many other factors that you do
  338. 10:43need to consider, which is why I highly
  339. 10:45recommend that you check out the free
  340. 10:47master AI agents resource from HubSpot
  341. 10:49Media. It provides good practical
  342. 10:50frameworks and examples on how to get
  343. 10:52started with building useful AI agents
  344. 10:54with business impact. It also includes
  345. 10:56real use cases in marketing, sales, and
  346. 10:58operations that you can directly use to
  347. 10:59start implementing your own agents to be
  348. 11:01able to do things like content
  349. 11:02production optimization, process large
  350. 11:04amounts of data, do customized outreach,
  351. 11:06and much, much more. There's also a
  352. 11:08second download and resource called how
  353. 11:09to use AI agents that has an excellent
  354. 11:11checklist to guide your organization
  355. 11:13through each phase of implementing AI
  356. 11:14agents. This is so useful and it's
  357. 11:16something that we actually use in our
  358. 11:18company whenever we're launching a new
  359. 11:19agent. Really recommend you check out
  360. 11:20the resource linked over here also in
  361. 11:22description. Thank you so much Hopsoft
  362. 11:24Media for providing practical free
  363. 11:25resources on building AI agents and for
  364. 11:28sponsoring this portion of the video.
  365. 11:29Now back to the video. But first, let's
  366. 11:31move on to modules two and three which
  367. 11:33are covering what are known as agentic
  368. 11:34design patterns. And what are agentic
  369. 11:37design patterns you may ask? Well,
  370. 11:39remember we talked about earlier about
  371. 11:40sending that email for the blender and a
  372. 11:42toaster. Got to first decompose that
  373. 11:44task and then translate that into a
  374. 11:47combination of large language model and
  375. 11:49tool use. So you could just kind of like
  376. 11:51figure that out yourself with any type
  377. 11:54of task that you want your agentic AI to
  378. 11:56do. But luckily, we do have a lot of
  379. 11:59precedence now for specific types of
  380. 12:01task and specific types of design
  381. 12:03patterns like how it is that we're
  382. 12:04decomposing it and fitting together the
  383. 12:06pieces that has proven to shown really
  384. 12:08good results because more likely than
  385. 12:09not, whatever it is that you're trying
  386. 12:10to accomplish will converge to one of
  387. 12:12these design patterns. In this course,
  388. 12:14Andrew covers four design patterns. two
  389. 12:16that are less autonomous and two that
  390. 12:18are more experimental, more autonomous.
  391. 12:21Let's now talk about the first design
  392. 12:23pattern, which is reflection. The
  393. 12:25reflection design pattern is super
  394. 12:26intuitive. For example, if you tell a
  395. 12:28human, hey, go write this email, and
  396. 12:29they're like, you know, distracted or
  397. 12:31something and maybe they just like had
  398. 12:32lunch, so they just write some sloppy
  399. 12:34email. But if you go like, hey, reflect
  400. 12:36on this and improve the email, probably
  401. 12:37the result you're going to get is going
  402. 12:39to be a lot better. Exactly the same
  403. 12:40with AI. If you tell an AI to write an
  404. 12:42email, it will first do a first draft,
  405. 12:44which might not be that great. But if
  406. 12:46you tell it to go back and reflect on it
  407. 12:48and improve it, you will end up having
  408. 12:49much better results. The second step,
  409. 12:51asking to reflect on it and come up with
  410. 12:53another draft, is literally the
  411. 12:55reflection design pattern. Simple but
  412. 12:57powerful. And remember how it is that
  413. 12:58way you can actually go about tweaking
  414. 12:59this and improving the systematically is
  415. 13:01by having the whole like evaluation
  416. 13:02section. The course also provides like
  417. 13:04code examples of how to implement design
  418. 13:06patterns like the reflection pattern,
  419. 13:08which you can see over here, but I'm not
  420. 13:09going to go into too much detail about
  421. 13:10this. You can actually go and try out
  422. 13:12the implementation with code if you want
  423. 13:13to in the course. It's actually free to
  424. 13:16get this code. So, I'll link it in the
  425. 13:17description. By the way, don't be
  426. 13:18discouraged and like rage quit right now
  427. 13:20if you don't know how to code. Um, this
  428. 13:21is absolutely doable with no code as
  429. 13:22well. They just don't show it in the
  430. 13:24course. Like, we teach the full no code
  431. 13:25approach for doing this in our agents
  432. 13:27boot camp ourselves. I also do have a
  433. 13:28video over here that covers the basics
  434. 13:30of how to implement um agents using no
  435. 13:32code approach too.
  436. 13:35Okay, let's go on to the second design
  437. 13:36pattern called tool use. So, this part I
  438. 13:38did get like a little confused because
  439. 13:40we did talk about tools uh when we
  440. 13:42talked about the building blocks, right,
  441. 13:43of like models and tools and and
  442. 13:45evaluations, but I think Andrew included
  443. 13:47this as like a full-on design pattern
  444. 13:49because he wanted to emphasize like how
  445. 13:50much tool use can elevate the
  446. 13:52performance of an agent. So, we'll go
  447. 13:54through it. Firstly, to define tool use,
  448. 13:56you're letting your agent have access to
  449. 13:58tools that's able to implement and do
  450. 14:00certain things that it normally cannot
  451. 14:02do just as a model itself. For example,
  452. 14:04if you have an agentic AI workflow in
  453. 14:06which you want to be like booking
  454. 14:08meetings and stuff, you would need to
  455. 14:09give your agent access to a calendar
  456. 14:11booking tool. What tool use? You can
  457. 14:12make it like very strict telling your
  458. 14:14agent that you have to use this exact
  459. 14:16tool to do this exact thing or you can
  460. 14:18make it more autonomous and give your
  461. 14:20agents more like a selection of tools
  462. 14:22that it can choose from and they can get
  463. 14:23to choose which tool it wants to use to
  464. 14:25accomplish a task. But whatever tools it
  465. 14:27is that you choose to give your AI agent
  466. 14:29and how much you restrict how it's using
  467. 14:30the tools. The key here is that you do
  468. 14:32need to define the tools and you need to
  469. 14:34let your agent know that you gave them
  470. 14:36access to these tools and what the tools
  471. 14:38are in the system itself. Or else your
  472. 14:40agent would just be like what is this? I
  473. 14:42don't know what this is. Like for
  474. 14:44example, if you have a personal
  475. 14:45assistant agentic AI system and you tell
  476. 14:48it to find a free slot on Thursday in my
  477. 14:50calendar and make an appointment with
  478. 14:51Alice, you would need to give it tools
  479. 14:53such as make appointment, check calendar
  480. 14:55and delete appointment and tell it in
  481. 14:57the system prompt you have access to the
  482. 14:59tools make appointment which is for
  483. 15:01making an appointment booking. Check
  484. 15:03calendar which is for checking a
  485. 15:05calendar to check for availabilities.
  486. 15:06Delete appointment in order to delete
  487. 15:08existing appointments. then you can tell
  488. 15:10your agent to like you know find that
  489. 15:12time slot between you and Alice and it
  490. 15:14should be able to use the tools at his
  491. 15:16disposal in order to accomplish the
  492. 15:17task.
  493. 15:20So how it is that you can actually
  494. 15:21define these tools for your agents.
  495. 15:23There's actually quite a few different
  496. 15:24ways of doing this. The most simple
  497. 15:26approach if you're using code is to
  498. 15:27literally just like write a function for
  499. 15:29it and then give it access to your AI
  500. 15:31agent through some type of framework. Um
  501. 15:33the example that they showed in the
  502. 15:34course is called the AI suite, but
  503. 15:36there's like a lot of other frameworks
  504. 15:37that you can use as well. Not going to
  505. 15:39go into too much detail about this, but
  506. 15:41if you are interested in implementing
  507. 15:42this, there is an assignment in the
  508. 15:44course as well, which I will link in the
  509. 15:46description. So yeah, keep that in mind.
  510. 15:48Angie also explains that there's other
  511. 15:50ways for you to give your agent tools as
  512. 15:51well. For example, through something
  513. 15:53called MCP, which is the model context
  514. 15:55protocol. This is a protocol that was
  515. 15:57developed by Anthropic that standardizes
  516. 15:59the way that tools are being defined so
  517. 16:02that it's very easy for agents to have
  518. 16:04access to different tools without having
  519. 16:05to like write custom functions for all
  520. 16:07of them. MCPS are actually really cool.
  521. 16:08A lot of developers are actually
  522. 16:09building MCPs and like selling their
  523. 16:11MCPs and stuff like that as well. So, if
  524. 16:13you are interested in learning more
  525. 16:14about it, I do have a full video which
  526. 16:15I'll link over here that goes into the
  527. 16:17depths of MCP. But for now, let's move
  528. 16:20on to the much anticipated module 4,
  529. 16:26practical tips for building agentic AI,
  530. 16:28where we talk a lot now about
  531. 16:30evaluations, the missing piece of
  532. 16:32agentic AI development and what makes
  533. 16:34this course so valuable. To properly
  534. 16:36define it, evaluations or evals are
  535. 16:38agent evaluation mechanisms that refer
  536. 16:41to processes and criteria used to assess
  537. 16:43the performance of agents within a
  538. 16:44system. We already talked about why we
  539. 16:46need evals and why it is the third major
  540. 16:48building block because you can build an
  541. 16:49AI agent, but how do you even know how
  542. 16:51good it is and how do you know how to
  543. 16:52improve it if you don't measure the
  544. 16:54results of it? And that is where eval
  545. 16:56come in. For example, say you have an
  546. 16:57invoice extraction agent. So you give
  547. 16:59the system PDF invoices which then it
  548. 17:01converts to text. Then the LM extracts
  549. 17:04four required fields like the biller,
  550. 17:05biller address, amount, due, and due
  551. 17:07date. It has access to a tool called
  552. 17:10update database which will then use in
  553. 17:12order to update the database with a new
  554. 17:14invoice and the record is created.
  555. 17:16Simple linear agentic workflow here.
  556. 17:18Now, if we want to evaluate this agentic
  557. 17:20system, one approach of doing this is to
  558. 17:22just now take 10 to 20 different
  559. 17:24invoices and then feed it through the
  560. 17:26agent and then make a record of the
  561. 17:28results. You might notice that invoice
  562. 17:30one was fine. um it was able to extract
  563. 17:32the four required fields and everything
  564. 17:34is great but you found that invoice 2 it
  565. 17:36actually mixed up the dates between the
  566. 17:38due date and the invoice date. Uh
  567. 17:40invoice 3 was okay as well and then you
  568. 17:41keep on going and you find that invoice
  569. 17:4319 it also makes up the dates and
  570. 17:44invoice 20 it makes up the dates as
  571. 17:46well. This is actually the most
  572. 17:47practical way of doing this. So from
  573. 17:49collecting this data you notice that one
  574. 17:51of the tendencies of this agentic
  575. 17:53workflow is that it mixes up dates which
  576. 17:55is not good. We actually want to improve
  577. 17:57that. So we want to build an eval for
  578. 17:59this to set as a metric for us to
  579. 18:01improve. One thing to note here is that
  580. 18:02often times you don't actually know what
  581. 18:04eval that you should be building before
  582. 18:06you actually build out your system and
  583. 18:08start running examples through it. It's
  584. 18:10actually kind of hard to think through
  585. 18:11like what are the potential like places
  586. 18:13of failure for your system until you try
  587. 18:15it out. So that's why I do like general
  588. 18:17recommendation is just like build
  589. 18:18something quickly, quick and dirty and
  590. 18:20then just like start passing through
  591. 18:22different examples and collecting that
  592. 18:23data to decide what the eval should be.
  593. 18:26Say we want to create an eval now for
  594. 18:28date extraction because we notice that
  595. 18:30it gets a date extraction wrong a lot.
  596. 18:32How do we go about doing this? Well,
  597. 18:33step number one is to manually extract
  598. 18:35the due dates from the 10 to 20 invoices
  599. 18:37that you pass through. Then you want to
  600. 18:39specify the output format of the data in
  601. 18:41the prompt itself. So then you can
  602. 18:43extract the date from the LM response
  603. 18:45using code and finally compare the large
  604. 18:48language model results to the ground
  605. 18:50truth. Does the extracted data equal to
  606. 18:53the actual data? This is an example of
  607. 18:55evaluation that uses code if you
  608. 18:57actually want to implement evaluation
  609. 18:58using code yourself. By the way, also in
  610. 19:00the course there is the code for this
  611. 19:01that you can run yourself.
  612. 19:05Anyways, this is an example of a ebal
  613. 19:06that you can implement so that in the
  614. 19:08future as you're like tweaking your
  615. 19:09prompt and then changing things around,
  616. 19:11you can actually measure if your invoice
  617. 19:13agent system is becoming more accurate
  618. 19:16or less accurate at date extraction.
  619. 19:18Hopefully more accurate. There are a lot
  620. 19:20of other types of doing it. And the way
  621. 19:21that the course frames it is that you
  622. 19:23can think about evals as falling under
  623. 19:25two different axes. This date extraction
  624. 19:27eval is an example of an objective eval
  625. 19:30because the result of it is that the
  626. 19:32extracted date is either equal to the
  627. 19:34actual date or it's not equal to the
  628. 19:36actual date. Very objective either right
  629. 19:38or wrong. With this type of eval system,
  630. 19:40you can usually implement it pretty
  631. 19:42easily just through code itself. Another
  632. 19:44type of eval is a subjective eval. This
  633. 19:46is when the result isn't just like a yes
  634. 19:48or no binary kind of situation. An
  635. 19:50example of a subject eval is if you're
  636. 19:52trying to say assess the results of a
  637. 19:54research agent. The agent is supposed to
  638. 19:56research on a certain topic and then
  639. 19:58output like some sort of essay at the
  640. 19:59end. Say if you give it a prompt like
  641. 20:01research and write an essay about recent
  642. 20:03black hole science. There's no way to
  643. 20:05objectively evaluate how good the
  644. 20:07research and how good the essay ends up.
  645. 20:09You can't just like, you know, count the
  646. 20:11number of words or something like that,
  647. 20:12right? So, what you need to do to create
  648. 20:14this kind of email is first choose a few
  649. 20:16different topics that you're going to be
  650. 20:18using as the prompt. Like maybe
  651. 20:20something about buck coal, something
  652. 20:21about robot harvesting, renting versus
  653. 20:23buying a home in Seattle, etc., etc. So,
  654. 20:25for each of these prompts, you need to
  655. 20:27actually manually establish a ground
  656. 20:28truth annotation, which are like golden
  657. 20:31standard talking points that you know
  658. 20:33needs to be researched and needs to be
  659. 20:35part of the final essay as well. For
  660. 20:37example, with black holes, these could
  661. 20:38be stuff like event horizon or radio
  662. 20:40telescope. for robotic harvesting. It
  663. 20:42could be like robo pick and pinchers. To
  664. 20:44actually do this, you probably need to
  665. 20:45like find someone who actually knows
  666. 20:46these topics or you yourself know these
  667. 20:48topics as well. So, this part is manual.
  668. 20:50But then the second part is when you can
  669. 20:52use the LM as a judge. So, you have
  670. 20:53another type of LM to count how many of
  671. 20:56these topics actually appear in the
  672. 20:58research and the final results. The
  673. 21:00reason why you need to use an LLM to to
  674. 21:02count this and not just like um
  675. 21:04objectively use code to count this is
  676. 21:06because a certain term can appear in a
  677. 21:08lot of different iterations. Like the
  678. 21:09term event horizon could be like event
  679. 21:11horizon, but it could also be like
  680. 21:12uppercase, lower case, could be the
  681. 21:13horizon on the type of event at the
  682. 21:16horizon or like when the horizon is near
  683. 21:19something like that, right? There's like
  684. 21:20so many different types of iterations of
  685. 21:21this that you need an LLM to be able to
  686. 21:24capture all the different types of these
  687. 21:25iterations. Then you can get the LM to
  688. 21:27give you a final score for each prompt
  689. 21:29based upon how many of these terms were
  690. 21:31captured in the final result. This on
  691. 21:33screen now is an example of a prompt for
  692. 21:35the LM as a judge. That's the first axis
  693. 21:38of this 2x2 matrix uh for the different
  694. 21:40types of evals. And this one is
  695. 21:42objective versus subjective. The second
  696. 21:44axis is whether something has a perample
  697. 21:46ground truth or does not have a perample
  698. 21:49ground truth. Both the examples that we
  699. 21:50just talked about, the invoice example
  700. 21:52and the LM research and essay writing
  701. 21:55example are agenting systems that have a
  702. 21:57perample ground truth. This means that
  703. 21:59for every example like every input that
  704. 22:01we give there is a established ground
  705. 22:04truth for that example. like the invoice
  706. 22:06extraction has a specific date that is
  707. 22:08the correct result that is different for
  708. 22:10all the different examples and the
  709. 22:11topics for the research agent are
  710. 22:13different for each of the examples too.
  711. 22:15However, you can also have emails that
  712. 22:16have no perample ground truth meaning
  713. 22:19that there's a singular universal
  714. 22:20standard that is there for every single
  715. 22:23example. Say if you have an agentic
  716. 22:25system that is for marketing copy length
  717. 22:27um and you just want to make sure that
  718. 22:28the marketing copy length is always
  719. 22:30going to be less than 10 words. So like
  720. 22:32the number 10 is like the universal
  721. 22:33magical number. This is also an
  722. 22:35objective eval. So you can actually
  723. 22:36write code. So you can literally just
  724. 22:38write if the length of the text is less
  725. 22:40than or equal to 10, then that is good.
  726. 22:42If it's not, then it's bad. The final
  727. 22:43quadrant of this 2x2 matrix is when you
  728. 22:46have something that is subjective and
  729. 22:48does not have a per example ground
  730. 22:50truth. Say you have an agentic system
  731. 22:51that is supposed to create different
  732. 22:53graphs. The judging of the graph is
  733. 22:55going to be subjective. The result of
  734. 22:57the graph is going to be subjective. And
  735. 22:59the way that you judge it is going to be
  736. 23:00based off this universal criteria where
  737. 23:02you would ask the LM as a judge to grade
  738. 23:05the graph to see whether it has axis
  739. 23:08labels. Are the colors easy to see? Is
  740. 23:10there any overlapping text? Etc., etc.,
  741. 23:12etc. So, that would be like a universal
  742. 23:14grading criteria.
  743. 23:17Okay, great. That covers the four
  744. 23:19general types of of evals that you can
  745. 23:21make. Some of the tips that the course
  746. 23:23suggests when you're creating these
  747. 23:24evals, uh firstly is to just quick and
  748. 23:27dirty is okay to start. just create
  749. 23:29something and then just create some
  750. 23:30evals and just get started like that.
  751. 23:32You you can always continue to expand
  752. 23:33the set of evals. You want to try to use
  753. 23:35a lot of different types of examples to
  754. 23:37try to capture all the different types
  755. 23:39of errors that you can change into
  756. 23:41evals. And finally, you want to look for
  757. 23:43places where the performance of the
  758. 23:45agentic system is worse than a humans.
  759. 23:47These are the best areas for
  760. 23:48improvement. Great. That was the module
  761. 23:50on evals. Even if you implement just
  762. 23:52like a few of these different evals,
  763. 23:55you'll be setting up a system that will
  764. 23:56help you dramatically improve the
  765. 23:58results of your agents. I feel like
  766. 24:00evals are such an open secret and by
  767. 24:02implementing them, massive return on
  768. 24:03investment.
  769. 24:05All right, we're almost done. We are at
  770. 24:07module five, which is the patterns for
  771. 24:09highly autonomous agents. We're going to
  772. 24:11be covering two of the more experimental
  773. 24:12agents that are also more autonomous.
  774. 24:14The two patterns that we covered
  775. 24:15earlier, reflection and tool use, are
  776. 24:18less autonomous and they're more
  777. 24:19predictable. While for these
  778. 24:20experimental ones, planning and multi-
  779. 24:22aent systems, these are less
  780. 24:24predictable, less controllable, but if
  781. 24:26you get it right, the results can be
  782. 24:28really, really good. Let's start off
  783. 24:29with planning and jump straight to an
  784. 24:31example. Say you have a customer service
  785. 24:33agent in which it has an inventory
  786. 24:35database of different types of products
  787. 24:37in a store. You have like ID, name,
  788. 24:39description, price, and stock. An
  789. 24:41example of a customer query that the
  790. 24:42agent would receive would be like, do
  791. 24:44you have any round sunglasses in stock
  792. 24:46that are under $100? So you see this is
  793. 24:48actually a pretty difficult query
  794. 24:50because it involves the customer survey
  795. 24:52agent to have to go through the data set
  796. 24:54and first like find what are the round
  797. 24:56styles. Then they has to go pick out
  798. 24:58which of these are in stock and finally
  799. 25:00which of these prices are going to be
  800. 25:01under $100 to be able to come up with
  801. 25:03the final answer of yes we have our
  802. 25:05classic sunglasses which are classic
  803. 25:06round metal frame and cost $60. And of
  804. 25:08course this is just a single example of
  805. 25:10the kind of questions that customers
  806. 25:12will be asking. There's a broad range of
  807. 25:14questions that will be asked and that
  808. 25:15can be answered by using this database.
  809. 25:17However, there is no like set path for
  810. 25:20you to actually answer these questions.
  811. 25:22Each time you would have to figure out
  812. 25:23and plan out what is the way to actually
  813. 25:26answer these questions. This is where
  814. 25:27the planning design pattern comes in.
  815. 25:29Going back to that example of asking a
  816. 25:31question, do you have any round
  817. 25:32sunglasses in stock that are under $100?
  818. 25:34The planning design pattern will first
  819. 25:35have an LLM figure out what is the
  820. 25:38actual plan to answer this question
  821. 25:40given the set of tools that it has. Like
  822. 25:42it has tools like get item descriptions,
  823. 25:44check inventory, process item return etc
  824. 25:47etc. So it would use this information
  825. 25:49and the information of the database and
  826. 25:50come up with a plan like number one have
  827. 25:52to use get item description tool to find
  828. 25:54the round sunglasses. Second step is to
  829. 25:56use check inventory to see if results
  830. 25:58are in stock. Then third step is to use
  831. 26:00get item price to see if in stock
  832. 26:01results are less than $100. Then the
  833. 26:03agent will take that plan and execute
  834. 26:05each of these steps to get the final
  835. 26:06result. By the way, if you want to dig
  836. 26:08into this and actually implement these
  837. 26:09evals yourself, there is again
  838. 26:11assignments that you can try out in the
  839. 26:13course which I will link below. What is
  840. 26:15cool about this planning design pattern
  841. 26:16is that the plan is not predetermined.
  842. 26:19So the agenda system has to actually
  843. 26:20figure out the plan first and then go
  844. 26:22and execute all of it. Which means of
  845. 26:24course that the downside of this
  846. 26:25planning design pattern is that you
  847. 26:27don't actually know what the plan is. So
  848. 26:29it's actually very hard to troubleshoot
  849. 26:31and anticipate what kind of result
  850. 26:32you're going to get until it actually
  851. 26:33goes through the process itself. The
  852. 26:35upside of this is this increased amount
  853. 26:36of flexibility and sometimes like it is
  854. 26:38pretty magical to witness an agent be
  855. 26:41able to plan out super complex tasks and
  856. 26:43then execute it and get like crazy good
  857. 26:45results. This is the kind of magic that
  858. 26:47more autonomous systems have. Less
  859. 26:49control but more probability of really
  860. 26:51cool results.
  861. 26:54The other more experimental autonomous
  862. 26:56design pattern is a multi-agentic
  863. 26:58workflow. The idea of a multi- aent
  864. 27:00system is to take an agentic system and
  865. 27:03actually have it like be multiple
  866. 27:04different agents working together to
  867. 27:06produce the final result. The intuition
  868. 27:08for why this can be better is kind of
  869. 27:09like a company, right? If you have like
  870. 27:11one person trying to do all the
  871. 27:13different tasks in a company, um it's
  872. 27:15probably not going to do the best job
  873. 27:16and it's going to get overwhelmed
  874. 27:17because it has to do like so many
  875. 27:18different things and juggle so many
  876. 27:19different things at the same time. But
  877. 27:21if you have different people in that
  878. 27:23company that specialize in different
  879. 27:24things, each person would be able to
  880. 27:26focus better on their specific task. and
  881. 27:27when you put it together, it's going to
  882. 27:29produce a better result. That's why we
  883. 27:30usually have multiple people in a
  884. 27:32company and not just like the CEO trying
  885. 27:33to do everything themselves. So that's
  886. 27:34the same intuition for developing
  887. 27:36multi-agentic systems. For example, say
  888. 27:38you have a marketing team agentic system
  889. 27:40and instead of just having one agent,
  890. 27:42you actually divide that into three
  891. 27:44different sub aents. A researcher, a
  892. 27:46graphic designer, and a writer. So that
  893. 27:49when you receive a query like create a
  894. 27:50summer marketing campaign for
  895. 27:52sunglasses, your research agent will
  896. 27:54focus on conducting research and figure
  897. 27:56out that here are current sunglasses
  898. 27:57trends and competitor offerings. It
  899. 27:59would pass along that information to a
  900. 28:01graphic designer agent which would then
  901. 28:03come up with like five different data
  902. 28:05visualizations and five different
  903. 28:06artwork options for the report. It would
  904. 28:09then pass all of this to the final
  905. 28:10writer agent which will then compile
  906. 28:13everything together and write the final
  907. 28:14report. So this would be a marketing
  908. 28:16team agentic system that's able to
  909. 28:18generate report for a marketing campaign
  910. 28:20on a specific topic. There's actually
  911. 28:22quite a lot of research done on multi-
  912. 28:23aent systems and in our boot camp as
  913. 28:25well like we have a whole week that
  914. 28:26covers multi- aent systems because the
  915. 28:28results can be really really good like
  916. 28:30much much better uh than if you just use
  917. 28:32a single agent. This course doesn't
  918. 28:33cover too much into it though but I will
  919. 28:35leave a link in the description for an
  920. 28:36anthropic article that goes into a lot
  921. 28:38more detail about multi-agent systems. I
  922. 28:40also have a video that covers multi-
  923. 28:41aent systems as well which you can check
  924. 28:43out at this link over here if you prefer
  925. 28:45like a video approach. Anyways, for this
  926. 28:47course they also have codebased
  927. 28:48assignments where you can implement
  928. 28:50these more autonomous systems too. Oh my
  929. 28:52god. Okay, we have come to the end of
  930. 28:54this video of this crash course. I
  931. 28:56literally had let's see 93 pages of
  932. 28:59notes which I have condensed for you
  933. 29:01guys here. I overall I think this course
  934. 29:03really was quite good. It has a very
  935. 29:05good overview of all of the different
  936. 29:07parts um for building agentic AI. My
  937. 29:10critique though, if I may, Andrew does
  938. 29:12approach it from a very like researchy
  939. 29:14theoretical perspective, which makes
  940. 29:15sense given his background as like a
  941. 29:16research scientist, but I do feel like
  942. 29:19he didn't really cover like the actual
  943. 29:21process of deployment and what does that
  944. 29:23actually look like when you put it into
  945. 29:25the world and get users to start using
  946. 29:26your agents and stuff like that. That's
  947. 29:27like a whole other beast which we cover
  948. 29:29a lot in our boot camps because we're
  949. 29:30coming from like a more practical
  950. 29:32approach so that we're building these
  951. 29:33agents and actually deploying them and
  952. 29:35people are using them and you know
  953. 29:36you're hopefully making money from them
  954. 29:37as well. Another critique that I do have
  955. 29:39um is that the assignments here they're
  956. 29:41all like based upon code. If you didn't
  957. 29:43know any better you would have the
  958. 29:44misconception that you have to know how
  959. 29:46to code in order to implement these
  960. 29:47systems and that is not true. So I just
  961. 29:49want to like make that super clear for
  962. 29:51you guys too. You can do this with no
  963. 29:52code as well. I'll link in description
  964. 29:54some of the videos that I cover where
  965. 29:56you can implement these systems with no
  966. 29:57code completely.
  967. 29:59All right, as promised, here is the
  968. 30:01little assessment in which if you can
  969. 30:03answer these questions, then
  970. 30:04congratulations. You are going to be
  971. 30:06able to retain all the information we've
  972. 30:07just covered from this 8hour course.
  973. 30:09Please write your answers in the
  974. 30:11comments. All right, thank you so much
  975. 30:12for watching until the end of this
  976. 30:13video. I really hope that you learned a
  977. 30:15lot about Agentic AI and you're super
  978. 30:16excited to start building your own
  979. 30:18agents as well. And I will see you guys
  980. 30:20in next video or live stream.

About this transcript

This page contains the full transcript of 8 Hour AI Agents Course in 30 Minutes (Deep Learning AI) by Tina Huang, generated from the public captions YouTube serves with the video. The transcript has 6,840 words across 980 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.