YouTube2Text

YouTube transcript (DGI4vucR71w) — Transcript

10,838 words · 1,760 segments · language en · Watch on YouTube

Full transcript

  1. 0:05Welcome everyone. This is CS 329Z,
  2. 0:09engineering AI agents.
  3. 0:12And today is the introduction to agentic
  4. 0:13systems.
  5. 0:15I'm D Young, assistant professor in
  6. 0:16computer science here. And I'm
  7. 0:18co-teaching this course with Michael
  8. 0:21Ryan and John Young.
  9. 0:23Today we will cover logistics and also
  10. 0:26give you a short history of agents,
  11. 0:28what a agent is made of, and also the
  12. 0:31open problems we will work on this
  13. 0:32quarter.
  14. 0:34I will first like to Michael and John
  15. 0:36introduce themselves.
  16. 0:38>> Hi everyone. I'm Michael. I'm a
  17. 0:40second-year PhD student here at Stanford
  18. 0:42and I'm advised by D and Percy.
  19. 0:45So my research is on open-source
  20. 0:48human-centered AI. And in the past
  21. 0:50agent-related work I've worked on the
  22. 0:52DSPy framework for
  23. 0:55programming, not prompting language
  24. 0:57models. So that's my little claim to
  25. 1:00agents.
  26. 1:01>> Hi, my name is John. I'm a third-year
  27. 1:03PhD at Stanford. I'm advised by D. And
  28. 1:06in the past I've worked on coding agents
  29. 1:09and coding evaluation. So I created a
  30. 1:11SweetBench, SweetAgent, ProgramBench,
  31. 1:14SweetSmith, if you've heard of those
  32. 1:16works.
  33. 1:18>> Great. Okay, so
  34. 1:21with the instructor team, I want to
  35. 1:24formally welcome you to this course.
  36. 1:26The primary contact to reach out to us
  37. 1:28is to use at discussion. If you want to
  38. 1:31be added to it, just send us an email.
  39. 1:33We prefer you send everything to this
  40. 1:37email list so that we can take care of
  41. 1:39your request very quickly.
  42. 1:41The website will maintain all the
  43. 1:42lectures, slides, and homeworks. So
  44. 1:45that's your primary website to check
  45. 1:48out.
  46. 1:49If you haven't enrolled in at
  47. 1:50discussion, please do since we post a
  48. 1:52lot of announcements there as well.
  49. 1:55So in today's lecture, Um, in addition
  50. 1:58to the logistics, I will mention in a
  51. 2:00few seconds. We're going to provide a
  52. 2:02history of agents. Uh, this is There is
  53. 2:05actually a really long history behind
  54. 2:07agents.
  55. 2:08Uh, we will talk about agent
  56. 2:10architecture. This is the core content
  57. 2:12today. And then compound AI systems, rag
  58. 2:15to use, and MCP. Uh, we will end with a
  59. 2:18brief highlight of key challenges,
  60. 2:20including the recent multi-agent risk
  61. 2:23and collusion we have been seeing.
  62. 2:26So, let's get it started.
  63. 2:28Uh, why we create the CS 329Z? Uh,
  64. 2:32so, when we think about large language
  65. 2:34models, agents these days, there seems
  66. 2:37to be very different perspectives
  67. 2:38involved. You can take a modeling
  68. 2:41perspective. You can also take a system
  69. 2:43perspective. A lot of what we do is
  70. 2:45actually around the engineering.
  71. 2:48And if you think about that, LM people,
  72. 2:50uh,
  73. 2:51I think they know how models are
  74. 2:53trained, pre-training, post-training,
  75. 2:55prompting, uh, alignment, and
  76. 2:57evaluation. On the other side, system
  77. 3:00people know how to build reliable tools,
  78. 3:02systems,
  79. 3:04um, more related to agents. We have
  80. 3:06tools, APIs, environment, and
  81. 3:08infrastructure.
  82. 3:10So, the expectation for this course is
  83. 3:12that we're going to cover a lot of the
  84. 3:15foundations. But more importantly, we
  85. 3:17want it to be more hands-on building
  86. 3:19plus evaluation of agents.
  87. 3:23A quick glance of the content. So, uh,
  88. 3:26week one to week five, uh, we will talk
  89. 3:28about large language models for
  90. 3:30builders, um, with a highlight of
  91. 3:33retrieval augmented generation and its
  92. 3:35variants in many, many follow-up agent
  93. 3:38architectures.
  94. 3:39Tool use and a function call will be
  95. 3:42another highlight here. We will slowly
  96. 3:44move to agent design, training, and
  97. 3:46scaffolding. And also the design of
  98. 3:49agent memory, uh, as well as
  99. 3:51orchestration and multi-agent systems.
  100. 3:55Um, the second half of this course, we
  101. 3:57are going to have optimization, harness,
  102. 4:00and data for agentic systems, data
  103. 4:03quality, data selection, etc.
  104. 4:05The evaluation and benchmark design will
  105. 4:08also be highlighted in different ways.
  106. 4:11Agent safety and the guardrail
  107. 4:13will also have two special highlights of
  108. 4:16coding agent and also a more emerging
  109. 4:20topic called a proactive agents.
  110. 4:24We have actually two special teams who
  111. 4:28are going to show up in our class this
  112. 4:30quarter.
  113. 4:31First, we will welcome the terminal
  114. 4:33bench team to talk about how they design
  115. 4:37terminal bench and a lot of questions
  116. 4:39you can share with them.
  117. 4:41We will also welcome the SweetBench team
  118. 4:43to come to chat with us. So, for both
  119. 4:47lectures, it will be a team reunion from
  120. 4:50those two teams that have created a very
  121. 4:52fundamental evaluations in our field.
  122. 4:55Course policy.
  123. 4:57First, I think
  124. 4:59the biggest message that I want to
  125. 5:00highlight is that
  126. 5:02you should use AI to learn.
  127. 5:06You may want to get yourself familiar
  128. 5:08with the Stanford Honor Code. And laptop
  129. 5:11use is only allowed for taking notes,
  130. 5:14etc. And we strongly discourage the use
  131. 5:17during the lecture so that you can be
  132. 5:20more focused.
  133. 5:21Prerequisite and the compute.
  134. 5:24So, first,
  135. 5:25we strongly recommend that you have
  136. 5:27taken any NLP or large language model
  137. 5:30related training.
  138. 5:32You are expected to know basic LLMs
  139. 5:34since we are not going to cover like
  140. 5:36fine-tuning or pre-training or
  141. 5:38alignment, etc.
  142. 5:40In terms of compute, we are very
  143. 5:42grateful
  144. 5:43that Lot Institute gives each student
  145. 5:46$200 in terms of credits. And Thinking
  146. 5:49Machine will provide a $250
  147. 5:52for Tinker credits.
  148. 5:55I want to switch a little bit to kind of
  149. 5:57conclude the logistics section by
  150. 5:59showing you a figure.
  151. 6:01This is a
  152. 6:03just like a specific instance of what's
  153. 6:05happening today.
  154. 6:08With a lot of data, if you show on the
  155. 6:10top, this is actually homework score
  156. 6:13from two populations. People who use AI
  157. 6:16and the people who do not use AI.
  158. 6:19Very interestingly, I think with AI,
  159. 6:22everyone's homework score are just so
  160. 6:24great. But then if you look at the
  161. 6:26bottom, this is actually the exam score
  162. 6:28for the same population.
  163. 6:30Before using AI,
  164. 6:32this is definitely more kind of shifted
  165. 6:34towards 100 here.
  166. 6:36With AI, we have way more people did
  167. 6:39worse in their exams.
  168. 6:41And I think we see this kind of pattern
  169. 6:43very frequent these days.
  170. 6:46The second example that I want to share
  171. 6:48with you is actually from Anthropic.
  172. 6:51They did a study with their own
  173. 6:53engineers to figure out how the use of
  174. 6:56coding agent actually impact coding.
  175. 6:59As you can see from the left side, if
  176. 7:02you use AI, the task completion time
  177. 7:05actually decreased.
  178. 7:06But then on the right, the quiz score in
  179. 7:09terms of your mastery actually
  180. 7:11significantly decreased. That is,
  181. 7:13despite the coding agent that can give
  182. 7:15us 1,000 line of code, it's a little bit
  183. 7:18unclear whether we fully understand such
  184. 7:21AI output.
  185. 7:22So, with that, I want to conclude the
  186. 7:25logistics by saying that we really hope
  187. 7:27that you can use AI to learn how to
  188. 7:29learn. And this is really just about you
  189. 7:32and your CS 329Z.
  190. 7:35With that, any quick questions for
  191. 7:37logistics?
  192. 7:41Okay. If there's no question, I will
  193. 7:43continue to the history of agents.
  194. 7:46One thing that I do want to mention is
  195. 7:48that
  196. 7:49the learning objective today is for you
  197. 7:52all to get a familiar with agent loop on
  198. 7:54top of large language models and the
  199. 7:56different key components.
  200. 7:59So, let's get it started.
  201. 8:01This is a word that's probably very very
  202. 8:04popular today.
  203. 8:05And uh I'm going to do next is to trace
  204. 8:08where this idea actually came from, how
  205. 8:11it developed over the decades, and why
  206. 8:13it suddenly take off with large language
  207. 8:16models.
  208. 8:18Um if you look at the uh more linguistic
  209. 8:21side of agents, uh the word agent
  210. 8:24actually comes from Latin, uh meaning
  211. 8:26that one uh who acts. And uh this term
  212. 8:30also uh entered English in the later um
  213. 8:3414th century. So, in AI, the first index
  214. 8:37use of it is in this specific IJCAI
  215. 8:40paper from 19 uh '73, uh where Hewitt
  216. 8:44and Bishop and Steiger talk about this
  217. 8:47architecture called actor algorithm,
  218. 8:50where each actor is an active agent that
  219. 8:53wait for a message and then does what
  220. 8:55the message tell them to do. So, the
  221. 8:57idea of software entities that act on
  222. 9:00their own is older than most people can
  223. 9:04think about.
  224. 9:06And then, I think like a few years after
  225. 9:08that, um the Contract Net Protocol
  226. 9:12actually talk about how agents dividing
  227. 9:14task with each other and then
  228. 9:16negotiating task.
  229. 9:19This trend actually continues. In Marvin
  230. 9:21Minsky's um Society of Minds, it
  231. 9:24actually talk about how a group of sub
  232. 9:26agents actually work together to get it
  233. 9:28something done. And if you kind of
  234. 9:31reflect on this, this is actually the
  235. 9:33earlier idea of multi-agent
  236. 9:35orchestration.
  237. 9:38I think the history actually continues.
  238. 9:40Around the 1990s, we actually see more
  239. 9:43and more emergence of uh software
  240. 9:46systems that uh serve as agents. Uh from
  241. 9:49Pattie's designing autonomous agents to
  242. 9:52later um
  243. 9:54people talk about uh agent as autonomy,
  244. 9:57uh social ability, reactivity,
  245. 9:59proactivity, trying to define agents in
  246. 10:02different ways.
  247. 10:03The most uh familiar um
  248. 10:05definition to many of us is actually
  249. 10:08from Russell and Norvig, uh which
  250. 10:10defines agents as anything that they
  251. 10:13perceive through sensors and act through
  252. 10:17actuators here.
  253. 10:20We now kind of like enter a different
  254. 10:22stage, which we call reinforcement
  255. 10:24learning error here.
  256. 10:26This is a classic agent environment loop
  257. 10:29that many of us get familiar during RL
  258. 10:32courses. So, what happened here is that
  259. 10:35agent take action in the environment and
  260. 10:38such action will
  261. 10:40receive a reward. And then they do the
  262. 10:42action based on specific state and the
  263. 10:44current reward. So, this is a very
  264. 10:47classic agent environment loop from
  265. 10:49Sutton here.
  266. 10:50And then I think like we see more and
  267. 10:53more promising from promises from this
  268. 10:56kind of setup.
  269. 10:58DQN refers to deep
  270. 11:01Q network. This actually from DeepMind's
  271. 11:04earlier days, where they build
  272. 11:07AI systems to actually solve specific
  273. 11:10games.
  274. 11:12Now, I think a later
  275. 11:13you also see that
  276. 11:15AlphaGo and OpenAI 5 showing that this
  277. 11:19kind of approach actually can reach
  278. 11:21superhuman performance
  279. 11:23in like Go and I think this one is Dota
  280. 11:252.
  281. 11:26So, those were agents, but each of them
  282. 11:29are actually trained for a specific
  283. 11:31task.
  284. 11:34So, what actually changed is
  285. 11:36introduction of large language models.
  286. 11:38I think the first kind of impressive
  287. 11:41capability of LLMs is actually in
  288. 11:43context learning.
  289. 11:45Built upon this kind of
  290. 11:47transformer-based architectures,
  291. 11:49GPT-3 actually showed that you can just
  292. 11:52talk about a task and give you a few
  293. 11:54demonstrations. Like in this particular
  294. 11:57example of few-shot, in order to do
  295. 12:00translation, you can offer certain
  296. 12:02examples and then the model can actually
  297. 12:04follow such examples and then do the
  298. 12:07task. This entire process, unlike For
  299. 12:10systems I showed you, there is no
  300. 12:12gradient update happening here.
  301. 12:18I think like not only like in terms of
  302. 12:20prompting or in context learning,
  303. 12:23prompting could do more
  304. 12:25than teach a model a new task. It turned
  305. 12:28out that models themselves, they can
  306. 12:30also reason step-by-step and then use
  307. 12:33tools for the things that they cannot do
  308. 12:36alone.
  309. 12:37So this particular example is called a
  310. 12:39chain of thought prompting. Uh you can
  311. 12:41see that asking the model to actually
  312. 12:43work through a problem step-by-step
  313. 12:46actually give more accurate answers in
  314. 12:48general.
  315. 12:50Uh WebGPT was proposed by OpenAI. This
  316. 12:54actually enables language model in
  317. 12:56earlier days to do to use like web
  318. 12:59browser so that it can actually look up
  319. 13:01things online instead of only rely on
  320. 13:04memory or parameters.
  321. 13:06Um this actually leaded to a lot a lot
  322. 13:09of follow-up work in terms of like web
  323. 13:11browser agent.
  324. 13:14Um one of the very interesting
  325. 13:16architecture is actually called a
  326. 13:18modular reasoning knowledge and the
  327. 13:20language.
  328. 13:21I would view it as one of the first
  329. 13:23earlier systems to think about to use.
  330. 13:27So basically this architecture actually
  331. 13:29decides which outside expert to send a
  332. 13:32request to. Is this going to be a
  333. 13:35weather request? So then I should use a
  334. 13:37weather API.
  335. 13:39If this is a coding task, maybe I should
  336. 13:41send it to a Python programming tool
  337. 13:44etc. So now the model actually can think
  338. 13:48and it can take actions through tools.
  339. 13:53So I think we now have like both
  340. 13:55important pieces here, that is models
  341. 13:57can reason step-by-step and the models
  342. 14:00that can use tools. Those are two very
  343. 14:02very important a prerequisite for the
  344. 14:05emergence of agents. I'm going to show
  345. 14:08you a very short timeline in what
  346. 14:10happened in history, that is 2 years
  347. 14:12ago.
  348. 14:13So
  349. 14:142022 React was
  350. 14:17proposed
  351. 14:19Um, so the name actually means reason
  352. 14:22plus act. So, instead of thinking once
  353. 14:25and then answering the question, what
  354. 14:27the model does here is actually a loop.
  355. 14:30It thinks about what to do next, take
  356. 14:33action, and then look at what happened,
  357. 14:36and then use that to think again. So,
  358. 14:38it's actually a loop happening here.
  359. 14:42After that, I think we see that uh more
  360. 14:45than more uh models get released. So,
  361. 14:48ChatGPT get a launched, and suddenly I
  362. 14:50think a millions of people start to talk
  363. 14:53to LLMs every day.
  364. 14:56And then GPT-4 was released. And the
  365. 14:58reason why I want to mention this is
  366. 15:00that if you think about agents, they
  367. 15:02really rely on the base model, the
  368. 15:04strong capability from those base
  369. 15:06models. So, when GPT-4 came out, we have
  370. 15:09a much stronger model that it could uh
  371. 15:11follow long instructions and plan more
  372. 15:15reliably.
  373. 15:17Um, within 2 weeks, there were actually
  374. 15:19two open-source projects came out that
  375. 15:22really shape what we think around
  376. 15:24agents. The first one is called the Baby
  377. 15:27AGI.
  378. 15:28Um,
  379. 15:29it basically gave the model a goal and
  380. 15:31let it write uh its own task list, walk
  381. 15:35through the task, and add a new ones as
  382. 15:38it goes.
  383. 15:40Uh, AutoGPT was released uh um I think
  384. 15:43in the same uh months, 2 days later. Um,
  385. 15:47basically try to connect the GPT-4 to
  386. 15:49tools such as web search, file systems,
  387. 15:52etc. And then let it uh keep uh going on
  388. 15:55on its own without a person really like
  389. 15:58typing the next prompt each time.
  390. 16:01I think uh like now if you look back, it
  391. 16:03was definitely very ahead of time ideas.
  392. 16:06The reality is that none of them
  393. 16:08actually really worked very well.
  394. 16:11Um, but they got the people's attention
  395. 16:13and really opened our imagination about
  396. 16:16what's possible.
  397. 16:17Um, I think that that's also probably
  398. 16:20the time when the word agent really
  399. 16:22enters like everyday use.
  400. 16:25So um
  401. 16:26I want to kind of go a little bit deeper
  402. 16:28here to share with you like the
  403. 16:30underlying model and also agents, how
  404. 16:33they have improved since 2023.
  405. 16:37Um so this page, uh if you look at that,
  406. 16:40I only show the GPT-3, GPT-4, and GPT-5.
  407. 16:43Um the underneath model actually is
  408. 16:47better and better.
  409. 16:49If you just contrast the GPT-3 versus
  410. 16:51GPT-4, we see that on MMLU, there's a
  411. 16:5443% increase. This is actually true for
  412. 16:58all the data sets that we have.
  413. 17:01From GPT-4 to GPT-5, some of those data
  414. 17:03set actually didn't show up here because
  415. 17:05they got solved or they saturated so
  416. 17:08that they cannot be used as very
  417. 17:10reliable signals.
  418. 17:12Instead, we actually introduced more
  419. 17:15evaluation such as GPQA, which is a very
  420. 17:18challenging QA setting, and also harder
  421. 17:22and harder math problems here.
  422. 17:25So the key message here is that each
  423. 17:27generation of base model actually made a
  424. 17:30really big jump in terms of their
  425. 17:32performance.
  426. 17:34And a stronger model means that the
  427. 17:36agent now can actually reason better.
  428. 17:38They can make fewer mistakes,
  429. 17:41hallucinate less,
  430. 17:43um and along the way they can handle
  431. 17:44longer and longer task.
  432. 17:47And then let's look at the agent side.
  433. 17:50So the agent side is also improving
  434. 17:54as well, not only just like the raw
  435. 17:56model capability.
  436. 17:57Uh so this page is actually around like
  437. 18:00SweetBench verified. Um as you can see
  438. 18:03uh in January 2025, I think uh the
  439. 18:06performance is only around a 30% but
  440. 18:09then over the last uh
  441. 18:12um 1 year and a half, now that all the
  442. 18:15numbers are actually reached a very high
  443. 18:17stage here.
  444. 18:20And then right side of this is from
  445. 18:21Astro release. So, you can see that OS
  446. 18:24world, which is a computer used
  447. 18:26evaluation. And also on terminal bench,
  448. 18:29which is a more like a command line
  449. 18:30task.
  450. 18:31Um the accuracy are just impressive.
  451. 18:35There's also a trend that people start
  452. 18:37to pay attention to the efficiency of
  453. 18:39the process. This is why you are seeing
  454. 18:41that most of those curves these days
  455. 18:43actually have both price and accuracy
  456. 18:46there together.
  457. 18:48So, this is how we got here. Um it's a
  458. 18:50long history in the last 16 years.
  459. 18:54Um so, what I'm going to do next is to
  460. 18:55really open one app and then see what
  461. 18:58our agent is actually made of.
  462. 19:03Any quick questions here?
  463. 19:07Okay. Um yes.
  464. 19:10>> Do you think models
  465. 19:12capacity or ability is going to saturate
  466. 19:14in in a few years or is going to still
  467. 19:17like develop for
  468. 19:19like 10 years or so?
  469. 19:22>> I think the
  470. 19:23first uh just to repeat the question.
  471. 19:25The question is will model capabilities
  472. 19:27saturate or will it keep improving?
  473. 19:30Um I think in terms of timeline it's
  474. 19:32really hard to make estimation into the
  475. 19:33future because a lot of things can
  476. 19:36actually happen in two to three months
  477. 19:37these days.
  478. 19:39Uh there has been a very interesting
  479. 19:41topic RSI,
  480. 19:43uh recursive self-improvement. And it
  481. 19:46has showed very interesting patterns on
  482. 19:48solving open math problems and many
  483. 19:50other tasks. I think with RSI this
  484. 19:53pattern will get accelerated a lot. So,
  485. 19:57let's see.
  486. 20:00Yes.
  487. 20:02>> Can you go back to the
  488. 20:04last slide? Yeah. So, when you when you
  489. 20:06see the improvement on the terminal
  490. 20:08bench for my note,
  491. 20:10uh we can see that you know, GPT-5.6
  492. 20:12sold is sort of doing really bad and
  493. 20:15over the
  494. 20:17period of API cost like as the API cost
  495. 20:20increased, the accuracy increased,
  496. 20:22right? Uh
  497. 20:23do you think
  498. 20:25is it What are the reasons
  499. 20:27for
  500. 20:29you doing better when the API cost
  501. 20:31increase? Also, what are the reasons for
  502. 20:35GPT-6 extra doing much better? Is it
  503. 20:37algorithmic development? Is it like
  504. 20:39bigger model?
  505. 20:41>> Yeah, great question. The question is
  506. 20:42that why when the models get better is
  507. 20:46much more expensive. And also why is
  508. 20:48Astra better than their earlier models.
  509. 20:51So in terms of the price and better
  510. 20:53performance,
  511. 20:54the API cost here is strongly associated
  512. 20:57with like the number of tokens get
  513. 21:00digested in the process. So if you think
  514. 21:02longer, the general pattern is that
  515. 21:05thinking longer, thinking longer is
  516. 21:08actually going to lead to improve the
  517. 21:10performance. So this is just a general
  518. 21:11trend. And then in terms of why models
  519. 21:15are getting better than their previous
  520. 21:17model,
  521. 21:18I think there are many reasons. This is
  522. 21:20also a topic we are going to go deeper
  523. 21:22later in terms of open weights models
  524. 21:25and the openness of large language
  525. 21:26models and the proprietary model
  526. 21:29aspects. The reality is that
  527. 21:32we do not know a lot about how those
  528. 21:34models are trained. Is it about a better
  529. 21:37data, better distillation or other
  530. 21:39aspects? Yeah.
  531. 21:42Okay,
  532. 21:44let's now move to the agent
  533. 21:45architecture.
  534. 21:48So
  535. 21:49I think like if you think about like
  536. 21:52agents, it has a really long history. It
  537. 21:55get really accelerated with large
  538. 21:57language models. And this is going to be
  539. 21:59the agent we are going to talk about.
  540. 22:02And I want to go back to this example
  541. 22:04from React that I showed to you.
  542. 22:06So in this particular example,
  543. 22:09the agent needs to
  544. 22:11think about what they are going to do.
  545. 22:14So this particular task is to put a
  546. 22:16paper shaker in a jar. And then if you
  547. 22:20read the right side
  548. 22:22top to the bottom, Uh, model first the
  549. 22:24model itself is actually a LM core. This
  550. 22:27is the the foundation model uh,
  551. 22:30underneath. And then the thinking step
  552. 22:32here, uh, basically it
  553. 22:34uh, it says things like, "I need to find
  554. 22:36a pepper shaker more likely in
  555. 22:38cabinets." Uh, those are what we call
  556. 22:40planning and the reasoning.
  557. 22:42And then the traces so far,
  558. 22:45everything uh, above the current step is
  559. 22:48what we call memory.
  560. 22:50And then the agent actually take action
  561. 22:52such as go to cabinet one or take a
  562. 22:55pepper shaker. Those are what we call
  563. 22:57uh, tool calls.
  564. 22:59And then those observations, um, those
  565. 23:02are literally just like after each
  566. 23:04action, what the agent uh, sees from the
  567. 23:07environment. Uh, you can see that at
  568. 23:09least there are five components if we
  569. 23:11just look at this very simple
  570. 23:14uh, example from React. So, we will now
  571. 23:16work walk through each of them.
  572. 23:19Um, so
  573. 23:21this is
  574. 23:22uh, meant to be a conceptual framework
  575. 23:25uh, just to highlight the different
  576. 23:27components. And then if you see
  577. 23:28different directions, it actually didn't
  578. 23:30carry a lot of weight.
  579. 23:32Uh, when we talk about agents, uh, I
  580. 23:34want to just share you some kind of like
  581. 23:36core components here. First, agents is
  582. 23:39built on top of large language models.
  583. 23:42So, stronger model in general means
  584. 23:44better agents. This is why we also have
  585. 23:47a debate in terms of should I think
  586. 23:49about training for agents or should I
  587. 23:50just care about the better hardness or
  588. 23:53other
  589. 23:54uh, improvement.
  590. 23:56Um, on top of LM core, there's a very
  591. 23:59important component called the planning
  592. 24:02and the reasoning. So, if you think
  593. 24:03about asking agents to do a very big
  594. 24:06task, they actually need to decompose
  595. 24:08them into smaller pieces. And then maybe
  596. 24:10have sub agent to deal with each of
  597. 24:12them.
  598. 24:13Uh, we may also want to add some
  599. 24:14explicit aspects to improve the
  600. 24:17reasoning here.
  601. 24:19And then as you see from the earlier
  602. 24:20example, we need to introduce a memory
  603. 24:22to make sure that as agents work on the
  604. 24:25task for more than 2 hours, they
  605. 24:27actually know what has happened so far
  606. 24:29and then can use that as context to do
  607. 24:31the reasoning.
  608. 24:34Even in earlier days, uh when model can
  609. 24:36solve very hard math problems, if you
  610. 24:39ask them like um where is CS329Z
  611. 24:43happening today on campus, probably
  612. 24:45models cannot answer that. Um
  613. 24:48yes, our systems also cannot answer
  614. 24:50that.
  615. 24:51Um
  616. 24:52>> [laughter]
  617. 24:53>> So,
  618. 24:54in order to really enable LLMs to do
  619. 24:57stuff, we actually could give them tools
  620. 25:00here. This could be information
  621. 25:02retrieval. If they do not know what's
  622. 25:04the classroom um location, they can
  623. 25:06actually go to Stanford website and then
  624. 25:08search for it. If they do not know the
  625. 25:10weather today at Stanford, they can
  626. 25:12actually search for weather from
  627. 25:14official database.
  628. 25:16Uh similarly, um
  629. 25:18I will show more examples in a few uh
  630. 25:21weeks. Um
  631. 25:22there has been a time period in history
  632. 25:25where a uh this fact that GPT-4 was like
  633. 25:27amazing. It actually cannot solve three
  634. 25:30digits times three digits, just like a
  635. 25:33simple multiplication problem.
  636. 25:35Uh so, there are lots of those
  637. 25:36scenarios. So, tool use really enable
  638. 25:40agents to take factual and uh more
  639. 25:43consistent actions.
  640. 25:46And all of those actually lead to
  641. 25:48different terms that you may heard of,
  642. 25:50uh such as tool call, uh interaction,
  643. 25:53response to users. This is what we call
  644. 25:55like agent actions.
  645. 25:58And I think more importantly, there is
  646. 26:00this like environment part that we
  647. 26:02haven't talked about. So, agent observe
  648. 26:04what's happening in the environment and
  649. 26:06then take actions.
  650. 26:08So, what you see on the screen right
  651. 26:09now, this is actually a skeleton of an
  652. 26:12agent. Uh of course, this is only one
  653. 26:15way to draw it and uh please do not uh
  654. 26:17uh take uh too much weights with this
  655. 26:19kind of directions. This is really meant
  656. 26:21to uh illustrate the process here.
  657. 26:25And then on top of this like a single
  658. 26:26agent, you can imagine that you can have
  659. 26:28different agents. Even within the same
  660. 26:31agent, maybe you can do different
  661. 26:32configuration optimization for each box
  662. 26:35that I listed here. This is why later
  663. 26:38like hardness and optimization become
  664. 26:40very important because
  665. 26:42everyone wants to find the most optimal
  666. 26:45configuration for this agent.
  667. 26:49Um as I said that um this is not the
  668. 26:52only way to talk about agents and our
  669. 26:54breakdown is not a unique. Uh different
  670. 26:56researchers actually use different
  671. 26:58labels. Um for a long time, I think
  672. 27:01there was a lot of debate around what is
  673. 27:03a agent, what is not a agent, what is
  674. 27:06agentic systems, what is not agentic
  675. 27:07systems. We are not going to do any
  676. 27:09debate uh in this class, but I think
  677. 27:12that you can see the common components
  678. 27:14showing up here.
  679. 27:15Uh the left side is from uh a OpenAI
  680. 27:18researcher back in 2023. Uh they talk
  681. 27:21about that, okay, if you think about
  682. 27:22agents, they use tools, they have
  683. 27:25memory, planning, action. And then on
  684. 27:28the right side, this is uh around a
  685. 27:30cognitive language agents. Um you can
  686. 27:33see that they have similar pieces such
  687. 27:35as memory with retrieval to read from it
  688. 27:38and also learning to write it to it,
  689. 27:40reasoning, etc. Despite that was our
  690. 27:43different vocabularies, I think it
  691. 27:45really kind of point to the same
  692. 27:48architecture here. That is an LLM uh
  693. 27:51reason, remember, and use tools, and
  694. 27:54acts in a loop uh within its
  695. 27:56environment.
  696. 27:58So now I'm going to go back to the uh
  697. 28:00diagram of those different components
  698. 28:03and start to share with you different
  699. 28:05modules we have here.
  700. 28:07So if you look at this, I think the
  701. 28:08first thing that it come to mind is
  702. 28:11actually this planning and reasoning
  703. 28:13from this particular figure.
  704. 28:15Um
  705. 28:16So then the question is what does that
  706. 28:19mean for LLMs to reason and why does it
  707. 28:22help agent?
  708. 28:25Um I want to draw some kind of
  709. 28:27uh
  710. 28:28metaphor to human here. Uh Uh this
  711. 28:31action is probably a little bit of
  712. 28:33controversial if you think about that.
  713. 28:34There has been a lot of debate to see
  714. 28:37whether the reasoning of AI is similar
  715. 28:39to human reasoning. Again, no uh debate
  716. 28:43in this course. Um but if you look at
  717. 28:45this process, I think uh for humans, uh
  718. 28:47what does reasoning mean? It means that
  719. 28:50uh some mental process uh such as
  720. 28:52deduction, analogy, and the planning
  721. 28:54that we all do for different task.
  722. 28:58For LLMs, I think uh you can think about
  723. 29:01them as like chain of thought. So, in
  724. 29:03this particular example here, uh most of
  725. 29:05the time we would have called this kind
  726. 29:07of chain of thought as a reasoning
  727. 29:09process.
  728. 29:10Although, one message I want to share
  729. 29:12with you all is that this chain of
  730. 29:15thought may not be faithful of what
  731. 29:18actually happened within LLMs.
  732. 29:22Um for agents, by reasoning we mean the
  733. 29:25internal actions that update the agent's
  734. 29:27own state. So, those are just like a
  735. 29:29different uh terms to understand the
  736. 29:31reasoning and the planning.
  737. 29:33So, now let's look at a concrete
  738. 29:35example.
  739. 29:36So, why is reasoning helpful for agents?
  740. 29:40So, imagine we have a agent that's doing
  741. 29:43actions right now. And at a T step, it
  742. 29:46observed that um you are cooking a dish
  743. 29:49and the scene salt is out.
  744. 29:51And then the next step, if you see a
  745. 29:53action of turn right.
  746. 29:56If you just look at those two,
  747. 29:58I at least I get confused like, okay,
  748. 30:00why why is the agent actually turn right
  749. 30:02to here? Etc. So, the observation to
  750. 30:05action mapping is actually huge. And
  751. 30:08then this can be very hard for
  752. 30:11audits or just like a for uh making
  753. 30:13agents better.
  754. 30:14So, if we could actually introduce a
  755. 30:16reasoning model here to let agents to
  756. 30:19explicitly reason, then this actually
  757. 30:21help a lot. Uh if you read the reasoning
  758. 30:24in the green box, it says that the dish
  759. 30:26should be savory. And since salt is out,
  760. 30:29I should have find something equivalent.
  761. 30:31Uh it is in the cabinet to my right. And
  762. 30:34then when the turn right, we know why
  763. 30:37the agent is doing this particular
  764. 30:38action.
  765. 30:40Um I think a reasoning helps in two
  766. 30:42ways. First is from a generalization
  767. 30:44perspective, because the same reasoning
  768. 30:46could actually work in new similar
  769. 30:49situations.
  770. 30:50It also help with alignment, because you
  771. 30:52can see why the agent did what it did.
  772. 30:56So this is why like reasoning was
  773. 30:58actually a first
  774. 30:59uh important component we are
  775. 31:02emphasizing.
  776. 31:03So now just to understand why systems
  777. 31:06like React beats existing prompting
  778. 31:09methods,
  779. 31:10um I'm going to give you another example
  780. 31:12here.
  781. 31:13Uh this is an example from Hotpot QA. Um
  782. 31:17So um it's asking certain questions, and
  783. 31:20then standard prompting fail. And if you
  784. 31:23just have a model that can reason like
  785. 31:25chain of thought, it also uh fail here.
  786. 31:28Basically, we have wrong answer.
  787. 31:30Um
  788. 31:31This is actually more dangerous, because
  789. 31:33you have a very convincing reasoning
  790. 31:35trace that's wrong.
  791. 31:38So now let's look at if you have this
  792. 31:40kind of a loop of think, taking action,
  793. 31:43think, ob- observation.
  794. 31:46So in this particular example, the model
  795. 31:49thinks that it needs to search Apple
  796. 31:51Remote, since this is in the query, to
  797. 31:53find out what was originally designed to
  798. 31:55interact with Apple Remote.
  799. 31:58Uh such a thought actually induced
  800. 32:00search actions, and it gets a long
  801. 32:02description of Apple Remote from
  802. 32:04Wikipedia.
  803. 32:06In the second step, the model actually
  804. 32:08used reasoning to extract the key
  805. 32:09information from that specific page, and
  806. 32:13uh they found that Apple Remote was
  807. 32:14actually designed to interact with Front
  808. 32:17Row here.
  809. 32:18Um then the model thinks that maybe they
  810. 32:20should search Front Row to find out who
  811. 32:22like what it controls it. But the there
  812. 32:25maybe like the specific Wikipedian page
  813. 32:28does not have such information, but some
  814. 32:30other articles have.
  815. 32:32So this process actually continues. Like
  816. 32:34in the uh third step, the model actually
  817. 32:37used reasoning to recover from
  818. 32:39extraction and then search from her role
  819. 32:42instead.
  820. 32:43So, when the model think that they have
  821. 32:45the correct answer, they actually finish
  822. 32:47the task and they return it.
  823. 32:49So, you can see that here the reasoning
  824. 32:51traces actually induce and adjust agent
  825. 32:54actions, which help agents obtain new
  826. 32:58knowledge to support their own reasoning
  827. 33:00process.
  828. 33:02And then one thing that I want to
  829. 33:03mention is that if you look at this, a
  830. 33:05lot of the trajectory here is very
  831. 33:08intuitive, interpretable, and somewhat
  832. 33:11factual.
  833. 33:13Um to show you how this actually works,
  834. 33:15uh chain of thought is a very uh
  835. 33:17straightforward prompting uh baseline.
  836. 33:20And then you can see that uh compared to
  837. 33:22like just
  838. 33:23uh doing actions, which is act here,
  839. 33:26react actually perform act constantly.
  840. 33:29Um one thing is that uh if you look at
  841. 33:32the failure case from different
  842. 33:34reasoning uh paradigms, um let's do a
  843. 33:37very detailed breakdown. If you compare
  844. 33:39chain of thoughts and the react uh in
  845. 33:42terms of failures, like reasoning error,
  846. 33:45actually react is very high. Um search
  847. 33:47result error, react is also very high
  848. 33:50because uh COT does not have search
  849. 33:52functions.
  850. 33:53Hallucination,
  851. 33:54uh hallucinated the reasoning traces or
  852. 33:56facts, react is zero.
  853. 33:59And then chain of thought is actually 56
  854. 34:02here. So, in general, hallucination is a
  855. 34:04very serious problem for chain of
  856. 34:06thoughts and the many of those prompting
  857. 34:08technique. And then react with this kind
  858. 34:10of reasoning module taking actions
  859. 34:13actually mitigate it a lot.
  860. 34:16So, you probably noticed that there is a
  861. 34:19term here called SC here.
  862. 34:21Uh SC refers to self-consistency.
  863. 34:24Uh this is related to chain of thought
  864. 34:26as a prompting technique and it's a very
  865. 34:29intuitive uh trick you can use.
  866. 34:31Uh so, apparently you can prompt the
  867. 34:33large function model uh using chain of
  868. 34:35thought. And instead of just getting one
  869. 34:38of them, you can actually get a diverse
  870. 34:40set of reasoning choices. You can
  871. 34:42aggregate by choosing the most
  872. 34:44consistent answer in the final answer
  873. 34:46set. So, this is a majority voting to
  874. 34:50some extent, but from a consistency
  875. 34:51perspective.
  876. 34:54So, after self-consistency,
  877. 34:56the
  878. 34:57uh the process of actually making agents
  879. 35:00better at the reasoning um like happens
  880. 35:03very very frequently.
  881. 35:05Uh one of the very notable example is
  882. 35:07called a reflection uh reflection.
  883. 35:10Uh basically, the agents are going to
  884. 35:12reflect through natural language
  885. 35:14feedback.
  886. 35:15Uh if you look at the left figure here,
  887. 35:18so let's see we have a current
  888. 35:19trajectory generated here. This is the
  889. 35:21short-term memory here.
  890. 35:23And then you can give that a to a
  891. 35:25evaluator. The evaluator is going to
  892. 35:27judge is this going to be correct or
  893. 35:30incorrect.
  894. 35:31And then with those feedback, the agent
  895. 35:33is going to do a self-reflection.
  896. 35:36And then with those self-reflected
  897. 35:38content, uh together with the
  898. 35:39experience, they can actually change
  899. 35:42what are the actor is going to do or
  900. 35:44going to see. So, this is a very earlier
  901. 35:47idea of letting models do self-reflect.
  902. 35:51Um
  903. 35:52just to show you that if you look at the
  904. 35:54all the steps after the evaluation, um
  905. 35:57the reasoning sorry, the reflection is
  906. 35:59actually uh quite interesting in terms
  907. 36:01of why the model thinks uh such error
  908. 36:04occurred or they received this
  909. 36:06particular evaluation outcome.
  910. 36:10So, a reflection actually works pretty
  911. 36:12well. Um I think if you reflect on the
  912. 36:14process, it actually reduce errors by
  913. 36:17using like a self-reflection to really
  914. 36:19distill like long field trajectories
  915. 36:22into relevant experience or rubrics or
  916. 36:25reward signals that can be used as
  917. 36:28self-hints in the future.
  918. 36:31So, from the figure, um you can see that
  919. 36:33uh compared to react only, uh with
  920. 36:36reflection, uh it's definitely doing way
  921. 36:38better than the react uh system. And uh
  922. 36:41similarly, in terms of the
  923. 36:43uh I think of this is the uh
  924. 36:45hallucination. You can also see that it
  925. 36:47reduced hallucination a lot with this
  926. 36:49kind of self-reflection technique.
  927. 36:54>> How is this behavior introduced to the
  928. 36:56model? Like is it like
  929. 36:59included as a part of the prompt or
  930. 37:00there's like some fine-tuning or
  931. 37:02training steps to perform into the
  932. 37:04weights?
  933. 37:05>> Yes. Good question. So, reflection and
  934. 37:08many other similar self-reflection
  935. 37:10techniques does not involve fine-tuning
  936. 37:13or any kind of gradient updates. Most of
  937. 37:15them are actually implemented using
  938. 37:19prompt.
  939. 37:20One one minor thing I want to point out
  940. 37:22is that if you read the reflection
  941. 37:24paper, it actually keep using this word
  942. 37:26called the reinforce. But reinforce is
  943. 37:29actually not achieved from a
  944. 37:30reinforcement learning perspective. It's
  945. 37:32only from a self-reflection prompt
  946. 37:34aspect.
  947. 37:36>> In this slide that you showed where you
  948. 37:39compared React with Chain of Thought,
  949. 37:40the results
  950. 37:42hallucination was zero. How is that
  951. 37:44different from reasoning? Like if
  952. 37:46there's a reasoning error and which is
  953. 37:48considerably high, how is the
  954. 37:50hallucination zero?
  955. 37:51>> Yeah, so
  956. 37:52there are going to be multiple figures,
  957. 37:54right? Sometimes you will fail not
  958. 37:56because of hallucination, but because
  959. 37:59such label may have ambiguity or because
  960. 38:02like um
  961. 38:04you need to
  962. 38:05make sense from the current context,
  963. 38:09which is not about like getting the
  964. 38:11right kind of
  965. 38:13tool or information or things like that,
  966. 38:15but more about like interpreting what
  967. 38:17has happened so far. So, they are
  968. 38:19slightly different in this process.
  969. 38:21>> So, what would you say would be
  970. 38:23reasoning error?
  971. 38:24>> Yes, so that would be reasoning error.
  972. 38:26Yeah.
  973. 38:27>> Thank you.
  974. 38:29So, is the main difference between Chain
  975. 38:31of Thought and React in the fact that
  976. 38:33like React and Chain of Thought are
  977. 38:36essentially the same reasoning
  978. 38:38strategies, just Chain of Thought is
  979. 38:39just like
  980. 38:41prompted and there's not much
  981. 38:42difference. Like [clears throat] they
  982. 38:43can't really track its work.
  983. 38:45>> Yeah, so in Chain of thought uh if you
  984. 38:48um we will do a more deep dive into the
  985. 38:51reasoning modules in 2 weeks. Um so in
  986. 38:54chain of thought it's literally just
  987. 38:55like a one prompt, one pass. In uh
  988. 38:58react, you can think about it as a loop.
  989. 39:01So the model think, look at the
  990. 39:03observation, think, take action, and
  991. 39:06then think, look at the observation,
  992. 39:08take action. So it's like a multiple
  993. 39:10step, it's a loop. So in this way, the
  994. 39:12model actually can revise its own errors
  995. 39:15or
  996. 39:16uh getting new information to update
  997. 39:18themselves. So this kind of richness is
  998. 39:20something chain of thought does not
  999. 39:21have.
  1000. 39:24>> That would
  1001. 39:25make you want to decide whether to agree
  1002. 39:27or disagree.
  1003. 39:30>> Okay, maybe in the last one actually,
  1004. 39:31yeah.
  1005. 39:32>> Okay, uh do you think uh the the
  1006. 39:35the
  1007. 39:36process is a distillation of the model
  1008. 39:38itself?
  1009. 39:40Um if if so, is the distillation going
  1010. 39:42to cost uh like like is this reflection
  1011. 39:45going to cost some overfitting to the uh
  1012. 39:48its action?
  1013. 39:50>> Yeah, so uh one thing I want to clarify
  1014. 39:53is that I do use the uh sorry, not this
  1015. 39:55page. I do use the word like this two
  1016. 39:58here, but this is actually not a very
  1017. 40:00classic distillation we are talking
  1018. 40:01about. Here, it's really about you have
  1019. 40:04a lot of like trajectories of what
  1020. 40:06happened so far, right? And then um in
  1021. 40:08general, you may just rely on models to
  1022. 40:10interpret what happened so far, and then
  1023. 40:13like do actions based on that. With
  1024. 40:15self-reflection, the model actually
  1025. 40:17think about that, and then try to
  1026. 40:19compress that into something that's
  1027. 40:21shorter. So in this process, um one word
  1028. 40:25to use is this this two. You can also
  1029. 40:27use it as abstract or synthesize. So
  1030. 40:31this is very different than the classic
  1031. 40:33distillation technique, which is more
  1032. 40:35about uh getting output from other
  1033. 40:37models, and then do training on top of
  1034. 40:39them.
  1035. 40:41Okay, let's uh continue. So in in
  1036. 40:44addition to this kind of like reflection
  1037. 40:46self-consistency, uh another trend that
  1038. 40:48that's very interesting is actually
  1039. 40:50around the multi-agent debate.
  1040. 40:52Um so earlier days, people researchers
  1041. 40:55use multi-agent debate to improve
  1042. 40:58reasoning performance. And this
  1043. 41:00literally means that you are going to
  1044. 41:02have multiple agents doing the same task
  1045. 41:05solving. And then after each round, you
  1046. 41:08let them read each other's reasoning and
  1047. 41:11let them kind of debate just like how
  1048. 41:12you talk about some decisions with your
  1049. 41:15teammates.
  1050. 41:16And then agents can actually like really
  1051. 41:18leverage those diverse perspectives and
  1052. 41:21maybe make some final decision here.
  1053. 41:25I think of this multi-agent technique in
  1054. 41:29earlier days is a very useful
  1055. 41:31prompting method. Um
  1056. 41:34the assumption at that time is that each
  1057. 41:36individual model is not strong enough.
  1058. 41:39So we actually use this kind of
  1059. 41:40multi-agent setup as a way to prompt the
  1060. 41:44model. When models get a very powerful
  1061. 41:46today, if your only goal is to get a
  1062. 41:49performance boost, I think a multi-agent
  1063. 41:51is not going to help you a lot in terms
  1064. 41:53of performance boost. But maybe there
  1065. 41:55are other
  1066. 41:57design choices or design considerations
  1067. 41:59that we can benefit from multi-agent,
  1068. 42:01such as efficiency, such as data access
  1069. 42:04control. Maybe this agent is
  1070. 42:06particularly designed to take care of
  1071. 42:08health care and that agent is designed
  1072. 42:11to help you with sensitive information,
  1073. 42:13etc.
  1074. 42:16So after multi-agent, another way to
  1075. 42:18really kind of like improve
  1076. 42:21this kind of reasoning is actually to
  1077. 42:23set up a many agents and then use a
  1078. 42:26orchestrator to orchestrate the
  1079. 42:28different agents to work towards the
  1080. 42:30same task.
  1081. 42:31So this is a very earlier system from
  1082. 42:33Microsoft called a
  1083. 42:35uh
  1084. 42:36MaginTic here. Basically, if you have a
  1085. 42:38bigger task, the orchestrator is going
  1086. 42:41to break them down into different
  1087. 42:42components. And then you can see that
  1088. 42:44the first part get assigned to
  1089. 42:47file server and then to a coder and then
  1090. 42:49to a computer terminal, etc. And each
  1091. 42:52agent basically take care of each
  1092. 42:54specific task. And then in the end you
  1093. 42:56can actually combine all of them and
  1094. 42:59then provide a final answer to users.
  1095. 43:03So this is about planning and reasoning
  1096. 43:05module. We will go deep [clears throat]
  1097. 43:07into this process in week two and three.
  1098. 43:11The next component that I want to
  1099. 43:12quickly mention is around the memory
  1100. 43:14here.
  1101. 43:16You can actually do a lot of very
  1102. 43:18creative design in terms of how we want
  1103. 43:20to make memory work.
  1104. 43:22Just to briefly share like why we need a
  1105. 43:26memory.
  1106. 43:27Um
  1107. 43:28two reasons. First is this factor our
  1108. 43:31context window for large language models
  1109. 43:33are getting bigger and bigger.
  1110. 43:35Still context window cannot hold all the
  1111. 43:38event stream happens so far. So if you
  1112. 43:41think about long horizon task we are
  1113. 43:43talking about today where agents can
  1114. 43:45actually work on your
  1115. 43:47project for more than three hours.
  1116. 43:49That's actually a lot a lot of data and
  1117. 43:51it's really hard to fit that into
  1118. 43:53context window.
  1119. 43:55The second is that even that's
  1120. 43:58that's possible, it may be really hard
  1121. 44:01for the agent to figure out what is a
  1122. 44:03relevant piece of information I should
  1123. 44:04use. Think about that if you have 1,000
  1124. 44:07documents there, you store them in your
  1125. 44:09context and then basically each time if
  1126. 44:12you need to find the right
  1127. 44:15document, this is actually very kind of
  1128. 44:17time consuming since you have such a big
  1129. 44:20search space.
  1130. 44:21So the bottom is actually very
  1131. 44:24interesting design from the generative
  1132. 44:26agents, the Small Ville project where
  1133. 44:29they were trying to build agents that
  1134. 44:31can simulate people living in a small
  1135. 44:33town. And then in their design, they
  1136. 44:36actually need to use the memory stream
  1137. 44:38to make sure that we can keep track of
  1138. 44:41what happened to each individual so that
  1139. 44:43the story can continue.
  1140. 44:46There are different type of memories you
  1141. 44:49can design.
  1142. 44:50If you think about
  1143. 44:52what kind of things to store?
  1144. 44:54Episodic memory here literally refers to
  1145. 44:57we store everything like raw logs, every
  1146. 45:00sentence the agent has shared with
  1147. 45:02users, every
  1148. 45:04sentence that the users have provided to
  1149. 45:06agents.
  1150. 45:07Semantic memory means that instead of
  1151. 45:10thinking about this kind of raw logs, we
  1152. 45:13may want to
  1153. 45:15remember like semantic events since
  1154. 45:17those may be more useful.
  1155. 45:20Procedural memory you store skills.
  1156. 45:23Instead of like all the specificity, you
  1157. 45:26remember like what is the skill, what is
  1158. 45:28a procedural of how we do certain type
  1159. 45:30of things.
  1160. 45:32So if you go deep into this process like
  1161. 45:35this is actually example of episodic
  1162. 45:37memory. You can see that literally we
  1163. 45:39are storing every like single time step
  1164. 45:42and then what happened at that time
  1165. 45:44step.
  1166. 45:46With semantic memory, we actually can do
  1167. 45:48some synthesize of what happened over
  1168. 45:51the agent trajectories and then
  1169. 45:53literally kind of like have this like a
  1170. 45:55small summary box as a way to like to
  1171. 45:59remember what has happened in the last
  1172. 46:01several minutes. So this is what we mean
  1173. 46:03by semantic memory.
  1174. 46:06Procedural memory you can actually check
  1175. 46:08out this Voyager paper from Nvidia where
  1176. 46:11they were trying to teach agents to play
  1177. 46:14I think this Minecraft here
  1178. 46:17where if the agent figure out how to
  1179. 46:20use
  1180. 46:21new skill or new tool, you actually want
  1181. 46:23to abstract that into a skill and store
  1182. 46:26it in your skill library. And then
  1183. 46:27whenever you have a new task, you want
  1184. 46:29to see like do I have a skill that I can
  1185. 46:32use from a skill library? If so, then
  1186. 46:34I'm going to retrieve it and use it. So
  1187. 46:36this is more from a procedural
  1188. 46:38perspective rather than like all the
  1189. 46:40logs that have happened so far.
  1190. 46:44So this is about the memory and then
  1191. 46:46another very important thing is about
  1192. 46:48the two use here.
  1193. 46:50So two use we will talk about this next
  1194. 46:53Wednesday. But just to give you a quick
  1195. 46:56highlight of what this means.
  1196. 46:58Um, we all know that right now you can
  1197. 47:00actually enable agents to use different
  1198. 47:03type of tools such as
  1199. 47:05uh search, such as
  1200. 47:07um coding agent, etc. So, in earlier
  1201. 47:10days, um the tool use is actually
  1202. 47:13triggered in the actual decoding
  1203. 47:15process. So, if you read the right part,
  1204. 47:18um the sentence it maybe let's just read
  1205. 47:22the first one. Uh the New England
  1206. 47:23Journal of Medicine is a registered
  1207. 47:26trademark of. So, if you need to do like
  1208. 47:29some kind of um like information
  1209. 47:31generation, and at this this point uh we
  1210. 47:34may want to trigger uh special token
  1211. 47:37called a QA, and then the QA literally
  1212. 47:40is who is the publisher of this uh
  1213. 47:43Journal of Medicine, and then you got it
  1214. 47:45the specific answer, and then you use
  1215. 47:47that answer in the process of
  1216. 47:49generation. So, in other words, uh
  1217. 47:52earlier when we want to teach model how
  1218. 47:54to use tools, we are actually learning
  1219. 47:57like
  1220. 47:58like where to call uh API, when to call
  1221. 48:02it, which API to call, etc.
  1222. 48:05So, Toolformer was actually the first uh
  1223. 48:08uh ish work that it does this kind of
  1224. 48:10teaching LLMs to use tools.
  1225. 48:13Um you probably uh can enumerate uh how
  1226. 48:17many tools they can use. At that time, I
  1227. 48:19think there are only five tools that we
  1228. 48:21teach to agents. Agents can use a
  1229. 48:24calculator here. They can use a question
  1230. 48:26answering system. They can use search
  1231. 48:28engine. They can use a translation
  1232. 48:29system. They can uh use calendar. So,
  1233. 48:32that's a very limited set of tools.
  1234. 48:35Today, I think uh our agents can
  1235. 48:38actually use a wider range of tools, but
  1236. 48:40they actually all get inspired on top of
  1237. 48:43uh setups like this.
  1238. 48:46More recently, um Gorilla from Berkeley
  1239. 48:50actually uh tried to teach LLMs to use
  1240. 48:52like solid ends of APIs.
  1241. 48:55Um so, what they did is that they
  1242. 48:57actually uh look at hugging face, uh
  1243. 49:00torch hub, and also tensor flow hub,
  1244. 49:03where there are like over 1,600 APIs.
  1245. 49:06And then they basically generate the
  1246. 49:08synthetic data from that, and then trend
  1247. 49:10a 7 billion parameter model on top of
  1248. 49:13that.
  1249. 49:14Um at the inference time, if we look at
  1250. 49:16this example, if the user says that I
  1251. 49:19want to see some cats dancing in
  1252. 49:22celebration,
  1253. 49:23um then you are going to do a retrieval
  1254. 49:25here. Probably you are going to use like
  1255. 49:27a stable diffusion, and then you get
  1256. 49:29this like a image out here.
  1257. 49:32Otherwise, if you think about this, like
  1258. 49:34the base model needs to have the
  1259. 49:36capability to understand this, and then
  1260. 49:38give this kind of like output of the
  1261. 49:40image. But now we can actually delegate
  1262. 49:43that to a external tool, such as stable
  1263. 49:45diffusion. Then we can get the task done
  1264. 49:48without the requiring the underneath
  1265. 49:50model to even um do this kind of image
  1266. 49:53generation.
  1267. 49:55Um with Gorilla, I think that the
  1268. 49:57results are quite impressive. So, if you
  1269. 49:59look at the reduction to hallucination,
  1270. 50:02uh it's actually pretty good. Like very
  1271. 50:05few hallucinations, and a pretty high
  1272. 50:07accuracy.
  1273. 50:08Um I think this actually makes sense,
  1274. 50:10because without the so many tools like
  1275. 50:13use or like understanding, you actually
  1276. 50:16have to rely on the model themselves to
  1277. 50:18do this task. But now you can actually
  1278. 50:20do more factual and uh reasonable
  1279. 50:23generation.
  1280. 50:25So, this is around the tools. Uh we will
  1281. 50:27go deep into that next Wednesday.
  1282. 50:31So, if you put everything together, uh
  1283. 50:33this is actually the skeleton of an
  1284. 50:35agent. Uh LLM core that reason and a
  1285. 50:38plan. The memory actually carries states
  1286. 50:41um across steps and different task. Um
  1287. 50:44tools to reach out to diverse APIs, um
  1288. 50:48available tools we have, and also an
  1289. 50:50environment that agents can actually
  1290. 50:52take actions and observe um states from.
  1291. 50:56So, every agent that you are going to
  1292. 50:58build, uh whether it's going to write
  1293. 51:00code, browse the web, or like runs the
  1294. 51:02terminal, it's some kind of version of
  1295. 51:05this diagram.
  1296. 51:06And what differs is actually how each
  1297. 51:09box is implemented, and how well the
  1298. 51:12pieces like fit together. So, that's
  1299. 51:14going to be a lot of the key aspects we
  1300. 51:16are going to cover later.
  1301. 51:20Um any quick questions until this point?
  1302. 51:28Um so, this is actually related to the
  1303. 51:30memory design.
  1304. 51:32Um
  1305. 51:33if you just think about like how to
  1306. 51:35design memory systems, instead of
  1307. 51:37letting them to decide like to decide
  1308. 51:40what to memorize, the episodic memory
  1309. 51:42literally just tell them to remember
  1310. 51:44everything.
  1311. 51:45So, everything happened so far. So, this
  1312. 51:47is more from a heuristic perspective.
  1313. 51:50And semantic memory, you may need to
  1314. 51:51have some separate module to teach them
  1315. 51:54that instead of like um instead of like
  1316. 51:58remember everything, you may want to
  1317. 51:59remember important information. Then you
  1318. 52:02can actually give demonstrations of what
  1319. 52:03is important information. What is the
  1320. 52:06format you want? What is the like um
  1321. 52:09format you want to even store in your
  1322. 52:10memory. So, those are the design choices
  1323. 52:12we are going to bring to the process. Um
  1324. 52:15this is also a place where things can
  1325. 52:18get a little bit tricky.
  1326. 52:20Um there was this example I think Do I
  1327. 52:23have it today? I don't have it on my
  1328. 52:25slides today, but like earlier this year
  1329. 52:27uh
  1330. 52:28some people are using Open Claw um to
  1331. 52:31manage their email. And then all of
  1332. 52:33sudden, Open Claw actually started to
  1333. 52:35delete this users' like email. Despite
  1334. 52:38the user says, "Please stop. Uh please
  1335. 52:40do not delete my email." It is still
  1336. 52:42kind of continuous. Um the user actually
  1337. 52:45specified in the very beginning that uh
  1338. 52:47whenever you do such actions, please
  1339. 52:49confirm with me. But then it's just like
  1340. 52:51so interesting like, "Okay, why that
  1341. 52:53happens?" And why the model actually is
  1342. 52:54not uh even listening to instructions.
  1343. 52:58It turns out that this is actually a
  1344. 52:59failure in the context memory
  1345. 53:01compression stage. Because you want to
  1346. 53:04make sure the memory is stored in a very
  1347. 53:06efficient way. So, you like the model
  1348. 53:08the model decided to just like keep
  1349. 53:09track of all important information.
  1350. 53:12Confirm before you do action is not
  1351. 53:15perceived as important from the agent's
  1352. 53:17perspective. So, this actually lead to a
  1353. 53:20lot of the email deletion. So, I think
  1354. 53:22this is a a very good question. It's
  1355. 53:24going to be a very critical design of
  1356. 53:26how we make sure the memory is faithful
  1357. 53:29and also can enable very reasonable
  1358. 53:31inductions.
  1359. 53:34Okay.
  1360. 53:36Um
  1361. 53:37if there's no more question, we will uh
  1362. 53:39continue.
  1363. 53:41Um so, next I want to talk about uh
  1364. 53:44compound AI systems. Uh we just like put
  1365. 53:46together different pieces. Um but now
  1366. 53:49like um
  1367. 53:51if you think about this like how real
  1368. 53:53products actually combine uh LLM with
  1369. 53:56other components.
  1370. 53:57Um this is what we call component AI
  1371. 53:59systems. I'm going to briefly talk about
  1372. 54:01different models here, but we have uh
  1373. 54:04specific lectures to go deeper there.
  1374. 54:07Um I think uh one thing people often get
  1375. 54:10uh
  1376. 54:11overwhelmed is that when we think about
  1377. 54:13agents, there seems to be a lot of
  1378. 54:15components happening. There are LLM
  1379. 54:17core, there are planning, memory, tool
  1380. 54:19use, environment, etc. And uh to keep
  1381. 54:23all of those like a manageable, um it
  1382. 54:25helps to think about agent engineering
  1383. 54:28as some kind of more principled ways of
  1384. 54:30understanding.
  1385. 54:32So, we are going to divide this into
  1386. 54:34three layers, and our course is also set
  1387. 54:37up in similar order.
  1388. 54:39Uh first, I think that this is the
  1389. 54:40system layer. This is the architecture,
  1390. 54:43which components agent has and how they
  1391. 54:45connect. Um this is just everything that
  1392. 54:48I have covered so far.
  1393. 54:49The second layer is the data.
  1394. 54:52That's how the agent actually learns
  1395. 54:54from or is prompt with. Uh here you will
  1396. 54:57have training examples to train agent
  1397. 55:00such as what we will cover in Swiss
  1398. 55:02Miss.
  1399. 55:03Um demonstration of the task or maybe
  1400. 55:06just the context we give to agents.
  1401. 55:09The third layer here is the evaluation.
  1402. 55:11That's how we actually evaluate whether
  1403. 55:13the agent actually works. And it the
  1404. 55:16also includes the metric itself.
  1405. 55:19Choosing the wrong metric can make a
  1406. 55:21weak agent look nice.
  1407. 55:24So the metric actually deserves a lot of
  1408. 55:26care when we think about this.
  1409. 55:29So most of what we are going to cover in
  1410. 55:31this course will be mapped into those
  1411. 55:33three layers. And the one agent is is
  1412. 55:36not doing well
  1413. 55:37or your agent is not like working as
  1414. 55:39expected. I think a good first question
  1415. 55:42to ask is which layer the problem may
  1416. 55:45occur. Is it in the system? Is it in the
  1417. 55:47data? Or is it in the evaluation? This
  1418. 55:49is also a guiding principle for us to
  1419. 55:51debug like how to make the agents
  1420. 55:54better.
  1421. 55:55So let's just quickly go through the
  1422. 55:57system layer here.
  1423. 56:00A key design choice at the system layer
  1424. 56:02is who decide the steps. Like who
  1425. 56:05decides like memory since like that. Is
  1426. 56:08it going to be
  1427. 56:10um
  1428. 56:10like the agents or is it going to be
  1429. 56:12something else? Um and I want to kind of
  1430. 56:14like a make this differentiation here of
  1431. 56:17workflow versus agents. Uh both what
  1432. 56:21people refer to as compound AI systems.
  1433. 56:24That is a large language model plus
  1434. 56:26other components. You may encounter
  1435. 56:28those two terms in different contexts.
  1436. 56:30So here a workflow is going to be
  1437. 56:34maybe like it's just going to fix
  1438. 56:37certain steps.
  1439. 56:38Um examples such as agent lists which
  1440. 56:41fix a GitHub issue in a very fixed set
  1441. 56:45of stage like finding the file, writing
  1442. 56:48patch, and doing test etc. So the large
  1443. 56:52language model works inside each stage
  1444. 56:54but they never pick the next one. So
  1445. 56:56it's actually a script. It's like a
  1446. 56:58fixed order of how they do stuff.
  1447. 57:01In the agent deciding the large language
  1448. 57:03model decide for itself. Um for example
  1449. 57:07in sweet agent it actually choose which
  1450. 57:09files to open, what it to run, and when
  1451. 57:12it's done. So, you can see that it's
  1452. 57:14actually more flexible. It also gives
  1453. 57:17more responsibility to the model
  1454. 57:19underneath it.
  1455. 57:22So, workflow, uh now like if I just like
  1456. 57:24stop here, you probably think about
  1457. 57:26like, "Okay, then why do we even need a
  1458. 57:28workflow here?"
  1459. 57:30Um so, I want to just share one example
  1460. 57:32here that is workflow agent, sorry,
  1461. 57:35workflow uh system can actually be very
  1462. 57:38very strong.
  1463. 57:39So, on my screen, this is actually
  1464. 57:41AlphaCode 2. It actually generates up to
  1465. 57:44like millions of candidate programs, and
  1466. 57:46then test and rank them. And then it
  1467. 57:49reach almost like uh 87% of human
  1468. 57:54competitors.
  1469. 57:55Um and this is a huge jump from like
  1470. 57:57AlphaCode 1.
  1471. 58:00The next uh common workflow pattern, um
  1472. 58:03I think uh has been widely used in
  1473. 58:05industry in the past, is called the
  1474. 58:07retrieval augmented generation, RAG.
  1475. 58:10Um this has a lot of sense to do with
  1476. 58:12like um
  1477. 58:13open question answering.
  1478. 58:15Um basically, RAG connects large
  1479. 58:18language models to external knowledge
  1480. 58:19source in real time. So, if you have a
  1481. 58:22specific query um that may rely on the
  1482. 58:25weather API or weather website, then you
  1483. 58:28go there, uh read the related content,
  1484. 58:31use that to uh augment your current
  1485. 58:33query as context. And then on top of
  1486. 58:36that, do generation to give users more
  1487. 58:40factual and consistent answers.
  1488. 58:44Um what actually happens in a single
  1489. 58:47tool call? I mean, previously I
  1490. 58:48mentioned about the uh like algorithms.
  1491. 58:52Um if you think about like what actually
  1492. 58:53happens from a system perspective, there
  1493. 58:56are actually multiple stage. Uh first,
  1494. 58:59the model needs to select a tool from
  1495. 59:01the schemas it has from the context.
  1496. 59:04And then arguments. It needs to fill in
  1497. 59:07all the type of field from the task.
  1498. 59:10Validation means that those arguments
  1499. 59:12need to be checked against the schema.
  1500. 59:15Sometimes there may be like enforcement
  1501. 59:17with constrained decoding so that the
  1502. 59:19model actually cannot produce a
  1503. 59:21ill-formed call here.
  1504. 59:23Final step is execution. The tool get
  1505. 59:26around and then you really in a sandbox.
  1506. 59:29We'll go to sandbox in a few minutes.
  1507. 59:32And then results go back to the context.
  1508. 59:35And then the the example in the bottom
  1509. 59:37actually shows you a specific tool call.
  1510. 59:39So if the user question is what does
  1511. 59:42test.py contain? And you can see that
  1512. 59:45the model actually read file. And then
  1513. 59:48fitting in the path information. And
  1514. 59:50then the path is checked as a string.
  1515. 59:53And then the file content actually come
  1516. 59:55back from the model for users to read.
  1517. 59:57So this is actually what's happening
  1518. 59:59from a execution perspective.
  1519. 1:00:03I think
  1520. 1:00:04the reality is that if every AI
  1521. 1:00:06application had to write a very custom
  1522. 1:00:09integrations for every tool, then
  1523. 1:00:11nothing will really like scale up.
  1524. 1:00:14The model context protocol MCP
  1525. 1:00:17is actually a standard protocol to
  1526. 1:00:20make sure such connection can happen.
  1527. 1:00:22So on the left side you can see that we
  1528. 1:00:24have different AI applications
  1529. 1:00:27such as Cloud Desk, Desktop, IDE, and
  1530. 1:00:30the code editors such as Cloud Code.
  1531. 1:00:33On the right those are different data
  1532. 1:00:35sources and tools. Database such as SQL,
  1533. 1:00:39Google Drive. Deployment tools or even
  1534. 1:00:42like productivity tools such as Slack,
  1535. 1:00:44Google Maps, etc.
  1536. 1:00:46So MCP sits in the middle as a standard
  1537. 1:00:50protocol with data flowing in both
  1538. 1:00:52directions.
  1539. 1:00:54You can then think about how to build a
  1540. 1:00:56tool once MCP server and any kind of MCP
  1541. 1:01:00client
  1542. 1:01:01can be set up and use it.
  1543. 1:01:04We will go back to this next class.
  1544. 1:01:08So in the
  1545. 1:01:09next 5 minutes I'm going to quickly
  1546. 1:01:11mention about key challenges here.
  1547. 1:01:13Just to share with you that this factor
  1548. 1:01:16agent have been very impressive. I think
  1549. 1:01:18that they also create a lot of um
  1550. 1:01:21emerging challenges for us to be aware
  1551. 1:01:24of.
  1552. 1:01:25So, to give you a very quick overview of
  1553. 1:01:28what we are going to cover, uh
  1554. 1:01:29reliability.
  1555. 1:01:31If you build agents, you know that error
  1556. 1:01:34there is going to be a lot of like error
  1557. 1:01:36uh compound cascade.
  1558. 1:01:38Um to true agents, the reward signal
  1559. 1:01:41sometimes can be very sparse. Uh it's
  1560. 1:01:43usually like long horizon task. Very
  1561. 1:01:46very expensive to do roll out.
  1562. 1:01:48The long horizon also give us a lot of
  1563. 1:01:51challenge in terms of context uh
  1564. 1:01:53increase.
  1565. 1:01:54Uh safety is a big topic. Also like
  1566. 1:01:56evaluation is very very hard. It's
  1567. 1:01:58probably like I think it's one of the
  1568. 1:02:00undervalued aspect when we think about
  1569. 1:02:03agents.
  1570. 1:02:04I want to give you a few quick examples.
  1571. 1:02:06So, uh reliability versus evaluation.
  1572. 1:02:10This is actually a effort from
  1573. 1:02:11Princeton. Um they were really trying to
  1574. 1:02:14understand that that is the accuracy and
  1575. 1:02:16the performance we are seeing a truthful
  1576. 1:02:19reflection of what's happening.
  1577. 1:02:21And they found the three dimensions such
  1578. 1:02:23as consistency. If you let the agent run
  1579. 1:02:26multiple times, do they give you the
  1580. 1:02:28same outcome?
  1581. 1:02:30Robustness. What if we add a little bit
  1582. 1:02:32of perturbation? How do agents actually
  1583. 1:02:34respond to those like a small,
  1584. 1:02:36realistic, and even irrelevant
  1585. 1:02:38information?
  1586. 1:02:39Uh predictable. Uh when agents fail to
  1587. 1:02:42do something, do they fail because um
  1588. 1:02:45some like unpredictable reasons or can
  1589. 1:02:48we actually understand the it? Like
  1590. 1:02:51legible and recoverable.
  1591. 1:02:53Uh just to show you some of those
  1592. 1:02:55numbers. Like if you look at the
  1593. 1:02:56accuracy, this is actually ranked in
  1594. 1:02:58terms of models.
  1595. 1:03:00But if you look at the reliability, um
  1596. 1:03:03those scores do not have a trend.
  1597. 1:03:06Which means that there is a gap between
  1598. 1:03:09capability and the reliability. The most
  1599. 1:03:12powerful agent may not be the most
  1600. 1:03:15reliable agent. And this is something we
  1601. 1:03:17need to keep in mind when we are doing
  1602. 1:03:18evaluations. So, it's not only just
  1603. 1:03:20about the model performance,
  1604. 1:03:22reliability, efficiency.
  1605. 1:03:25Similar arguments are hold when you
  1606. 1:03:27think about like hardness audit. Uh
  1607. 1:03:29recent work showed that if you like
  1608. 1:03:32audit those agents, uh the green bar
  1609. 1:03:35here refers to the task completion. And
  1610. 1:03:38then the color bar refers to the
  1611. 1:03:40hardness safety. And you can see that
  1612. 1:03:42the safety dimension is just way lower
  1613. 1:03:44compared to the task completion.
  1614. 1:03:48Um there is a very interesting example
  1615. 1:03:50that I also want to share with you. So,
  1616. 1:03:52we talk about agents. Um they they do
  1617. 1:03:55stuff. Um but they may also have issues.
  1618. 1:03:59This is a example from Operator, which
  1619. 1:04:01is a computer use agent introduced by
  1620. 1:04:04OpenAI in January of 2025. Uh the user
  1621. 1:04:07said that my most frustrating experience
  1622. 1:04:10with Operator was my first one, trying
  1623. 1:04:12to order groceries. Uh the user said,
  1624. 1:04:14"Help me buy groceries at Instacart."
  1625. 1:04:16Expecting it to ask some basic questions
  1626. 1:04:19such as where do I live? What a store do
  1627. 1:04:21I usually buy groceries from? What kind
  1628. 1:04:23of things do I want?" It didn't ask me
  1629. 1:04:25any of that. Instead, Operator opened
  1630. 1:04:27the Instacart in the browser and began
  1631. 1:04:29searching for milk in grocery stores
  1632. 1:04:31located in Iowa.
  1633. 1:04:33So, think about this. Like, the agents
  1634. 1:04:36can actually navigate the web. They can
  1635. 1:04:39They can use your credit card to make
  1636. 1:04:41purchase. But, in this process, they are
  1637. 1:04:44making a lot of assumption. And they
  1638. 1:04:46didn't really have this kind of
  1639. 1:04:47awareness to collaborate with people.
  1640. 1:04:50So, this is actually like a failure from
  1641. 1:04:52the interaction side.
  1642. 1:04:55Uh the next thing I want to show you is
  1643. 1:04:57that uh the robustness of agents is
  1644. 1:04:59actually going to face a lot of
  1645. 1:05:01challenge when they enter our real world
  1646. 1:05:04because we just have so many like dark
  1647. 1:05:05patterns.
  1648. 1:05:07Uh in this particular example, um
  1649. 1:05:09researchers actually just pop up this
  1650. 1:05:12kind of like I mean, we have a lot of
  1651. 1:05:13pop-ups on different websites. So, we
  1652. 1:05:15pop up this kind of black box. Nothing
  1653. 1:05:18uh like nothing adversary or malicious
  1654. 1:05:21is there. But then, because of this
  1655. 1:05:23distraction, agents actually fail to
  1656. 1:05:25complete their task 87% of the time.
  1657. 1:05:30This is what we call a adversary attack.
  1658. 1:05:32And this is really like interesting.
  1659. 1:05:34Like, if your agent goes to do shopping
  1660. 1:05:36for you, and then a pop-up from the
  1661. 1:05:38website actually could get your agents
  1662. 1:05:40distracted, and they cannot finish the
  1663. 1:05:42original task.
  1664. 1:05:44Um, the other example that I want to
  1665. 1:05:46show you here is actually from a privacy
  1666. 1:05:48and a security perspective. So, um, this
  1667. 1:05:51is an agent that's supposed to write a,
  1668. 1:05:54um, weekly report for workers. It
  1669. 1:05:56actually did a great job of, like, how
  1670. 1:05:57many lines of code, uh, you wrote, how
  1671. 1:05:59many features you launched. It also
  1672. 1:06:01mentioned that you had a lunch with
  1673. 1:06:03recruiter from a different company.
  1674. 1:06:06So, this actually highlighted something
  1675. 1:06:09that's very critical. That is, privacy
  1676. 1:06:12today. We give information, all our
  1677. 1:06:14information, to agents, to AI systems.
  1678. 1:06:17They know everything about us. They
  1679. 1:06:19probably know that the PI information
  1680. 1:06:21not something they should have used. But
  1681. 1:06:22a lot of the sensitive personal
  1682. 1:06:24informations can really be
  1683. 1:06:26consequential. And there's no way to
  1684. 1:06:28teach agents about how to handle
  1685. 1:06:30different access control.
  1686. 1:06:34Um, this is another kind of, like,
  1687. 1:06:36important topic we will go deep later.
  1688. 1:06:39Um, misalignment and the syncophasy.
  1689. 1:06:41Um, if you look at the left side, this
  1690. 1:06:43is actually called a insincere, um,
  1691. 1:06:46uh, syncophasy. So, um, basically the
  1692. 1:06:49agent think of your, your writing is
  1693. 1:06:51terrible. But, uh, they do not want to
  1694. 1:06:53hurt your feelings. Instead, the agent
  1695. 1:06:55says that they'll give it a five out of
  1696. 1:06:57five. So, this is also a reminder that
  1697. 1:06:59you should not only just give your
  1698. 1:07:01homework to agents, because they will
  1699. 1:07:02say great things about it without really
  1700. 1:07:04giving you feedback.
  1701. 1:07:06Um, the second part is actually about a
  1702. 1:07:08model hacks its own code. So, um, you
  1703. 1:07:12the model is supposed to help the user,
  1704. 1:07:14like, check, like, reverse the learning.
  1705. 1:07:16And then, the agent, uh, internal
  1706. 1:07:18thinking is that, "Oh, I saw there is
  1707. 1:07:20this unit test, which, um, can check the
  1708. 1:07:23reward function, and I cannot modify
  1709. 1:07:25that, but I could change the unit test
  1710. 1:07:27first. So, they change everything here
  1711. 1:07:30so that everything looks good on the
  1712. 1:07:32surface.
  1713. 1:07:34Um I think the probably the most uh
  1714. 1:07:36surprising thing, like very close to us,
  1715. 1:07:39is what happened like in July, uh around
  1716. 1:07:42this kind of open AI hacking face
  1717. 1:07:44incident.
  1718. 1:07:45Um
  1719. 1:07:45and this is actually a very interesting
  1720. 1:07:48demonstration of how we think about like
  1721. 1:07:50agent safety moving forward. So, what
  1722. 1:07:53happens is that uh um
  1723. 1:07:55open AI has this kind of like sandbox
  1724. 1:07:56agent, and they are asked to do a
  1725. 1:07:59impossible uh task. They get it stuck,
  1726. 1:08:03and then agent start to explore its
  1727. 1:08:05environment and looking for other ways
  1728. 1:08:07to maybe get a higher reward or cheat on
  1729. 1:08:10the task. Um they found out that there
  1730. 1:08:12is actually a message board where over a
  1731. 1:08:15thousand agent collaborated to cheat on
  1732. 1:08:17their task. And then they just join uh
  1733. 1:08:20with each other, and then um basically a
  1734. 1:08:23lot of things have happened.
  1735. 1:08:25Um just to show you what happened. Um
  1736. 1:08:28I think like within a few days, uh it
  1737. 1:08:30actually sent over 17,000 attacks to
  1738. 1:08:33hacking face uh infrastructure. Here, if
  1739. 1:08:36you showed um like the blue color refers
  1740. 1:08:39to intended like uh uh solution. And
  1741. 1:08:42then you can see the replaced target,
  1742. 1:08:44which is more about like a manipulation.
  1743. 1:08:46Understand the manipulated uh error, I'm
  1744. 1:08:48sorry, logs. And then attack like a
  1745. 1:08:50hacking face. You can see the volume of
  1746. 1:08:53what happened in the process.
  1747. 1:08:55So, I think uh this is indeed a very
  1748. 1:08:57concerning topic for us to kind of
  1749. 1:08:59reflect, especially given that agents
  1750. 1:09:02can now function for so many hours, and
  1751. 1:09:05maybe even longer.
  1752. 1:09:08So, uh with that, this is a brief
  1753. 1:09:10overview of the uh introduction to
  1754. 1:09:13agentic uh AI. I think like we did a
  1755. 1:09:16history of agents, and then different
  1756. 1:09:19components of agent architectures. Uh we
  1757. 1:09:22talk about the compound AI systems, and
  1758. 1:09:24a lot of key challenge we are going to
  1759. 1:09:26go through. Uh with that, thank you all
  1760. 1:09:28and see you next class.

About this transcript

This page contains the full transcript of YouTube transcript (DGI4vucR71w) , generated from the public captions YouTube serves with the video. The transcript has 10,838 words across 1,760 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.