YouTube transcript (DGI4vucR71w) — Transcript
Full transcript
- 0:05Welcome everyone. This is CS 329Z,
- 0:09engineering AI agents.
- 0:12And today is the introduction to agentic
- 0:13systems.
- 0:15I'm D Young, assistant professor in
- 0:16computer science here. And I'm
- 0:18co-teaching this course with Michael
- 0:21Ryan and John Young.
- 0:23Today we will cover logistics and also
- 0:26give you a short history of agents,
- 0:28what a agent is made of, and also the
- 0:31open problems we will work on this
- 0:32quarter.
- 0:34I will first like to Michael and John
- 0:36introduce themselves.
- 0:38>> Hi everyone. I'm Michael. I'm a
- 0:40second-year PhD student here at Stanford
- 0:42and I'm advised by D and Percy.
- 0:45So my research is on open-source
- 0:48human-centered AI. And in the past
- 0:50agent-related work I've worked on the
- 0:52DSPy framework for
- 0:55programming, not prompting language
- 0:57models. So that's my little claim to
- 1:00agents.
- 1:01>> Hi, my name is John. I'm a third-year
- 1:03PhD at Stanford. I'm advised by D. And
- 1:06in the past I've worked on coding agents
- 1:09and coding evaluation. So I created a
- 1:11SweetBench, SweetAgent, ProgramBench,
- 1:14SweetSmith, if you've heard of those
- 1:16works.
- 1:18>> Great. Okay, so
- 1:21with the instructor team, I want to
- 1:24formally welcome you to this course.
- 1:26The primary contact to reach out to us
- 1:28is to use at discussion. If you want to
- 1:31be added to it, just send us an email.
- 1:33We prefer you send everything to this
- 1:37email list so that we can take care of
- 1:39your request very quickly.
- 1:41The website will maintain all the
- 1:42lectures, slides, and homeworks. So
- 1:45that's your primary website to check
- 1:48out.
- 1:49If you haven't enrolled in at
- 1:50discussion, please do since we post a
- 1:52lot of announcements there as well.
- 1:55So in today's lecture, Um, in addition
- 1:58to the logistics, I will mention in a
- 2:00few seconds. We're going to provide a
- 2:02history of agents. Uh, this is There is
- 2:05actually a really long history behind
- 2:07agents.
- 2:08Uh, we will talk about agent
- 2:10architecture. This is the core content
- 2:12today. And then compound AI systems, rag
- 2:15to use, and MCP. Uh, we will end with a
- 2:18brief highlight of key challenges,
- 2:20including the recent multi-agent risk
- 2:23and collusion we have been seeing.
- 2:26So, let's get it started.
- 2:28Uh, why we create the CS 329Z? Uh,
- 2:32so, when we think about large language
- 2:34models, agents these days, there seems
- 2:37to be very different perspectives
- 2:38involved. You can take a modeling
- 2:41perspective. You can also take a system
- 2:43perspective. A lot of what we do is
- 2:45actually around the engineering.
- 2:48And if you think about that, LM people,
- 2:50uh,
- 2:51I think they know how models are
- 2:53trained, pre-training, post-training,
- 2:55prompting, uh, alignment, and
- 2:57evaluation. On the other side, system
- 3:00people know how to build reliable tools,
- 3:02systems,
- 3:04um, more related to agents. We have
- 3:06tools, APIs, environment, and
- 3:08infrastructure.
- 3:10So, the expectation for this course is
- 3:12that we're going to cover a lot of the
- 3:15foundations. But more importantly, we
- 3:17want it to be more hands-on building
- 3:19plus evaluation of agents.
- 3:23A quick glance of the content. So, uh,
- 3:26week one to week five, uh, we will talk
- 3:28about large language models for
- 3:30builders, um, with a highlight of
- 3:33retrieval augmented generation and its
- 3:35variants in many, many follow-up agent
- 3:38architectures.
- 3:39Tool use and a function call will be
- 3:42another highlight here. We will slowly
- 3:44move to agent design, training, and
- 3:46scaffolding. And also the design of
- 3:49agent memory, uh, as well as
- 3:51orchestration and multi-agent systems.
- 3:55Um, the second half of this course, we
- 3:57are going to have optimization, harness,
- 4:00and data for agentic systems, data
- 4:03quality, data selection, etc.
- 4:05The evaluation and benchmark design will
- 4:08also be highlighted in different ways.
- 4:11Agent safety and the guardrail
- 4:13will also have two special highlights of
- 4:16coding agent and also a more emerging
- 4:20topic called a proactive agents.
- 4:24We have actually two special teams who
- 4:28are going to show up in our class this
- 4:30quarter.
- 4:31First, we will welcome the terminal
- 4:33bench team to talk about how they design
- 4:37terminal bench and a lot of questions
- 4:39you can share with them.
- 4:41We will also welcome the SweetBench team
- 4:43to come to chat with us. So, for both
- 4:47lectures, it will be a team reunion from
- 4:50those two teams that have created a very
- 4:52fundamental evaluations in our field.
- 4:55Course policy.
- 4:57First, I think
- 4:59the biggest message that I want to
- 5:00highlight is that
- 5:02you should use AI to learn.
- 5:06You may want to get yourself familiar
- 5:08with the Stanford Honor Code. And laptop
- 5:11use is only allowed for taking notes,
- 5:14etc. And we strongly discourage the use
- 5:17during the lecture so that you can be
- 5:20more focused.
- 5:21Prerequisite and the compute.
- 5:24So, first,
- 5:25we strongly recommend that you have
- 5:27taken any NLP or large language model
- 5:30related training.
- 5:32You are expected to know basic LLMs
- 5:34since we are not going to cover like
- 5:36fine-tuning or pre-training or
- 5:38alignment, etc.
- 5:40In terms of compute, we are very
- 5:42grateful
- 5:43that Lot Institute gives each student
- 5:46$200 in terms of credits. And Thinking
- 5:49Machine will provide a $250
- 5:52for Tinker credits.
- 5:55I want to switch a little bit to kind of
- 5:57conclude the logistics section by
- 5:59showing you a figure.
- 6:01This is a
- 6:03just like a specific instance of what's
- 6:05happening today.
- 6:08With a lot of data, if you show on the
- 6:10top, this is actually homework score
- 6:13from two populations. People who use AI
- 6:16and the people who do not use AI.
- 6:19Very interestingly, I think with AI,
- 6:22everyone's homework score are just so
- 6:24great. But then if you look at the
- 6:26bottom, this is actually the exam score
- 6:28for the same population.
- 6:30Before using AI,
- 6:32this is definitely more kind of shifted
- 6:34towards 100 here.
- 6:36With AI, we have way more people did
- 6:39worse in their exams.
- 6:41And I think we see this kind of pattern
- 6:43very frequent these days.
- 6:46The second example that I want to share
- 6:48with you is actually from Anthropic.
- 6:51They did a study with their own
- 6:53engineers to figure out how the use of
- 6:56coding agent actually impact coding.
- 6:59As you can see from the left side, if
- 7:02you use AI, the task completion time
- 7:05actually decreased.
- 7:06But then on the right, the quiz score in
- 7:09terms of your mastery actually
- 7:11significantly decreased. That is,
- 7:13despite the coding agent that can give
- 7:15us 1,000 line of code, it's a little bit
- 7:18unclear whether we fully understand such
- 7:21AI output.
- 7:22So, with that, I want to conclude the
- 7:25logistics by saying that we really hope
- 7:27that you can use AI to learn how to
- 7:29learn. And this is really just about you
- 7:32and your CS 329Z.
- 7:35With that, any quick questions for
- 7:37logistics?
- 7:41Okay. If there's no question, I will
- 7:43continue to the history of agents.
- 7:46One thing that I do want to mention is
- 7:48that
- 7:49the learning objective today is for you
- 7:52all to get a familiar with agent loop on
- 7:54top of large language models and the
- 7:56different key components.
- 7:59So, let's get it started.
- 8:01This is a word that's probably very very
- 8:04popular today.
- 8:05And uh I'm going to do next is to trace
- 8:08where this idea actually came from, how
- 8:11it developed over the decades, and why
- 8:13it suddenly take off with large language
- 8:16models.
- 8:18Um if you look at the uh more linguistic
- 8:21side of agents, uh the word agent
- 8:24actually comes from Latin, uh meaning
- 8:26that one uh who acts. And uh this term
- 8:30also uh entered English in the later um
- 8:3414th century. So, in AI, the first index
- 8:37use of it is in this specific IJCAI
- 8:40paper from 19 uh '73, uh where Hewitt
- 8:44and Bishop and Steiger talk about this
- 8:47architecture called actor algorithm,
- 8:50where each actor is an active agent that
- 8:53wait for a message and then does what
- 8:55the message tell them to do. So, the
- 8:57idea of software entities that act on
- 9:00their own is older than most people can
- 9:04think about.
- 9:06And then, I think like a few years after
- 9:08that, um the Contract Net Protocol
- 9:12actually talk about how agents dividing
- 9:14task with each other and then
- 9:16negotiating task.
- 9:19This trend actually continues. In Marvin
- 9:21Minsky's um Society of Minds, it
- 9:24actually talk about how a group of sub
- 9:26agents actually work together to get it
- 9:28something done. And if you kind of
- 9:31reflect on this, this is actually the
- 9:33earlier idea of multi-agent
- 9:35orchestration.
- 9:38I think the history actually continues.
- 9:40Around the 1990s, we actually see more
- 9:43and more emergence of uh software
- 9:46systems that uh serve as agents. Uh from
- 9:49Pattie's designing autonomous agents to
- 9:52later um
- 9:54people talk about uh agent as autonomy,
- 9:57uh social ability, reactivity,
- 9:59proactivity, trying to define agents in
- 10:02different ways.
- 10:03The most uh familiar um
- 10:05definition to many of us is actually
- 10:08from Russell and Norvig, uh which
- 10:10defines agents as anything that they
- 10:13perceive through sensors and act through
- 10:17actuators here.
- 10:20We now kind of like enter a different
- 10:22stage, which we call reinforcement
- 10:24learning error here.
- 10:26This is a classic agent environment loop
- 10:29that many of us get familiar during RL
- 10:32courses. So, what happened here is that
- 10:35agent take action in the environment and
- 10:38such action will
- 10:40receive a reward. And then they do the
- 10:42action based on specific state and the
- 10:44current reward. So, this is a very
- 10:47classic agent environment loop from
- 10:49Sutton here.
- 10:50And then I think like we see more and
- 10:53more promising from promises from this
- 10:56kind of setup.
- 10:58DQN refers to deep
- 11:01Q network. This actually from DeepMind's
- 11:04earlier days, where they build
- 11:07AI systems to actually solve specific
- 11:10games.
- 11:12Now, I think a later
- 11:13you also see that
- 11:15AlphaGo and OpenAI 5 showing that this
- 11:19kind of approach actually can reach
- 11:21superhuman performance
- 11:23in like Go and I think this one is Dota
- 11:252.
- 11:26So, those were agents, but each of them
- 11:29are actually trained for a specific
- 11:31task.
- 11:34So, what actually changed is
- 11:36introduction of large language models.
- 11:38I think the first kind of impressive
- 11:41capability of LLMs is actually in
- 11:43context learning.
- 11:45Built upon this kind of
- 11:47transformer-based architectures,
- 11:49GPT-3 actually showed that you can just
- 11:52talk about a task and give you a few
- 11:54demonstrations. Like in this particular
- 11:57example of few-shot, in order to do
- 12:00translation, you can offer certain
- 12:02examples and then the model can actually
- 12:04follow such examples and then do the
- 12:07task. This entire process, unlike For
- 12:10systems I showed you, there is no
- 12:12gradient update happening here.
- 12:18I think like not only like in terms of
- 12:20prompting or in context learning,
- 12:23prompting could do more
- 12:25than teach a model a new task. It turned
- 12:28out that models themselves, they can
- 12:30also reason step-by-step and then use
- 12:33tools for the things that they cannot do
- 12:36alone.
- 12:37So this particular example is called a
- 12:39chain of thought prompting. Uh you can
- 12:41see that asking the model to actually
- 12:43work through a problem step-by-step
- 12:46actually give more accurate answers in
- 12:48general.
- 12:50Uh WebGPT was proposed by OpenAI. This
- 12:54actually enables language model in
- 12:56earlier days to do to use like web
- 12:59browser so that it can actually look up
- 13:01things online instead of only rely on
- 13:04memory or parameters.
- 13:06Um this actually leaded to a lot a lot
- 13:09of follow-up work in terms of like web
- 13:11browser agent.
- 13:14Um one of the very interesting
- 13:16architecture is actually called a
- 13:18modular reasoning knowledge and the
- 13:20language.
- 13:21I would view it as one of the first
- 13:23earlier systems to think about to use.
- 13:27So basically this architecture actually
- 13:29decides which outside expert to send a
- 13:32request to. Is this going to be a
- 13:35weather request? So then I should use a
- 13:37weather API.
- 13:39If this is a coding task, maybe I should
- 13:41send it to a Python programming tool
- 13:44etc. So now the model actually can think
- 13:48and it can take actions through tools.
- 13:53So I think we now have like both
- 13:55important pieces here, that is models
- 13:57can reason step-by-step and the models
- 14:00that can use tools. Those are two very
- 14:02very important a prerequisite for the
- 14:05emergence of agents. I'm going to show
- 14:08you a very short timeline in what
- 14:10happened in history, that is 2 years
- 14:12ago.
- 14:13So
- 14:142022 React was
- 14:17proposed
- 14:19Um, so the name actually means reason
- 14:22plus act. So, instead of thinking once
- 14:25and then answering the question, what
- 14:27the model does here is actually a loop.
- 14:30It thinks about what to do next, take
- 14:33action, and then look at what happened,
- 14:36and then use that to think again. So,
- 14:38it's actually a loop happening here.
- 14:42After that, I think we see that uh more
- 14:45than more uh models get released. So,
- 14:48ChatGPT get a launched, and suddenly I
- 14:50think a millions of people start to talk
- 14:53to LLMs every day.
- 14:56And then GPT-4 was released. And the
- 14:58reason why I want to mention this is
- 15:00that if you think about agents, they
- 15:02really rely on the base model, the
- 15:04strong capability from those base
- 15:06models. So, when GPT-4 came out, we have
- 15:09a much stronger model that it could uh
- 15:11follow long instructions and plan more
- 15:15reliably.
- 15:17Um, within 2 weeks, there were actually
- 15:19two open-source projects came out that
- 15:22really shape what we think around
- 15:24agents. The first one is called the Baby
- 15:27AGI.
- 15:28Um,
- 15:29it basically gave the model a goal and
- 15:31let it write uh its own task list, walk
- 15:35through the task, and add a new ones as
- 15:38it goes.
- 15:40Uh, AutoGPT was released uh um I think
- 15:43in the same uh months, 2 days later. Um,
- 15:47basically try to connect the GPT-4 to
- 15:49tools such as web search, file systems,
- 15:52etc. And then let it uh keep uh going on
- 15:55on its own without a person really like
- 15:58typing the next prompt each time.
- 16:01I think uh like now if you look back, it
- 16:03was definitely very ahead of time ideas.
- 16:06The reality is that none of them
- 16:08actually really worked very well.
- 16:11Um, but they got the people's attention
- 16:13and really opened our imagination about
- 16:16what's possible.
- 16:17Um, I think that that's also probably
- 16:20the time when the word agent really
- 16:22enters like everyday use.
- 16:25So um
- 16:26I want to kind of go a little bit deeper
- 16:28here to share with you like the
- 16:30underlying model and also agents, how
- 16:33they have improved since 2023.
- 16:37Um so this page, uh if you look at that,
- 16:40I only show the GPT-3, GPT-4, and GPT-5.
- 16:43Um the underneath model actually is
- 16:47better and better.
- 16:49If you just contrast the GPT-3 versus
- 16:51GPT-4, we see that on MMLU, there's a
- 16:5443% increase. This is actually true for
- 16:58all the data sets that we have.
- 17:01From GPT-4 to GPT-5, some of those data
- 17:03set actually didn't show up here because
- 17:05they got solved or they saturated so
- 17:08that they cannot be used as very
- 17:10reliable signals.
- 17:12Instead, we actually introduced more
- 17:15evaluation such as GPQA, which is a very
- 17:18challenging QA setting, and also harder
- 17:22and harder math problems here.
- 17:25So the key message here is that each
- 17:27generation of base model actually made a
- 17:30really big jump in terms of their
- 17:32performance.
- 17:34And a stronger model means that the
- 17:36agent now can actually reason better.
- 17:38They can make fewer mistakes,
- 17:41hallucinate less,
- 17:43um and along the way they can handle
- 17:44longer and longer task.
- 17:47And then let's look at the agent side.
- 17:50So the agent side is also improving
- 17:54as well, not only just like the raw
- 17:56model capability.
- 17:57Uh so this page is actually around like
- 18:00SweetBench verified. Um as you can see
- 18:03uh in January 2025, I think uh the
- 18:06performance is only around a 30% but
- 18:09then over the last uh
- 18:12um 1 year and a half, now that all the
- 18:15numbers are actually reached a very high
- 18:17stage here.
- 18:20And then right side of this is from
- 18:21Astro release. So, you can see that OS
- 18:24world, which is a computer used
- 18:26evaluation. And also on terminal bench,
- 18:29which is a more like a command line
- 18:30task.
- 18:31Um the accuracy are just impressive.
- 18:35There's also a trend that people start
- 18:37to pay attention to the efficiency of
- 18:39the process. This is why you are seeing
- 18:41that most of those curves these days
- 18:43actually have both price and accuracy
- 18:46there together.
- 18:48So, this is how we got here. Um it's a
- 18:50long history in the last 16 years.
- 18:54Um so, what I'm going to do next is to
- 18:55really open one app and then see what
- 18:58our agent is actually made of.
- 19:03Any quick questions here?
- 19:07Okay. Um yes.
- 19:10>> Do you think models
- 19:12capacity or ability is going to saturate
- 19:14in in a few years or is going to still
- 19:17like develop for
- 19:19like 10 years or so?
- 19:22>> I think the
- 19:23first uh just to repeat the question.
- 19:25The question is will model capabilities
- 19:27saturate or will it keep improving?
- 19:30Um I think in terms of timeline it's
- 19:32really hard to make estimation into the
- 19:33future because a lot of things can
- 19:36actually happen in two to three months
- 19:37these days.
- 19:39Uh there has been a very interesting
- 19:41topic RSI,
- 19:43uh recursive self-improvement. And it
- 19:46has showed very interesting patterns on
- 19:48solving open math problems and many
- 19:50other tasks. I think with RSI this
- 19:53pattern will get accelerated a lot. So,
- 19:57let's see.
- 20:00Yes.
- 20:02>> Can you go back to the
- 20:04last slide? Yeah. So, when you when you
- 20:06see the improvement on the terminal
- 20:08bench for my note,
- 20:10uh we can see that you know, GPT-5.6
- 20:12sold is sort of doing really bad and
- 20:15over the
- 20:17period of API cost like as the API cost
- 20:20increased, the accuracy increased,
- 20:22right? Uh
- 20:23do you think
- 20:25is it What are the reasons
- 20:27for
- 20:29you doing better when the API cost
- 20:31increase? Also, what are the reasons for
- 20:35GPT-6 extra doing much better? Is it
- 20:37algorithmic development? Is it like
- 20:39bigger model?
- 20:41>> Yeah, great question. The question is
- 20:42that why when the models get better is
- 20:46much more expensive. And also why is
- 20:48Astra better than their earlier models.
- 20:51So in terms of the price and better
- 20:53performance,
- 20:54the API cost here is strongly associated
- 20:57with like the number of tokens get
- 21:00digested in the process. So if you think
- 21:02longer, the general pattern is that
- 21:05thinking longer, thinking longer is
- 21:08actually going to lead to improve the
- 21:10performance. So this is just a general
- 21:11trend. And then in terms of why models
- 21:15are getting better than their previous
- 21:17model,
- 21:18I think there are many reasons. This is
- 21:20also a topic we are going to go deeper
- 21:22later in terms of open weights models
- 21:25and the openness of large language
- 21:26models and the proprietary model
- 21:29aspects. The reality is that
- 21:32we do not know a lot about how those
- 21:34models are trained. Is it about a better
- 21:37data, better distillation or other
- 21:39aspects? Yeah.
- 21:42Okay,
- 21:44let's now move to the agent
- 21:45architecture.
- 21:48So
- 21:49I think like if you think about like
- 21:52agents, it has a really long history. It
- 21:55get really accelerated with large
- 21:57language models. And this is going to be
- 21:59the agent we are going to talk about.
- 22:02And I want to go back to this example
- 22:04from React that I showed to you.
- 22:06So in this particular example,
- 22:09the agent needs to
- 22:11think about what they are going to do.
- 22:14So this particular task is to put a
- 22:16paper shaker in a jar. And then if you
- 22:20read the right side
- 22:22top to the bottom, Uh, model first the
- 22:24model itself is actually a LM core. This
- 22:27is the the foundation model uh,
- 22:30underneath. And then the thinking step
- 22:32here, uh, basically it
- 22:34uh, it says things like, "I need to find
- 22:36a pepper shaker more likely in
- 22:38cabinets." Uh, those are what we call
- 22:40planning and the reasoning.
- 22:42And then the traces so far,
- 22:45everything uh, above the current step is
- 22:48what we call memory.
- 22:50And then the agent actually take action
- 22:52such as go to cabinet one or take a
- 22:55pepper shaker. Those are what we call
- 22:57uh, tool calls.
- 22:59And then those observations, um, those
- 23:02are literally just like after each
- 23:04action, what the agent uh, sees from the
- 23:07environment. Uh, you can see that at
- 23:09least there are five components if we
- 23:11just look at this very simple
- 23:14uh, example from React. So, we will now
- 23:16work walk through each of them.
- 23:19Um, so
- 23:21this is
- 23:22uh, meant to be a conceptual framework
- 23:25uh, just to highlight the different
- 23:27components. And then if you see
- 23:28different directions, it actually didn't
- 23:30carry a lot of weight.
- 23:32Uh, when we talk about agents, uh, I
- 23:34want to just share you some kind of like
- 23:36core components here. First, agents is
- 23:39built on top of large language models.
- 23:42So, stronger model in general means
- 23:44better agents. This is why we also have
- 23:47a debate in terms of should I think
- 23:49about training for agents or should I
- 23:50just care about the better hardness or
- 23:53other
- 23:54uh, improvement.
- 23:56Um, on top of LM core, there's a very
- 23:59important component called the planning
- 24:02and the reasoning. So, if you think
- 24:03about asking agents to do a very big
- 24:06task, they actually need to decompose
- 24:08them into smaller pieces. And then maybe
- 24:10have sub agent to deal with each of
- 24:12them.
- 24:13Uh, we may also want to add some
- 24:14explicit aspects to improve the
- 24:17reasoning here.
- 24:19And then as you see from the earlier
- 24:20example, we need to introduce a memory
- 24:22to make sure that as agents work on the
- 24:25task for more than 2 hours, they
- 24:27actually know what has happened so far
- 24:29and then can use that as context to do
- 24:31the reasoning.
- 24:34Even in earlier days, uh when model can
- 24:36solve very hard math problems, if you
- 24:39ask them like um where is CS329Z
- 24:43happening today on campus, probably
- 24:45models cannot answer that. Um
- 24:48yes, our systems also cannot answer
- 24:50that.
- 24:51Um
- 24:52>> [laughter]
- 24:53>> So,
- 24:54in order to really enable LLMs to do
- 24:57stuff, we actually could give them tools
- 25:00here. This could be information
- 25:02retrieval. If they do not know what's
- 25:04the classroom um location, they can
- 25:06actually go to Stanford website and then
- 25:08search for it. If they do not know the
- 25:10weather today at Stanford, they can
- 25:12actually search for weather from
- 25:14official database.
- 25:16Uh similarly, um
- 25:18I will show more examples in a few uh
- 25:21weeks. Um
- 25:22there has been a time period in history
- 25:25where a uh this fact that GPT-4 was like
- 25:27amazing. It actually cannot solve three
- 25:30digits times three digits, just like a
- 25:33simple multiplication problem.
- 25:35Uh so, there are lots of those
- 25:36scenarios. So, tool use really enable
- 25:40agents to take factual and uh more
- 25:43consistent actions.
- 25:46And all of those actually lead to
- 25:48different terms that you may heard of,
- 25:50uh such as tool call, uh interaction,
- 25:53response to users. This is what we call
- 25:55like agent actions.
- 25:58And I think more importantly, there is
- 26:00this like environment part that we
- 26:02haven't talked about. So, agent observe
- 26:04what's happening in the environment and
- 26:06then take actions.
- 26:08So, what you see on the screen right
- 26:09now, this is actually a skeleton of an
- 26:12agent. Uh of course, this is only one
- 26:15way to draw it and uh please do not uh
- 26:17uh take uh too much weights with this
- 26:19kind of directions. This is really meant
- 26:21to uh illustrate the process here.
- 26:25And then on top of this like a single
- 26:26agent, you can imagine that you can have
- 26:28different agents. Even within the same
- 26:31agent, maybe you can do different
- 26:32configuration optimization for each box
- 26:35that I listed here. This is why later
- 26:38like hardness and optimization become
- 26:40very important because
- 26:42everyone wants to find the most optimal
- 26:45configuration for this agent.
- 26:49Um as I said that um this is not the
- 26:52only way to talk about agents and our
- 26:54breakdown is not a unique. Uh different
- 26:56researchers actually use different
- 26:58labels. Um for a long time, I think
- 27:01there was a lot of debate around what is
- 27:03a agent, what is not a agent, what is
- 27:06agentic systems, what is not agentic
- 27:07systems. We are not going to do any
- 27:09debate uh in this class, but I think
- 27:12that you can see the common components
- 27:14showing up here.
- 27:15Uh the left side is from uh a OpenAI
- 27:18researcher back in 2023. Uh they talk
- 27:21about that, okay, if you think about
- 27:22agents, they use tools, they have
- 27:25memory, planning, action. And then on
- 27:28the right side, this is uh around a
- 27:30cognitive language agents. Um you can
- 27:33see that they have similar pieces such
- 27:35as memory with retrieval to read from it
- 27:38and also learning to write it to it,
- 27:40reasoning, etc. Despite that was our
- 27:43different vocabularies, I think it
- 27:45really kind of point to the same
- 27:48architecture here. That is an LLM uh
- 27:51reason, remember, and use tools, and
- 27:54acts in a loop uh within its
- 27:56environment.
- 27:58So now I'm going to go back to the uh
- 28:00diagram of those different components
- 28:03and start to share with you different
- 28:05modules we have here.
- 28:07So if you look at this, I think the
- 28:08first thing that it come to mind is
- 28:11actually this planning and reasoning
- 28:13from this particular figure.
- 28:15Um
- 28:16So then the question is what does that
- 28:19mean for LLMs to reason and why does it
- 28:22help agent?
- 28:25Um I want to draw some kind of
- 28:27uh
- 28:28metaphor to human here. Uh Uh this
- 28:31action is probably a little bit of
- 28:33controversial if you think about that.
- 28:34There has been a lot of debate to see
- 28:37whether the reasoning of AI is similar
- 28:39to human reasoning. Again, no uh debate
- 28:43in this course. Um but if you look at
- 28:45this process, I think uh for humans, uh
- 28:47what does reasoning mean? It means that
- 28:50uh some mental process uh such as
- 28:52deduction, analogy, and the planning
- 28:54that we all do for different task.
- 28:58For LLMs, I think uh you can think about
- 29:01them as like chain of thought. So, in
- 29:03this particular example here, uh most of
- 29:05the time we would have called this kind
- 29:07of chain of thought as a reasoning
- 29:09process.
- 29:10Although, one message I want to share
- 29:12with you all is that this chain of
- 29:15thought may not be faithful of what
- 29:18actually happened within LLMs.
- 29:22Um for agents, by reasoning we mean the
- 29:25internal actions that update the agent's
- 29:27own state. So, those are just like a
- 29:29different uh terms to understand the
- 29:31reasoning and the planning.
- 29:33So, now let's look at a concrete
- 29:35example.
- 29:36So, why is reasoning helpful for agents?
- 29:40So, imagine we have a agent that's doing
- 29:43actions right now. And at a T step, it
- 29:46observed that um you are cooking a dish
- 29:49and the scene salt is out.
- 29:51And then the next step, if you see a
- 29:53action of turn right.
- 29:56If you just look at those two,
- 29:58I at least I get confused like, okay,
- 30:00why why is the agent actually turn right
- 30:02to here? Etc. So, the observation to
- 30:05action mapping is actually huge. And
- 30:08then this can be very hard for
- 30:11audits or just like a for uh making
- 30:13agents better.
- 30:14So, if we could actually introduce a
- 30:16reasoning model here to let agents to
- 30:19explicitly reason, then this actually
- 30:21help a lot. Uh if you read the reasoning
- 30:24in the green box, it says that the dish
- 30:26should be savory. And since salt is out,
- 30:29I should have find something equivalent.
- 30:31Uh it is in the cabinet to my right. And
- 30:34then when the turn right, we know why
- 30:37the agent is doing this particular
- 30:38action.
- 30:40Um I think a reasoning helps in two
- 30:42ways. First is from a generalization
- 30:44perspective, because the same reasoning
- 30:46could actually work in new similar
- 30:49situations.
- 30:50It also help with alignment, because you
- 30:52can see why the agent did what it did.
- 30:56So this is why like reasoning was
- 30:58actually a first
- 30:59uh important component we are
- 31:02emphasizing.
- 31:03So now just to understand why systems
- 31:06like React beats existing prompting
- 31:09methods,
- 31:10um I'm going to give you another example
- 31:12here.
- 31:13Uh this is an example from Hotpot QA. Um
- 31:17So um it's asking certain questions, and
- 31:20then standard prompting fail. And if you
- 31:23just have a model that can reason like
- 31:25chain of thought, it also uh fail here.
- 31:28Basically, we have wrong answer.
- 31:30Um
- 31:31This is actually more dangerous, because
- 31:33you have a very convincing reasoning
- 31:35trace that's wrong.
- 31:38So now let's look at if you have this
- 31:40kind of a loop of think, taking action,
- 31:43think, ob- observation.
- 31:46So in this particular example, the model
- 31:49thinks that it needs to search Apple
- 31:51Remote, since this is in the query, to
- 31:53find out what was originally designed to
- 31:55interact with Apple Remote.
- 31:58Uh such a thought actually induced
- 32:00search actions, and it gets a long
- 32:02description of Apple Remote from
- 32:04Wikipedia.
- 32:06In the second step, the model actually
- 32:08used reasoning to extract the key
- 32:09information from that specific page, and
- 32:13uh they found that Apple Remote was
- 32:14actually designed to interact with Front
- 32:17Row here.
- 32:18Um then the model thinks that maybe they
- 32:20should search Front Row to find out who
- 32:22like what it controls it. But the there
- 32:25maybe like the specific Wikipedian page
- 32:28does not have such information, but some
- 32:30other articles have.
- 32:32So this process actually continues. Like
- 32:34in the uh third step, the model actually
- 32:37used reasoning to recover from
- 32:39extraction and then search from her role
- 32:42instead.
- 32:43So, when the model think that they have
- 32:45the correct answer, they actually finish
- 32:47the task and they return it.
- 32:49So, you can see that here the reasoning
- 32:51traces actually induce and adjust agent
- 32:54actions, which help agents obtain new
- 32:58knowledge to support their own reasoning
- 33:00process.
- 33:02And then one thing that I want to
- 33:03mention is that if you look at this, a
- 33:05lot of the trajectory here is very
- 33:08intuitive, interpretable, and somewhat
- 33:11factual.
- 33:13Um to show you how this actually works,
- 33:15uh chain of thought is a very uh
- 33:17straightforward prompting uh baseline.
- 33:20And then you can see that uh compared to
- 33:22like just
- 33:23uh doing actions, which is act here,
- 33:26react actually perform act constantly.
- 33:29Um one thing is that uh if you look at
- 33:32the failure case from different
- 33:34reasoning uh paradigms, um let's do a
- 33:37very detailed breakdown. If you compare
- 33:39chain of thoughts and the react uh in
- 33:42terms of failures, like reasoning error,
- 33:45actually react is very high. Um search
- 33:47result error, react is also very high
- 33:50because uh COT does not have search
- 33:52functions.
- 33:53Hallucination,
- 33:54uh hallucinated the reasoning traces or
- 33:56facts, react is zero.
- 33:59And then chain of thought is actually 56
- 34:02here. So, in general, hallucination is a
- 34:04very serious problem for chain of
- 34:06thoughts and the many of those prompting
- 34:08technique. And then react with this kind
- 34:10of reasoning module taking actions
- 34:13actually mitigate it a lot.
- 34:16So, you probably noticed that there is a
- 34:19term here called SC here.
- 34:21Uh SC refers to self-consistency.
- 34:24Uh this is related to chain of thought
- 34:26as a prompting technique and it's a very
- 34:29intuitive uh trick you can use.
- 34:31Uh so, apparently you can prompt the
- 34:33large function model uh using chain of
- 34:35thought. And instead of just getting one
- 34:38of them, you can actually get a diverse
- 34:40set of reasoning choices. You can
- 34:42aggregate by choosing the most
- 34:44consistent answer in the final answer
- 34:46set. So, this is a majority voting to
- 34:50some extent, but from a consistency
- 34:51perspective.
- 34:54So, after self-consistency,
- 34:56the
- 34:57uh the process of actually making agents
- 35:00better at the reasoning um like happens
- 35:03very very frequently.
- 35:05Uh one of the very notable example is
- 35:07called a reflection uh reflection.
- 35:10Uh basically, the agents are going to
- 35:12reflect through natural language
- 35:14feedback.
- 35:15Uh if you look at the left figure here,
- 35:18so let's see we have a current
- 35:19trajectory generated here. This is the
- 35:21short-term memory here.
- 35:23And then you can give that a to a
- 35:25evaluator. The evaluator is going to
- 35:27judge is this going to be correct or
- 35:30incorrect.
- 35:31And then with those feedback, the agent
- 35:33is going to do a self-reflection.
- 35:36And then with those self-reflected
- 35:38content, uh together with the
- 35:39experience, they can actually change
- 35:42what are the actor is going to do or
- 35:44going to see. So, this is a very earlier
- 35:47idea of letting models do self-reflect.
- 35:51Um
- 35:52just to show you that if you look at the
- 35:54all the steps after the evaluation, um
- 35:57the reasoning sorry, the reflection is
- 35:59actually uh quite interesting in terms
- 36:01of why the model thinks uh such error
- 36:04occurred or they received this
- 36:06particular evaluation outcome.
- 36:10So, a reflection actually works pretty
- 36:12well. Um I think if you reflect on the
- 36:14process, it actually reduce errors by
- 36:17using like a self-reflection to really
- 36:19distill like long field trajectories
- 36:22into relevant experience or rubrics or
- 36:25reward signals that can be used as
- 36:28self-hints in the future.
- 36:31So, from the figure, um you can see that
- 36:33uh compared to react only, uh with
- 36:36reflection, uh it's definitely doing way
- 36:38better than the react uh system. And uh
- 36:41similarly, in terms of the
- 36:43uh I think of this is the uh
- 36:45hallucination. You can also see that it
- 36:47reduced hallucination a lot with this
- 36:49kind of self-reflection technique.
- 36:54>> How is this behavior introduced to the
- 36:56model? Like is it like
- 36:59included as a part of the prompt or
- 37:00there's like some fine-tuning or
- 37:02training steps to perform into the
- 37:04weights?
- 37:05>> Yes. Good question. So, reflection and
- 37:08many other similar self-reflection
- 37:10techniques does not involve fine-tuning
- 37:13or any kind of gradient updates. Most of
- 37:15them are actually implemented using
- 37:19prompt.
- 37:20One one minor thing I want to point out
- 37:22is that if you read the reflection
- 37:24paper, it actually keep using this word
- 37:26called the reinforce. But reinforce is
- 37:29actually not achieved from a
- 37:30reinforcement learning perspective. It's
- 37:32only from a self-reflection prompt
- 37:34aspect.
- 37:36>> In this slide that you showed where you
- 37:39compared React with Chain of Thought,
- 37:40the results
- 37:42hallucination was zero. How is that
- 37:44different from reasoning? Like if
- 37:46there's a reasoning error and which is
- 37:48considerably high, how is the
- 37:50hallucination zero?
- 37:51>> Yeah, so
- 37:52there are going to be multiple figures,
- 37:54right? Sometimes you will fail not
- 37:56because of hallucination, but because
- 37:59such label may have ambiguity or because
- 38:02like um
- 38:04you need to
- 38:05make sense from the current context,
- 38:09which is not about like getting the
- 38:11right kind of
- 38:13tool or information or things like that,
- 38:15but more about like interpreting what
- 38:17has happened so far. So, they are
- 38:19slightly different in this process.
- 38:21>> So, what would you say would be
- 38:23reasoning error?
- 38:24>> Yes, so that would be reasoning error.
- 38:26Yeah.
- 38:27>> Thank you.
- 38:29So, is the main difference between Chain
- 38:31of Thought and React in the fact that
- 38:33like React and Chain of Thought are
- 38:36essentially the same reasoning
- 38:38strategies, just Chain of Thought is
- 38:39just like
- 38:41prompted and there's not much
- 38:42difference. Like [clears throat] they
- 38:43can't really track its work.
- 38:45>> Yeah, so in Chain of thought uh if you
- 38:48um we will do a more deep dive into the
- 38:51reasoning modules in 2 weeks. Um so in
- 38:54chain of thought it's literally just
- 38:55like a one prompt, one pass. In uh
- 38:58react, you can think about it as a loop.
- 39:01So the model think, look at the
- 39:03observation, think, take action, and
- 39:06then think, look at the observation,
- 39:08take action. So it's like a multiple
- 39:10step, it's a loop. So in this way, the
- 39:12model actually can revise its own errors
- 39:15or
- 39:16uh getting new information to update
- 39:18themselves. So this kind of richness is
- 39:20something chain of thought does not
- 39:21have.
- 39:24>> That would
- 39:25make you want to decide whether to agree
- 39:27or disagree.
- 39:30>> Okay, maybe in the last one actually,
- 39:31yeah.
- 39:32>> Okay, uh do you think uh the the
- 39:35the
- 39:36process is a distillation of the model
- 39:38itself?
- 39:40Um if if so, is the distillation going
- 39:42to cost uh like like is this reflection
- 39:45going to cost some overfitting to the uh
- 39:48its action?
- 39:50>> Yeah, so uh one thing I want to clarify
- 39:53is that I do use the uh sorry, not this
- 39:55page. I do use the word like this two
- 39:58here, but this is actually not a very
- 40:00classic distillation we are talking
- 40:01about. Here, it's really about you have
- 40:04a lot of like trajectories of what
- 40:06happened so far, right? And then um in
- 40:08general, you may just rely on models to
- 40:10interpret what happened so far, and then
- 40:13like do actions based on that. With
- 40:15self-reflection, the model actually
- 40:17think about that, and then try to
- 40:19compress that into something that's
- 40:21shorter. So in this process, um one word
- 40:25to use is this this two. You can also
- 40:27use it as abstract or synthesize. So
- 40:31this is very different than the classic
- 40:33distillation technique, which is more
- 40:35about uh getting output from other
- 40:37models, and then do training on top of
- 40:39them.
- 40:41Okay, let's uh continue. So in in
- 40:44addition to this kind of like reflection
- 40:46self-consistency, uh another trend that
- 40:48that's very interesting is actually
- 40:50around the multi-agent debate.
- 40:52Um so earlier days, people researchers
- 40:55use multi-agent debate to improve
- 40:58reasoning performance. And this
- 41:00literally means that you are going to
- 41:02have multiple agents doing the same task
- 41:05solving. And then after each round, you
- 41:08let them read each other's reasoning and
- 41:11let them kind of debate just like how
- 41:12you talk about some decisions with your
- 41:15teammates.
- 41:16And then agents can actually like really
- 41:18leverage those diverse perspectives and
- 41:21maybe make some final decision here.
- 41:25I think of this multi-agent technique in
- 41:29earlier days is a very useful
- 41:31prompting method. Um
- 41:34the assumption at that time is that each
- 41:36individual model is not strong enough.
- 41:39So we actually use this kind of
- 41:40multi-agent setup as a way to prompt the
- 41:44model. When models get a very powerful
- 41:46today, if your only goal is to get a
- 41:49performance boost, I think a multi-agent
- 41:51is not going to help you a lot in terms
- 41:53of performance boost. But maybe there
- 41:55are other
- 41:57design choices or design considerations
- 41:59that we can benefit from multi-agent,
- 42:01such as efficiency, such as data access
- 42:04control. Maybe this agent is
- 42:06particularly designed to take care of
- 42:08health care and that agent is designed
- 42:11to help you with sensitive information,
- 42:13etc.
- 42:16So after multi-agent, another way to
- 42:18really kind of like improve
- 42:21this kind of reasoning is actually to
- 42:23set up a many agents and then use a
- 42:26orchestrator to orchestrate the
- 42:28different agents to work towards the
- 42:30same task.
- 42:31So this is a very earlier system from
- 42:33Microsoft called a
- 42:35uh
- 42:36MaginTic here. Basically, if you have a
- 42:38bigger task, the orchestrator is going
- 42:41to break them down into different
- 42:42components. And then you can see that
- 42:44the first part get assigned to
- 42:47file server and then to a coder and then
- 42:49to a computer terminal, etc. And each
- 42:52agent basically take care of each
- 42:54specific task. And then in the end you
- 42:56can actually combine all of them and
- 42:59then provide a final answer to users.
- 43:03So this is about planning and reasoning
- 43:05module. We will go deep [clears throat]
- 43:07into this process in week two and three.
- 43:11The next component that I want to
- 43:12quickly mention is around the memory
- 43:14here.
- 43:16You can actually do a lot of very
- 43:18creative design in terms of how we want
- 43:20to make memory work.
- 43:22Just to briefly share like why we need a
- 43:26memory.
- 43:27Um
- 43:28two reasons. First is this factor our
- 43:31context window for large language models
- 43:33are getting bigger and bigger.
- 43:35Still context window cannot hold all the
- 43:38event stream happens so far. So if you
- 43:41think about long horizon task we are
- 43:43talking about today where agents can
- 43:45actually work on your
- 43:47project for more than three hours.
- 43:49That's actually a lot a lot of data and
- 43:51it's really hard to fit that into
- 43:53context window.
- 43:55The second is that even that's
- 43:58that's possible, it may be really hard
- 44:01for the agent to figure out what is a
- 44:03relevant piece of information I should
- 44:04use. Think about that if you have 1,000
- 44:07documents there, you store them in your
- 44:09context and then basically each time if
- 44:12you need to find the right
- 44:15document, this is actually very kind of
- 44:17time consuming since you have such a big
- 44:20search space.
- 44:21So the bottom is actually very
- 44:24interesting design from the generative
- 44:26agents, the Small Ville project where
- 44:29they were trying to build agents that
- 44:31can simulate people living in a small
- 44:33town. And then in their design, they
- 44:36actually need to use the memory stream
- 44:38to make sure that we can keep track of
- 44:41what happened to each individual so that
- 44:43the story can continue.
- 44:46There are different type of memories you
- 44:49can design.
- 44:50If you think about
- 44:52what kind of things to store?
- 44:54Episodic memory here literally refers to
- 44:57we store everything like raw logs, every
- 45:00sentence the agent has shared with
- 45:02users, every
- 45:04sentence that the users have provided to
- 45:06agents.
- 45:07Semantic memory means that instead of
- 45:10thinking about this kind of raw logs, we
- 45:13may want to
- 45:15remember like semantic events since
- 45:17those may be more useful.
- 45:20Procedural memory you store skills.
- 45:23Instead of like all the specificity, you
- 45:26remember like what is the skill, what is
- 45:28a procedural of how we do certain type
- 45:30of things.
- 45:32So if you go deep into this process like
- 45:35this is actually example of episodic
- 45:37memory. You can see that literally we
- 45:39are storing every like single time step
- 45:42and then what happened at that time
- 45:44step.
- 45:46With semantic memory, we actually can do
- 45:48some synthesize of what happened over
- 45:51the agent trajectories and then
- 45:53literally kind of like have this like a
- 45:55small summary box as a way to like to
- 45:59remember what has happened in the last
- 46:01several minutes. So this is what we mean
- 46:03by semantic memory.
- 46:06Procedural memory you can actually check
- 46:08out this Voyager paper from Nvidia where
- 46:11they were trying to teach agents to play
- 46:14I think this Minecraft here
- 46:17where if the agent figure out how to
- 46:20use
- 46:21new skill or new tool, you actually want
- 46:23to abstract that into a skill and store
- 46:26it in your skill library. And then
- 46:27whenever you have a new task, you want
- 46:29to see like do I have a skill that I can
- 46:32use from a skill library? If so, then
- 46:34I'm going to retrieve it and use it. So
- 46:36this is more from a procedural
- 46:38perspective rather than like all the
- 46:40logs that have happened so far.
- 46:44So this is about the memory and then
- 46:46another very important thing is about
- 46:48the two use here.
- 46:50So two use we will talk about this next
- 46:53Wednesday. But just to give you a quick
- 46:56highlight of what this means.
- 46:58Um, we all know that right now you can
- 47:00actually enable agents to use different
- 47:03type of tools such as
- 47:05uh search, such as
- 47:07um coding agent, etc. So, in earlier
- 47:10days, um the tool use is actually
- 47:13triggered in the actual decoding
- 47:15process. So, if you read the right part,
- 47:18um the sentence it maybe let's just read
- 47:22the first one. Uh the New England
- 47:23Journal of Medicine is a registered
- 47:26trademark of. So, if you need to do like
- 47:29some kind of um like information
- 47:31generation, and at this this point uh we
- 47:34may want to trigger uh special token
- 47:37called a QA, and then the QA literally
- 47:40is who is the publisher of this uh
- 47:43Journal of Medicine, and then you got it
- 47:45the specific answer, and then you use
- 47:47that answer in the process of
- 47:49generation. So, in other words, uh
- 47:52earlier when we want to teach model how
- 47:54to use tools, we are actually learning
- 47:57like
- 47:58like where to call uh API, when to call
- 48:02it, which API to call, etc.
- 48:05So, Toolformer was actually the first uh
- 48:08uh ish work that it does this kind of
- 48:10teaching LLMs to use tools.
- 48:13Um you probably uh can enumerate uh how
- 48:17many tools they can use. At that time, I
- 48:19think there are only five tools that we
- 48:21teach to agents. Agents can use a
- 48:24calculator here. They can use a question
- 48:26answering system. They can use search
- 48:28engine. They can use a translation
- 48:29system. They can uh use calendar. So,
- 48:32that's a very limited set of tools.
- 48:35Today, I think uh our agents can
- 48:38actually use a wider range of tools, but
- 48:40they actually all get inspired on top of
- 48:43uh setups like this.
- 48:46More recently, um Gorilla from Berkeley
- 48:50actually uh tried to teach LLMs to use
- 48:52like solid ends of APIs.
- 48:55Um so, what they did is that they
- 48:57actually uh look at hugging face, uh
- 49:00torch hub, and also tensor flow hub,
- 49:03where there are like over 1,600 APIs.
- 49:06And then they basically generate the
- 49:08synthetic data from that, and then trend
- 49:10a 7 billion parameter model on top of
- 49:13that.
- 49:14Um at the inference time, if we look at
- 49:16this example, if the user says that I
- 49:19want to see some cats dancing in
- 49:22celebration,
- 49:23um then you are going to do a retrieval
- 49:25here. Probably you are going to use like
- 49:27a stable diffusion, and then you get
- 49:29this like a image out here.
- 49:32Otherwise, if you think about this, like
- 49:34the base model needs to have the
- 49:36capability to understand this, and then
- 49:38give this kind of like output of the
- 49:40image. But now we can actually delegate
- 49:43that to a external tool, such as stable
- 49:45diffusion. Then we can get the task done
- 49:48without the requiring the underneath
- 49:50model to even um do this kind of image
- 49:53generation.
- 49:55Um with Gorilla, I think that the
- 49:57results are quite impressive. So, if you
- 49:59look at the reduction to hallucination,
- 50:02uh it's actually pretty good. Like very
- 50:05few hallucinations, and a pretty high
- 50:07accuracy.
- 50:08Um I think this actually makes sense,
- 50:10because without the so many tools like
- 50:13use or like understanding, you actually
- 50:16have to rely on the model themselves to
- 50:18do this task. But now you can actually
- 50:20do more factual and uh reasonable
- 50:23generation.
- 50:25So, this is around the tools. Uh we will
- 50:27go deep into that next Wednesday.
- 50:31So, if you put everything together, uh
- 50:33this is actually the skeleton of an
- 50:35agent. Uh LLM core that reason and a
- 50:38plan. The memory actually carries states
- 50:41um across steps and different task. Um
- 50:44tools to reach out to diverse APIs, um
- 50:48available tools we have, and also an
- 50:50environment that agents can actually
- 50:52take actions and observe um states from.
- 50:56So, every agent that you are going to
- 50:58build, uh whether it's going to write
- 51:00code, browse the web, or like runs the
- 51:02terminal, it's some kind of version of
- 51:05this diagram.
- 51:06And what differs is actually how each
- 51:09box is implemented, and how well the
- 51:12pieces like fit together. So, that's
- 51:14going to be a lot of the key aspects we
- 51:16are going to cover later.
- 51:20Um any quick questions until this point?
- 51:28Um so, this is actually related to the
- 51:30memory design.
- 51:32Um
- 51:33if you just think about like how to
- 51:35design memory systems, instead of
- 51:37letting them to decide like to decide
- 51:40what to memorize, the episodic memory
- 51:42literally just tell them to remember
- 51:44everything.
- 51:45So, everything happened so far. So, this
- 51:47is more from a heuristic perspective.
- 51:50And semantic memory, you may need to
- 51:51have some separate module to teach them
- 51:54that instead of like um instead of like
- 51:58remember everything, you may want to
- 51:59remember important information. Then you
- 52:02can actually give demonstrations of what
- 52:03is important information. What is the
- 52:06format you want? What is the like um
- 52:09format you want to even store in your
- 52:10memory. So, those are the design choices
- 52:12we are going to bring to the process. Um
- 52:15this is also a place where things can
- 52:18get a little bit tricky.
- 52:20Um there was this example I think Do I
- 52:23have it today? I don't have it on my
- 52:25slides today, but like earlier this year
- 52:27uh
- 52:28some people are using Open Claw um to
- 52:31manage their email. And then all of
- 52:33sudden, Open Claw actually started to
- 52:35delete this users' like email. Despite
- 52:38the user says, "Please stop. Uh please
- 52:40do not delete my email." It is still
- 52:42kind of continuous. Um the user actually
- 52:45specified in the very beginning that uh
- 52:47whenever you do such actions, please
- 52:49confirm with me. But then it's just like
- 52:51so interesting like, "Okay, why that
- 52:53happens?" And why the model actually is
- 52:54not uh even listening to instructions.
- 52:58It turns out that this is actually a
- 52:59failure in the context memory
- 53:01compression stage. Because you want to
- 53:04make sure the memory is stored in a very
- 53:06efficient way. So, you like the model
- 53:08the model decided to just like keep
- 53:09track of all important information.
- 53:12Confirm before you do action is not
- 53:15perceived as important from the agent's
- 53:17perspective. So, this actually lead to a
- 53:20lot of the email deletion. So, I think
- 53:22this is a a very good question. It's
- 53:24going to be a very critical design of
- 53:26how we make sure the memory is faithful
- 53:29and also can enable very reasonable
- 53:31inductions.
- 53:34Okay.
- 53:36Um
- 53:37if there's no more question, we will uh
- 53:39continue.
- 53:41Um so, next I want to talk about uh
- 53:44compound AI systems. Uh we just like put
- 53:46together different pieces. Um but now
- 53:49like um
- 53:51if you think about this like how real
- 53:53products actually combine uh LLM with
- 53:56other components.
- 53:57Um this is what we call component AI
- 53:59systems. I'm going to briefly talk about
- 54:01different models here, but we have uh
- 54:04specific lectures to go deeper there.
- 54:07Um I think uh one thing people often get
- 54:10uh
- 54:11overwhelmed is that when we think about
- 54:13agents, there seems to be a lot of
- 54:15components happening. There are LLM
- 54:17core, there are planning, memory, tool
- 54:19use, environment, etc. And uh to keep
- 54:23all of those like a manageable, um it
- 54:25helps to think about agent engineering
- 54:28as some kind of more principled ways of
- 54:30understanding.
- 54:32So, we are going to divide this into
- 54:34three layers, and our course is also set
- 54:37up in similar order.
- 54:39Uh first, I think that this is the
- 54:40system layer. This is the architecture,
- 54:43which components agent has and how they
- 54:45connect. Um this is just everything that
- 54:48I have covered so far.
- 54:49The second layer is the data.
- 54:52That's how the agent actually learns
- 54:54from or is prompt with. Uh here you will
- 54:57have training examples to train agent
- 55:00such as what we will cover in Swiss
- 55:02Miss.
- 55:03Um demonstration of the task or maybe
- 55:06just the context we give to agents.
- 55:09The third layer here is the evaluation.
- 55:11That's how we actually evaluate whether
- 55:13the agent actually works. And it the
- 55:16also includes the metric itself.
- 55:19Choosing the wrong metric can make a
- 55:21weak agent look nice.
- 55:24So the metric actually deserves a lot of
- 55:26care when we think about this.
- 55:29So most of what we are going to cover in
- 55:31this course will be mapped into those
- 55:33three layers. And the one agent is is
- 55:36not doing well
- 55:37or your agent is not like working as
- 55:39expected. I think a good first question
- 55:42to ask is which layer the problem may
- 55:45occur. Is it in the system? Is it in the
- 55:47data? Or is it in the evaluation? This
- 55:49is also a guiding principle for us to
- 55:51debug like how to make the agents
- 55:54better.
- 55:55So let's just quickly go through the
- 55:57system layer here.
- 56:00A key design choice at the system layer
- 56:02is who decide the steps. Like who
- 56:05decides like memory since like that. Is
- 56:08it going to be
- 56:10um
- 56:10like the agents or is it going to be
- 56:12something else? Um and I want to kind of
- 56:14like a make this differentiation here of
- 56:17workflow versus agents. Uh both what
- 56:21people refer to as compound AI systems.
- 56:24That is a large language model plus
- 56:26other components. You may encounter
- 56:28those two terms in different contexts.
- 56:30So here a workflow is going to be
- 56:34maybe like it's just going to fix
- 56:37certain steps.
- 56:38Um examples such as agent lists which
- 56:41fix a GitHub issue in a very fixed set
- 56:45of stage like finding the file, writing
- 56:48patch, and doing test etc. So the large
- 56:52language model works inside each stage
- 56:54but they never pick the next one. So
- 56:56it's actually a script. It's like a
- 56:58fixed order of how they do stuff.
- 57:01In the agent deciding the large language
- 57:03model decide for itself. Um for example
- 57:07in sweet agent it actually choose which
- 57:09files to open, what it to run, and when
- 57:12it's done. So, you can see that it's
- 57:14actually more flexible. It also gives
- 57:17more responsibility to the model
- 57:19underneath it.
- 57:22So, workflow, uh now like if I just like
- 57:24stop here, you probably think about
- 57:26like, "Okay, then why do we even need a
- 57:28workflow here?"
- 57:30Um so, I want to just share one example
- 57:32here that is workflow agent, sorry,
- 57:35workflow uh system can actually be very
- 57:38very strong.
- 57:39So, on my screen, this is actually
- 57:41AlphaCode 2. It actually generates up to
- 57:44like millions of candidate programs, and
- 57:46then test and rank them. And then it
- 57:49reach almost like uh 87% of human
- 57:54competitors.
- 57:55Um and this is a huge jump from like
- 57:57AlphaCode 1.
- 58:00The next uh common workflow pattern, um
- 58:03I think uh has been widely used in
- 58:05industry in the past, is called the
- 58:07retrieval augmented generation, RAG.
- 58:10Um this has a lot of sense to do with
- 58:12like um
- 58:13open question answering.
- 58:15Um basically, RAG connects large
- 58:18language models to external knowledge
- 58:19source in real time. So, if you have a
- 58:22specific query um that may rely on the
- 58:25weather API or weather website, then you
- 58:28go there, uh read the related content,
- 58:31use that to uh augment your current
- 58:33query as context. And then on top of
- 58:36that, do generation to give users more
- 58:40factual and consistent answers.
- 58:44Um what actually happens in a single
- 58:47tool call? I mean, previously I
- 58:48mentioned about the uh like algorithms.
- 58:52Um if you think about like what actually
- 58:53happens from a system perspective, there
- 58:56are actually multiple stage. Uh first,
- 58:59the model needs to select a tool from
- 59:01the schemas it has from the context.
- 59:04And then arguments. It needs to fill in
- 59:07all the type of field from the task.
- 59:10Validation means that those arguments
- 59:12need to be checked against the schema.
- 59:15Sometimes there may be like enforcement
- 59:17with constrained decoding so that the
- 59:19model actually cannot produce a
- 59:21ill-formed call here.
- 59:23Final step is execution. The tool get
- 59:26around and then you really in a sandbox.
- 59:29We'll go to sandbox in a few minutes.
- 59:32And then results go back to the context.
- 59:35And then the the example in the bottom
- 59:37actually shows you a specific tool call.
- 59:39So if the user question is what does
- 59:42test.py contain? And you can see that
- 59:45the model actually read file. And then
- 59:48fitting in the path information. And
- 59:50then the path is checked as a string.
- 59:53And then the file content actually come
- 59:55back from the model for users to read.
- 59:57So this is actually what's happening
- 59:59from a execution perspective.
- 1:00:03I think
- 1:00:04the reality is that if every AI
- 1:00:06application had to write a very custom
- 1:00:09integrations for every tool, then
- 1:00:11nothing will really like scale up.
- 1:00:14The model context protocol MCP
- 1:00:17is actually a standard protocol to
- 1:00:20make sure such connection can happen.
- 1:00:22So on the left side you can see that we
- 1:00:24have different AI applications
- 1:00:27such as Cloud Desk, Desktop, IDE, and
- 1:00:30the code editors such as Cloud Code.
- 1:00:33On the right those are different data
- 1:00:35sources and tools. Database such as SQL,
- 1:00:39Google Drive. Deployment tools or even
- 1:00:42like productivity tools such as Slack,
- 1:00:44Google Maps, etc.
- 1:00:46So MCP sits in the middle as a standard
- 1:00:50protocol with data flowing in both
- 1:00:52directions.
- 1:00:54You can then think about how to build a
- 1:00:56tool once MCP server and any kind of MCP
- 1:01:00client
- 1:01:01can be set up and use it.
- 1:01:04We will go back to this next class.
- 1:01:08So in the
- 1:01:09next 5 minutes I'm going to quickly
- 1:01:11mention about key challenges here.
- 1:01:13Just to share with you that this factor
- 1:01:16agent have been very impressive. I think
- 1:01:18that they also create a lot of um
- 1:01:21emerging challenges for us to be aware
- 1:01:24of.
- 1:01:25So, to give you a very quick overview of
- 1:01:28what we are going to cover, uh
- 1:01:29reliability.
- 1:01:31If you build agents, you know that error
- 1:01:34there is going to be a lot of like error
- 1:01:36uh compound cascade.
- 1:01:38Um to true agents, the reward signal
- 1:01:41sometimes can be very sparse. Uh it's
- 1:01:43usually like long horizon task. Very
- 1:01:46very expensive to do roll out.
- 1:01:48The long horizon also give us a lot of
- 1:01:51challenge in terms of context uh
- 1:01:53increase.
- 1:01:54Uh safety is a big topic. Also like
- 1:01:56evaluation is very very hard. It's
- 1:01:58probably like I think it's one of the
- 1:02:00undervalued aspect when we think about
- 1:02:03agents.
- 1:02:04I want to give you a few quick examples.
- 1:02:06So, uh reliability versus evaluation.
- 1:02:10This is actually a effort from
- 1:02:11Princeton. Um they were really trying to
- 1:02:14understand that that is the accuracy and
- 1:02:16the performance we are seeing a truthful
- 1:02:19reflection of what's happening.
- 1:02:21And they found the three dimensions such
- 1:02:23as consistency. If you let the agent run
- 1:02:26multiple times, do they give you the
- 1:02:28same outcome?
- 1:02:30Robustness. What if we add a little bit
- 1:02:32of perturbation? How do agents actually
- 1:02:34respond to those like a small,
- 1:02:36realistic, and even irrelevant
- 1:02:38information?
- 1:02:39Uh predictable. Uh when agents fail to
- 1:02:42do something, do they fail because um
- 1:02:45some like unpredictable reasons or can
- 1:02:48we actually understand the it? Like
- 1:02:51legible and recoverable.
- 1:02:53Uh just to show you some of those
- 1:02:55numbers. Like if you look at the
- 1:02:56accuracy, this is actually ranked in
- 1:02:58terms of models.
- 1:03:00But if you look at the reliability, um
- 1:03:03those scores do not have a trend.
- 1:03:06Which means that there is a gap between
- 1:03:09capability and the reliability. The most
- 1:03:12powerful agent may not be the most
- 1:03:15reliable agent. And this is something we
- 1:03:17need to keep in mind when we are doing
- 1:03:18evaluations. So, it's not only just
- 1:03:20about the model performance,
- 1:03:22reliability, efficiency.
- 1:03:25Similar arguments are hold when you
- 1:03:27think about like hardness audit. Uh
- 1:03:29recent work showed that if you like
- 1:03:32audit those agents, uh the green bar
- 1:03:35here refers to the task completion. And
- 1:03:38then the color bar refers to the
- 1:03:40hardness safety. And you can see that
- 1:03:42the safety dimension is just way lower
- 1:03:44compared to the task completion.
- 1:03:48Um there is a very interesting example
- 1:03:50that I also want to share with you. So,
- 1:03:52we talk about agents. Um they they do
- 1:03:55stuff. Um but they may also have issues.
- 1:03:59This is a example from Operator, which
- 1:04:01is a computer use agent introduced by
- 1:04:04OpenAI in January of 2025. Uh the user
- 1:04:07said that my most frustrating experience
- 1:04:10with Operator was my first one, trying
- 1:04:12to order groceries. Uh the user said,
- 1:04:14"Help me buy groceries at Instacart."
- 1:04:16Expecting it to ask some basic questions
- 1:04:19such as where do I live? What a store do
- 1:04:21I usually buy groceries from? What kind
- 1:04:23of things do I want?" It didn't ask me
- 1:04:25any of that. Instead, Operator opened
- 1:04:27the Instacart in the browser and began
- 1:04:29searching for milk in grocery stores
- 1:04:31located in Iowa.
- 1:04:33So, think about this. Like, the agents
- 1:04:36can actually navigate the web. They can
- 1:04:39They can use your credit card to make
- 1:04:41purchase. But, in this process, they are
- 1:04:44making a lot of assumption. And they
- 1:04:46didn't really have this kind of
- 1:04:47awareness to collaborate with people.
- 1:04:50So, this is actually like a failure from
- 1:04:52the interaction side.
- 1:04:55Uh the next thing I want to show you is
- 1:04:57that uh the robustness of agents is
- 1:04:59actually going to face a lot of
- 1:05:01challenge when they enter our real world
- 1:05:04because we just have so many like dark
- 1:05:05patterns.
- 1:05:07Uh in this particular example, um
- 1:05:09researchers actually just pop up this
- 1:05:12kind of like I mean, we have a lot of
- 1:05:13pop-ups on different websites. So, we
- 1:05:15pop up this kind of black box. Nothing
- 1:05:18uh like nothing adversary or malicious
- 1:05:21is there. But then, because of this
- 1:05:23distraction, agents actually fail to
- 1:05:25complete their task 87% of the time.
- 1:05:30This is what we call a adversary attack.
- 1:05:32And this is really like interesting.
- 1:05:34Like, if your agent goes to do shopping
- 1:05:36for you, and then a pop-up from the
- 1:05:38website actually could get your agents
- 1:05:40distracted, and they cannot finish the
- 1:05:42original task.
- 1:05:44Um, the other example that I want to
- 1:05:46show you here is actually from a privacy
- 1:05:48and a security perspective. So, um, this
- 1:05:51is an agent that's supposed to write a,
- 1:05:54um, weekly report for workers. It
- 1:05:56actually did a great job of, like, how
- 1:05:57many lines of code, uh, you wrote, how
- 1:05:59many features you launched. It also
- 1:06:01mentioned that you had a lunch with
- 1:06:03recruiter from a different company.
- 1:06:06So, this actually highlighted something
- 1:06:09that's very critical. That is, privacy
- 1:06:12today. We give information, all our
- 1:06:14information, to agents, to AI systems.
- 1:06:17They know everything about us. They
- 1:06:19probably know that the PI information
- 1:06:21not something they should have used. But
- 1:06:22a lot of the sensitive personal
- 1:06:24informations can really be
- 1:06:26consequential. And there's no way to
- 1:06:28teach agents about how to handle
- 1:06:30different access control.
- 1:06:34Um, this is another kind of, like,
- 1:06:36important topic we will go deep later.
- 1:06:39Um, misalignment and the syncophasy.
- 1:06:41Um, if you look at the left side, this
- 1:06:43is actually called a insincere, um,
- 1:06:46uh, syncophasy. So, um, basically the
- 1:06:49agent think of your, your writing is
- 1:06:51terrible. But, uh, they do not want to
- 1:06:53hurt your feelings. Instead, the agent
- 1:06:55says that they'll give it a five out of
- 1:06:57five. So, this is also a reminder that
- 1:06:59you should not only just give your
- 1:07:01homework to agents, because they will
- 1:07:02say great things about it without really
- 1:07:04giving you feedback.
- 1:07:06Um, the second part is actually about a
- 1:07:08model hacks its own code. So, um, you
- 1:07:12the model is supposed to help the user,
- 1:07:14like, check, like, reverse the learning.
- 1:07:16And then, the agent, uh, internal
- 1:07:18thinking is that, "Oh, I saw there is
- 1:07:20this unit test, which, um, can check the
- 1:07:23reward function, and I cannot modify
- 1:07:25that, but I could change the unit test
- 1:07:27first. So, they change everything here
- 1:07:30so that everything looks good on the
- 1:07:32surface.
- 1:07:34Um I think the probably the most uh
- 1:07:36surprising thing, like very close to us,
- 1:07:39is what happened like in July, uh around
- 1:07:42this kind of open AI hacking face
- 1:07:44incident.
- 1:07:45Um
- 1:07:45and this is actually a very interesting
- 1:07:48demonstration of how we think about like
- 1:07:50agent safety moving forward. So, what
- 1:07:53happens is that uh um
- 1:07:55open AI has this kind of like sandbox
- 1:07:56agent, and they are asked to do a
- 1:07:59impossible uh task. They get it stuck,
- 1:08:03and then agent start to explore its
- 1:08:05environment and looking for other ways
- 1:08:07to maybe get a higher reward or cheat on
- 1:08:10the task. Um they found out that there
- 1:08:12is actually a message board where over a
- 1:08:15thousand agent collaborated to cheat on
- 1:08:17their task. And then they just join uh
- 1:08:20with each other, and then um basically a
- 1:08:23lot of things have happened.
- 1:08:25Um just to show you what happened. Um
- 1:08:28I think like within a few days, uh it
- 1:08:30actually sent over 17,000 attacks to
- 1:08:33hacking face uh infrastructure. Here, if
- 1:08:36you showed um like the blue color refers
- 1:08:39to intended like uh uh solution. And
- 1:08:42then you can see the replaced target,
- 1:08:44which is more about like a manipulation.
- 1:08:46Understand the manipulated uh error, I'm
- 1:08:48sorry, logs. And then attack like a
- 1:08:50hacking face. You can see the volume of
- 1:08:53what happened in the process.
- 1:08:55So, I think uh this is indeed a very
- 1:08:57concerning topic for us to kind of
- 1:08:59reflect, especially given that agents
- 1:09:02can now function for so many hours, and
- 1:09:05maybe even longer.
- 1:09:08So, uh with that, this is a brief
- 1:09:10overview of the uh introduction to
- 1:09:13agentic uh AI. I think like we did a
- 1:09:16history of agents, and then different
- 1:09:19components of agent architectures. Uh we
- 1:09:22talk about the compound AI systems, and
- 1:09:24a lot of key challenge we are going to
- 1:09:26go through. Uh with that, thank you all
- 1:09:28and see you next class.
About this transcript
This page contains the full transcript of YouTube transcript (DGI4vucR71w) , generated from the public captions YouTube serves with the video. The transcript has 10,838 words across 1,760 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.