YT YouTube2Text

YouTube2Text → Transcripts → YouTube transcript (6YnLB0XbTnI)

YouTube transcript (6YnLB0XbTnI) — Transcript

11,603 words · 1,840 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:05Welcome everyone to fall quarter and
0:07welcome to CS 329A.
0:10I hope you made it to the right class.
0:11This class is on self-improving AI
0:13agents.
0:15Anyone
0:16made it to the right class here?
0:19Yes? Okay.
0:21How many of you are here
0:24and doing masters at Stanford?
0:28PhD?
0:30Okay.
0:31A few of you and then undergrad?
0:34Okay. So masters is the big crowd here.
0:37>> There's a good balance. Yeah.
0:39>> It's a really good balance. Uh
0:41So we'll introduce ourselves. So I'm
0:43Akanksha. I have been working in large
0:45language models for a while and I'm an
0:47adjunct professor at Stanford and I'm
0:48also in research
0:50at a startup called Reflection AI.
0:52>> Hi, I'm Azalia Mirhoseini. I'm a I'm an
0:56assistant professor in the CS department
0:59and
1:01Akanksha and I met each other when we
1:04were
1:05at Google Google Brain Google Brain.
1:07Yeah, back in the day and then Google
1:09DeepMind.
1:11And I also worked on Cod and Anthropic
1:13and Gemini at Google
1:15DeepMind and we are so excited to
1:18teach this course again. This is the
1:20second time we're teaching. Of course,
1:22we have made some updates both to the
1:24lectures and also the way we are
1:27kind of conducting everything during the
1:30the for the course.
1:32And we are we were just so amazed by the
1:34amount of interest
1:36for people to join this course. So
1:39we are excited to have you here.
1:41>> The course website is at
1:42cs329a.stanford.edu.
1:45We have updated the website in terms of
1:47all the lecture materials and and the
1:50papers that you'll be reading as well as
1:52what's the homework schedule to expect
1:54the project proposal and project
1:56structures as well.
1:58So, in today's lecture we'll start with
2:00an overview of large language models.
2:03Just how the scaling trends have evolved
2:05over the last 5 years and then we'll
2:08focus more on what we will cover in this
2:10class. And then from there
2:12in the second half of the
2:14this lecture we'll cover the course
2:15logistics and that part you should pay a
2:18lot of attention to because that
2:19influences your grades.
2:23So, let's take a look at an overview of
2:25the trends for large language models.
2:27So, one of the aspects that has been
2:29mind-blowing since GP3 came out is that
2:34there is a
2:35as you scale up the number of parameters
2:38in a model they get better and what this
2:40means is that language models like BERT
2:44and T5 were good but as we increase the
2:46number of parameters they got really
2:48better. They they perform much better
2:50and what this led to is something called
2:51a scaling loss for large language models
2:54which are used in building the base
2:56model for the pre-trained base model
3:00as you increase the amount of compute.
3:01So, there are three graphs here. As you
3:03increase the amount of compute on the
3:04x-axis the loss function on the y-axis
3:08goes down and and the more compute you
3:11have put in the more
3:13the test test loss goes down which leads
3:15to a better model. So, that was one big
3:17aspect that led to GP3 and then all the
3:20subsequent chat GPT, Palm,
3:23Gemini and so on. And then similarly as
3:26you increase the data set size on x-axis
3:29the test loss goes down further on
3:31y-axis. So, that's another aspect of
3:33like another axis of scaling that leads
3:36to better language models. And
3:39the fourth one is of course the third
3:40one is of course parameter count. So, if
3:42you increase the number of layers in a
3:44transformer or the amount of parameters
3:46that are there in the language model
3:48that improves the loss our and that that
3:51leads to a better model. And then this
3:52has been the foundation for a lot of the
3:55this has been the foundation for a long
3:57time until last year where this was
3:59starting to hit some kind of a
4:01saturation point.
4:04So,
4:05from 2018 to almost 2024,
4:08this graph is a little bit outdated but
4:11from 2018 to 2024, the as the model size
4:14has gone up
4:16almost consistently, we have been
4:17increasing the the size of the model.
4:19So, we started with something like BERT
4:21was 340 million parameters, GPT-2 was
4:231.5 billion parameters, and then GPT-3
4:26was 175 billion parameters, and then
4:29PaLM was 540 billion, and then
4:32there was a bunch of subsequent models,
4:34and GPT-4 is estimated to be
4:36there which is like trillions of
4:38parameters. So, you can imagine that
4:39there has been an exponential growth in
4:41just the model size of for large
4:43language models. The large stands for
4:45that they're growing in size, and that
4:47has been one of the key areas in which
4:50the models have been getting better.
4:53So, why do we care about the models
4:55getting bigger? What does this lead to?
4:57So, as I was explaining that as the
5:00models get bigger, they continuously
5:02improve in performance on natural
5:04language benchmarks, on various other
5:06benchmarks, reasoning benchmarks, and so
5:08on.
5:09And then other interesting aspect that I
5:11will show you in a moment is that they
5:13learn how to do few-shot learning. So,
5:15earlier you have to fine-tune the model
5:17for a specific domain, but just by
5:19giving a few examples, the model can
5:22follow that template and and reason in
5:24that same vein which makes it extremely
5:26easy to prototype things. And then the
5:30third aspect is that as the models
5:32become bigger, they have emergent
5:33behavior where the capabilities like
5:35reasoning only emerge in larger models.
5:39So, just to understand few-shot learning
5:41or zero-shot learning here. So,
5:44if if the model is given a question like
5:47or a task like translate English to
5:49French,
5:50it's able to give So this is called a
5:52prompt. Like it's given a task translate
5:54English to French and then it's asked to
5:55translate cheese to whatever is the
5:57French word. If the model can predict
5:59the answer without
6:01actually being trained on the specific
6:03task, then that would be called
6:04zero-shot learning. And then few-shot
6:06learning,
6:07in addition to giving the
6:09description of the task, you're also
6:10giving a few examples. So in this
6:12particular case, it's given translate
6:14English to French and it's given a few
6:15examples of the translation from English
6:18to French and then it's asked to
6:19translate cheese and it's able to
6:21translate that and you can see that this
6:23capability shows up in the large
6:26language models across a large number of
6:28tasks and that has been one of the key
6:29areas that GPT-3 and Palm and other
6:33models enabled and then that
6:35subsequent innovations came from
6:37reasoning aspects of this.
6:39So on the reasoning side,
6:41as models have become bigger, they have
6:44this emergent behavior of reasoning
6:45which Azalia will tell us a little bit
6:46more about.
6:48>> So as the models become bigger, not only
6:51we have this kind of a predictive
6:53scaling laws property that we kind of
6:56know how the loss function's going to go
6:58down as we throw in more compute data
7:00and parameters in the model, there are
7:03new kind of behaviors that appear in the
7:07model that they didn't have before and
7:08we never could predict that until we had
7:11these bigger models and saw these kind
7:12of behavior in them. And one of the most
7:16important
7:18one of those behaviors is the emergent
7:21is the chain of thought behavior. And
7:24for those of you who are not familiar
7:26with this, here is how it works. So in a
7:29usual prompt prompting of a model, we
7:32can ask a model in this case a
7:34a math problem
7:36and then we can give it a one-shot
7:38example. Here is a math problem, here is
7:40the answer. Now I ask you a similar but
7:44not exact and not exactly the same math
7:46problem, and the model is expected to
7:48answer that.
7:50Um
7:51And the model might be able to use the
7:53the one-shot example or it might not be.
7:55But we can do better here by not only
7:59give an example in but also provide the
8:02reasoning or the chain of thought on how
8:05we got to that answer. So, in this case,
8:07the question is a very simple question.
8:10Roger has five tennis balls. He buys two
8:13more cans of tennis balls. Each can Each
8:16can has three tennis balls. How many How
8:19many total tennis balls does he have
8:21now? And instead of just saying 11, like
8:23we're walking the model through how to
8:25get to this.
8:26Um Roger started with five balls. Two
8:28cans of three tennis balls is six. And
8:32uh 5 + 6 is 11.
8:34Uh And with this example, then the model
8:37has become familiar with the process to
8:41get to the answer, and it can leverage
8:42that to answer new problems.
8:46Well, obviously this is a very simple
8:48problem. Any like one-B parameter model
8:50nowadays can solve this. Doesn't need a
8:53an example or a few shot or a chain of
8:56thought. But the chain of thought
8:57property itself is a property that is
9:00holding to this day and it's a very very
9:02important property for the reasoning
9:05models and all the thinking models and
9:07all the prog and many much of the
9:09progress that we have seen um
9:11in the past year or so.
9:14Now, um
9:15this property again appears as we have
9:18larger and larger models. Here are three
9:21models, Lambda, GPT, and PaLM. And as
9:24you can see, the chain of thought and
9:26this is some results on the math some
9:28math data set. And these models
9:32when they're smaller, like eight billion
9:35parameter for Lambda or
9:37um deep for GPT around 7 billion. They
9:40can't really benefit from chain of
9:42thought. Like it doesn't do anything for
9:44them. But as they become larger
9:47or for a palm model again at 8 billion,
9:50it can leverage this chain of thought
9:52property and learn from the reasoning
9:55that is and the the process that is
9:57provided in the context in order to
10:00solve problems better.
10:03And not only that, there are other kind
10:05of abilities to solve new task as we
10:08increase the size of the models. Like
10:10for example, small models can all of a
10:12sudden solve mod arithmetics better or
10:15solve
10:17word unscramble problem just by having
10:20this larger size and this property. You
10:23see that all of a sudden it appears at a
10:25certain size.
10:27And because of all of this, we still
10:30as in frontier companies and labs are
10:32still interested in pushing the the the
10:35scaling laws and not only get the
10:38natural kind of progression, but also we
10:41are we we will see more and more of
10:42these kind of emergent behavior that we
10:45are
10:45that is going to be very important.
10:49Now, a little bit of history here. So
10:52chat GPT was launched in
10:55November 2022.
10:58Obviously, it was one of the most
10:59successful
11:01kind of
11:02apps or software ever created. The time
11:05to reach 1 million users for it was
11:095 days, which is significantly
11:13faster than any that many of the other
11:16famous
11:17software that you can see there or
11:18services that you can see there. And
11:20there were pieces of
11:22kind of innovation on top of just
11:25scaling up the parameter size of the
11:27model that were very important to make
11:30chat GPT what it was and to make that
11:33kind of leapfrog and over the previous
11:36models like GPT-3 and so on. And two two
11:40important piece here but were
11:43um the instruction tuning and um the
11:46reinforcement learning from human
11:48feedback which which we're going to
11:50briefly mention in the next slides but
11:53throughout this course we're going to
11:54learn more and more about them. Um so
11:58let's
12:00first and then um we're going to learn
12:02about all of these in more details but
12:05let's just take a step back and see how
12:08the process works. So the pre-training
12:10step is
12:12the step that
12:13the the beginning of any language model
12:16training.
12:17Uh that's the easiest step and we just
12:19train the model to predict the next
12:21token out of all sort of text and data
12:24and so on.
12:25And then this is the fine-tuning step
12:27that was a key differentiating factor
12:30for a model like chat GPT. Um and let's
12:33start with um
12:36uh the first aspect of fine fine-tuning
12:39which is kind of alignment between human
12:41preferences.
12:43So when a model is pre-trained it has no
12:46sense it has seen all data on the
12:48internet on the in the books everywhere
12:51but it has no sense of like what is
12:53right and wrong and what is it just have
12:56it this statistically knows about the
12:57state of the world but does it it
12:59doesn't exactly know things or know how
13:01to follow instructions. So the next step
13:04in training these models
13:06uh is the the the the [clears throat]
13:08step where we try to steer the AI models
13:11to follow the goals and preferences uh
13:15and values of humans. Um
13:18and this is still a big problem it's a
13:20big issue we we haven't like mastered it
13:23yet but as the models become as we
13:26progress models are becoming better and
13:28much more uh kind of like
13:31uh
13:32powerful in the things that they can do.
13:35So, it turns out that we can fine-tune
13:37the models to be aligned. For example,
13:40here are some graphs that show what if
13:42we take our base base model that's just
13:45pre-trained and fine-tune it for things
13:48like sensibleness or safety. And
13:52these are like these are data that that
13:55we have collected. High-quality data for
13:58kind of showing the model what it what
14:00we mean by what's safe and what's unsafe
14:03or what's sensible and what doesn't make
14:05sense. And these are like more curated
14:07like more highly curated data that we
14:10pass to the model for fine-tuning and to
14:12make the model more aligned with the
14:13values or the or the preferences of
14:16humans.
14:18Um
14:21So,
14:22going back, let's just walk through this
14:25process again. So,
14:28the first stop when we take a
14:30pre-trained language model is
14:33fine-tuning with next token prediction,
14:35which follows the same process as
14:38pre-training, but this time on data that
14:41is much more higher quality.
14:44For example, this could be data from
14:47books, creative essays,
14:50data that companies pay a whole lot
14:52millions of dollars or hundreds of
14:54millions of dollars potentially to buy
14:56this data that's really high quality,
14:58and then we train the we fine-tune the
15:00model on those. And the model becomes
15:03much better as a result of that.
15:06Um
15:07The The next step is instruction tuning.
15:11And by instruction tuning, what we mean
15:13is that we show the model again
15:16traditionally in the past 2 years, this
15:20instruction tuning data is like a
15:22combination of human generated data or
15:24templates and then and then and the
15:26synthetic data. But, the way this
15:28instruction to fine-tuning works is that
15:31we do have
15:33instruction and then question answer
15:36pairs
15:37where the model learns how to follow the
15:40questions and answers them. For example,
15:42here an instruction for
15:45tuning data set could be about the
15:47weather like please answer the following
15:49question. What is the boiling point of
15:52nitrogen? And then that this is the
15:53answer.
15:54Or, we can have chain of thought
15:56fine-tuning where we
15:59show the model how it can walk through a
16:01process to get to the answer. And then
16:04we have the label, we have the results
16:06over here. And this is like these pairs
16:08of like questions and instructions and
16:10answers. And again, this this data set
16:13is very um is
16:16is there's a lot of effort that goes
16:18into this data set and the quality of
16:19data set and the uh generality of it uh
16:22has a lot of impact on the quality of
16:25the model that we get at this stage.
16:27Um and after this point in the model
16:30becomes more and more like um
16:34like the models we see today. Like you
16:35can ask them questions, you can go back
16:38and forth with them. They they have this
16:40sense of like how to uh perceive a
16:43question and how to walk through kind of
16:46uh some sort of like process to get to
16:48to an answer.
16:51And the next step after instruction
16:53fine-tuning is this process of
16:56um using human preferences
16:59and fine-tuning the model based on that.
17:01And this process is called RLHF or
17:03reinforce reinforcement learning from
17:05human feedback. Um the way this is
17:08different from the previous system is
17:10the way we create the objective function
17:13and uh create the data. So, in this
17:15case, instead of like just creating some
17:18supervised prompt and labels, we are
17:21basically creating a reward model out of
17:25human preferences. So, companies spend a
17:29lot of money
17:30uh
17:31asking humans, sometimes experts,
17:33sometimes just
17:35humans, normal humans, uh about
17:37questions and the answers generated by
17:40the model. And we ask the the humans to
17:43rate them
17:44to see which one is correct and is not
17:46correct. And that's how we can create
17:49some sort of a reward model here.
17:51Um and this reward model is instead
17:54instead of human, then we can use the
17:56reward model in order to kind of like
17:58guide the parameters of our LLM towards
18:02uh generating answers that are in line
18:04with this reward model. So, basically,
18:06we want the the generations by the model
18:10uh to be such that this reward model
18:11says, "Yes, these are good generations."
18:14Um we're going to learn more about this
18:16here, but I wanted We wanted to you to
18:18have um the high-level idea of what
18:21RLAIF means. Uh and we can collect We
18:24can have different reward type different
18:26types of reward. For example, we could
18:28ask the model to reward a pair of
18:30answers based on their correctness or
18:32helpfulness or specificity or
18:35harmlessness, all sort of reward
18:37functions that we can define. And
18:39depending on how we want the models to
18:41be and what we care most about, we can
18:44weight these different reward models and
18:46kind of use that in the RLAIF process.
18:50So, these step at a at a high level, of
18:53course, there's a lot of effort goes
18:55into that, but like pre-training and
18:57then
18:58um
18:59fine-tuning on higher-quality data,
19:01instruction tuning, and RLAIF are were
19:04the core kind of components that made um
19:08a model like ChatGPT over over something
19:12that existed over all the large models
19:15um
19:16like GPT-3 and prior models. So, that
19:18made the huge difference in how good and
19:21capable the models were.
19:23Um
19:24Uh
19:25so
19:26so that was very cool up until last year
19:29or a year a year year and a half ago,
19:32pre-training and then fine-tuning were
19:33the big pieces.
19:35Uh but since a year and a half ago, it
19:37turns out inference is also a frontier
19:41for making the models much more capable.
19:44Um
19:45so
19:46and that
19:48opened like a whole set of whole lot of
19:50new research and directions and ways
19:52that we can make the model better.
19:54Um so let's think let's talk about it a
19:58little bit. Like this is a work we did
20:00in my lab last year
20:03uh last summer. It it was called large
20:05language monkeys. Um
20:08and it's kind of inspired by the
20:09infinite monkey theorem and it's related
20:12to inference scaling. Um
20:14how many of you have heard of the
20:16infinite monkey theorem?
20:18Okay, that's a good amount. So, the idea
20:21of the infinite monkey theorem uh what
20:23it says is that
20:24it's not proven, but what it says is
20:27that if you have a monkey and a kind of
20:29a typing machine and then we keep
20:32letting the monkey type all day forever,
20:36um after some time you will have the
20:38works of William Shakespeare in in the
20:40things that the monkey types.
20:42Um
20:43so we were inspired by that to name our
20:45project. Uh but the idea here was that
20:48let's let's have the LLM be the monkey
20:52and uh let's
20:54uh ask the model over and over again to
20:57solve a given input problem.
21:01Uh you can think of it as parallel
21:02sampling or parallel generation. So,
21:05instead of asking the model once to
21:06solve the input problem, you can ask it
21:08many times. And also, let's assume you
21:11have some sort of a verifier or
21:13selection mechanism, so you can pick
21:16which one of these generated responses
21:19were correct.
21:21Um and then you output that in your
21:23system. That becomes the out the final
21:25output. So, the model, instead of
21:27generating one answer, generates many,
21:29many answers. And then your verifier
21:31selects which one of them is correct and
21:33then outputs one of the correct ones. Um
21:36you can think of a verifier as, say,
21:38some unit tests. If the problem is a
21:41generating code, the verifier can could
21:43be the unit test that we run against the
21:45code and we see which which one of the
21:48code generations passes all the unit
21:50tests and so on.
21:52Um The reason this is very This is a a
21:56This can be a possible possibility here
21:58is that models are um do not are not uh
22:03there is some variance in the data the
22:05models generate responses. It's not
22:07deterministic. And you can even control
22:10the variation in the model responses
22:12with this thing called temperature. So,
22:14you can force the model or encourage the
22:16model to generate more different
22:18responses as you ask a question.
22:22Um
22:23And here, now let's see some of the
22:26results of this repeated sampling. For a
22:29bunch of tasks like math and coding
22:32benchmarks, we increased the number of
22:36samples per problem from one, this is
22:39the normal one,
22:41uh to 10,000.
22:43And here we are showing the coverage or
22:45the fraction of problems that are solved
22:48by at least one of these samples.
22:50And what we are seeing here is that um
22:54so the red line, the red dashed one, was
22:56the GPT-40
22:58model. And while these blue and green
23:02lines, which were the llama 8B, llama 3
23:058B, and 7B, were worse than GPT for our
23:08model with one sample, but if we
23:11increase the number of samples from
23:13these models, in all of these cases,
23:16they do better than the GPT 4 model.
23:21And what it says is that it kind of
23:24seems like the model or models already
23:26know a whole lot more
23:29than what you get out of them when you
23:31just ask them once, right? So, this is
23:33the kind of the underlying property of
23:37inference scaling because it seems like
23:39we can get a whole lot of a whole lot
23:41more capability and problem-solving kind
23:44of ability from these models if we just
23:47bring in this inference scaling. Here,
23:50you're showing repeated sampling, but
23:51there are many other ways to do that.
23:53And we are So, the reason it's called
23:56inference scaling is that we're not
23:57touching any of the parameters of the
23:59model. The model is fixed, and we are
24:01just like at inference time, we are
24:03creating different ways to generate to
24:06kind of produce generations from the
24:08model and use them to generate
24:11higher-quality answers or answers to
24:13harder problems for the model. Um
24:15the interesting part here is that for
24:17some of these problems, out of these
24:1910,000 solutions, maybe like three or
24:22four of them were correct for a single
24:25problem. So, it shows that how how kind
24:28of
24:29um
24:30uh how important it is to to scale the
24:33inference to really get to the
24:35core capability of the models.
24:56Sorry, what's the question?
24:57>> What is the difference of GPT-3 and when
24:59we select traversing all the possible
25:01answers and we do
25:03and select and select all the
25:04>> So, the possible answers need to be
25:06generated somehow by the model, right?
25:10So, so this is like the space of all
25:13possible answers to the model is way
25:15bigger, like like right? We can like if
25:18we do like a tree search or a random
25:20search, it's like way bigger, right?
25:22There's no way to like hit even like
25:24imagine like we are asking a human like
25:28just sit down and solve this problem,
25:30right? There is a limit to
25:32Um of course there is no limit to human
25:35creativity, human's creativity, but
25:37there is a limit to how many generations
25:39we can have, right? Or if you're
25:41randomizing, there's a whole lot set of
25:44answers that can be generated and it's
25:46and this is like highly still highly
25:48sample efficient, right? 10,000 is not a
25:51whole lot, especially like for example
25:53for the math um F2F,
25:56some of these problems are really hard
25:58like IMO level problems. Like these are
26:00really hard problems that a tiny model
26:03like a 7 and a Llama 8B can solve.
26:08>> How does it compare how how does the
26:09latency compare with
26:12like when you do actually generate all
26:14the samples like compared to the GPT-3
26:17or all, right? Like how does the latency
26:21>> Yeah, of course there is a trade-off um
26:24and we we have we touch on that in the
26:26in the lecture that I go more details
26:28toward this. The parallel samples, the
26:30good thing about them that they're they
26:32can be running parallel.
26:34Um so, from a latency perspective,
26:36that's less of an issue, but of course
26:38there is the frontier of cost trade-offs
26:40between how much compute you're spending
26:43and and we will look into that. And it
26:45varies, it varies depending on the type
26:47of and complexity of the problems.
26:50Yes.
26:52>> The latency when you have
26:53a pair viable depending
26:57Yeah.
26:59Yes, great question again. It's a very
27:01hard When you have verifiers, it's much
27:04easier. When you don't have verifiers,
27:06there's this whole set of research on
27:08how to train LLMs judge or LLMs reward
27:12functions or LLMs with tools becoming
27:15and we're going to learn about all of
27:16it. We have a lecture just dedicated to
27:18verifiers, so.
27:22Yeah.
27:35No,
27:37yeah, we can't like have the temperature
27:38being too high because it's just it's
27:40going to be gibberish, like the model
27:42doesn't generate. So, we
27:45There's actually like a research project
27:46last year was like on tuning that or
27:49like figuring out what temperature.
27:50Usually, if you go beyond like 1.2 or
27:54so, it's
27:55it's not great, yeah.
27:57Yeah.
27:58But there are there are other tricks
27:59that you can do to
28:00kind of kind of uh
28:03make the model be more um
28:05like have more diversity in the type of
28:08answers
28:09and that can be useful.
28:11Uh
28:12All right.
28:13Um
28:16So,
28:18let's look at this. So, now, again,
28:21going back to the events of last year,
28:24year and a half, um we had the DeepSeek
28:27model. I think it came out in
28:29>> December.
28:30>> December.
28:32>> DeepSeek came out December 2024.
28:35Uh oh, well.
28:36>> It's been out for a
28:38>> Short time or long time, I don't know.
28:40The both. Uh so, so then models like
28:44DeepSeek came last year. So, basically,
28:46a core kind of innovation in Deep Seek
28:49and then the 01 series Gemini thinking
28:51and such were that. Now, we are kind of
28:54bringing this fine-tuning and test time
28:56scaling together because with this test
28:59time scaling and the models themselves,
29:01we have this new like kind of engine
29:04that we can generate a whole lot of
29:06synthetic data um
29:08for for example, for problems like for
29:11math problems where we know the answer,
29:13we can have the model generate different
29:15like kind of answers that leads to that
29:17final final kind of golden answer. Or
29:21for the coding problems, we can use the
29:23model to generate tons of data during
29:26test time scaling,
29:27uh tons of like quality data to solve
29:30coding problems, and that becomes part
29:32of our training set now. We can use that
29:34to fine-tune the model
29:36to to become better. So, that kind of
29:38bringing these two together became a big
29:41piece in thinking models and reasoning
29:44models. Um and this this is like a This
29:47is like a an area that is like really
29:50really important. There's just so much
29:53to be explored in this because again,
29:55there there is no It's kind of
29:56open-ended, right? There's no boundary
29:58in how good the models can become with
30:01test time scaling and then bringing that
30:04back to the process of training the
30:06model or fine-tuning the model to become
30:07better is very exciting, and that's kind
30:10of the self-improving piece that we are
30:13very excited about.
30:16Um so, a little bit about reasoning
30:18models. Um
30:20in the language monkeys, what we were
30:21showing was the coverage. Like if we had
30:24access to a verifiers, how uh we we can
30:27see the kind of a log linear kind of
30:29scaling laws with the number of samples.
30:31Um
30:32and all OpenAI, when they released 01
30:36last September, um they showed a a log
30:40linear relationship this time with pass
30:42at one for these like hard hard Amy uh
30:47benchmark which is like I said a really
30:49difficult uh math questions. So, as
30:53they're show what they're showing here
30:54is that as they're increasing test time
30:56compute and this is in log scale
30:59um
31:00uh the the accuracy the pass at one
31:02accuracy of the model goes up and this
31:05is for test time alone. Uh previously
31:09uh it this has been shown for training
31:11but it's very interesting that this kind
31:12of scaling applies to test time as well
31:16um without changing the parameter count
31:18of the model. Um
31:22Um I'm going to walk you through like
31:24one a few examples of how this kind of
31:27reasoning or test time uh scaling
31:30uh kind of is
31:32works with models like O3 and or Gemini.
31:35So, one motivation here is that for
31:38difficult problems just like humans
31:41think a lot more, spend a lot more time,
31:43or consider many different strategies,
31:46uh thinking models do the same and they
31:50may use a chain of thought or other
31:52techniques to just go go about like
31:54solving a problem um
31:57and
31:58there are different steps to that. For
32:00example, um problem analysis is one step
32:04like the model first sees a problem and
32:06tries to analyze it. Um then it can do
32:09the task decomposition breaking a task
32:12into kind of simpler tasks that are more
32:15addressable. Um then there is the self
32:18uh evolution strategies where the model
32:21kind of try something, sees the
32:23feedback, for example runs some
32:26tests on the code or use a uses a
32:29calculator or just judges the answer and
32:32it can use that feedback to optimize
32:34itself. Um there's self-correction and
32:37there are alternative proposals if
32:39something doesn't work, the model can
32:41backtrack and kind of, um,
32:44try different approach. These are some
32:46key principle principles that the model,
32:50uh, probably at some point, uh, pieces
32:53of it were trained as part of the
32:55training kind of data set were curated
32:57by humans, but a big part of it is also
33:00the model kind of acquiring these skills
33:03during this fine-tuning process and RL
33:07like RL from synthetic data process that
33:10the model does. Um, here is an example
33:13of like how O 1 does analysis. Um,
33:17uh, the model is asked to write a bash
33:19script, um,
33:21uh, that takes a matrix and then
33:23outputs, uh, the transpose of the that
33:25matrix. The model starts with just like
33:28humans, like starts like thinking, what
33:32are the pieces important pieces here?
33:34So, the user is requesting a bash
33:36script, um, let's understand the input
33:39and output, uh, formats of this matrix,
33:42um,
33:43and and so on. So, the model just thinks
33:45itself,
33:46um, and amazingly this kind of like
33:50it's like a progression of chain of
33:52thought. This is like helping the model
33:54finding the answer, but the difference
33:56between chain of thought and this is
33:57that the model itself is
33:59producing this chain of thought.
34:02Um, there's of course task
34:03decomposition, like in this case for the
34:07the approach, uh,
34:09the model thinks about parsing the
34:11input, building the matrix as an array
34:13of, uh, as as an array of arrays, and so
34:15on. Um,
34:17there's also self-correction, which is a
34:19big important piece of, um,
34:21this is is an important piece here, and
34:23I don't know how many of you have seen
34:25that when trying with the models that
34:27you can see the thinking traces. Uh, and
34:30this is actually pretty common, like the
34:31model starts something and then in the
34:33middle says, "Wait, the correct for like
34:36there's something wrong, maybe. Um, or
34:39and maybe I need to fix that." And it's
34:42very, very cool that the model can do
34:43that itself. Um, so reasoning models
34:46when they came out, like models like 01,
34:49compared to a model like GPT-4o,
34:52which model, um, they tend to be better
34:55in, obviously, the reasoning task. For
34:58example, in math calculation, data
35:01analysis, programming and such, they all
35:05kind of outperform GPT-4o, but not
35:08necessarily in personal writing or
35:10editing text and so on.
35:12Yeah. Question?
35:15>> Do you think the gains due to this
35:18asking for a reasoning process due to
35:20actually the act of generating this
35:22reasoning process or just the act of
35:24asking the model to think or to use the
35:27composition at all? Like, for example,
35:29if you
35:31think step-by-step but don't generate
35:33out loud,
35:34like don't actually generate those
35:36thought processes which are there but
35:37without saying
35:39>> Yeah.
35:40Um,
35:42I repeat the question. Like, do you
35:43you're saying that is it the Do you
35:45think the reasoning models are becoming
35:47what they are is based on the Is it
35:50based on the loud way of thinking?
35:53>> I I think the question is that Is it
35:55because of the, um, reasoning process or
35:58is it because of things step-by-step?
35:59The chain of thought.
36:01Like, is it the data thinking difference
36:03or is it the the instruction? Um,
36:05>> It's I mean, so the data basically has
36:08made the model kind of become a
36:10generalized thinker, right?
36:13Like, the model can and the all of it
36:16All of these steps, like breaking down
36:18the problem, being able to backtrack,
36:20being being able to do analysis, all of
36:22these are these skills that the model
36:25has learned during this thinking
36:27optimization process that now generalize
36:30to other things and they're helping the
36:32model right now.
36:34>> So, I think the other way to look at it
36:35is as Ali presented repeated sampling.
36:37So, if you had just a base model right
36:40after pre-training, it would be able to
36:42do some reasoning.
36:44But, at the same time it will generate
36:45different kinds of reasoning chains and
36:47it doesn't quite know which one is
36:49correct. So, a lot of the
36:52what we will cover in train time or test
36:54time scaling really comes down to it
36:55learns which is correct. So, pass at one
36:58accuracy goes up which is what you were
37:00seeing in the reasoning models as
37:01opposed to like pass at K or coverage
37:04which is what you were seeing in the
37:05repeated sampling results.
37:07>> Yeah. And and one other way to think
37:10about like yes, these kind of like kind
37:12of
37:13deep thinking and things that the model
37:16generates is actually helping the model
37:19every time to solve a new problem.
37:21Um
37:22and although it's like very expensive
37:25for the model to generate those things,
37:27still like we want them to generate
37:30those because it
37:31leads to better answers.
37:34Um
37:35Any other questions? Yes.
37:37>> Do you use a different model for the
37:39reasoning stuff versus generating the
37:41final answer? Like maybe a smaller or
37:43like a probabilistic one just for the
37:44reasoning and the final answer is the
37:46large model.
37:48>> Um
37:49So, typically the reasoning capabilities
37:51have gone up with the model size. So, if
37:54if anything uh you would use the larger
37:57model for reasoning traces and then
37:59maybe you collect a bunch of reasoning
38:01traces and then have a smaller model
38:02summarize the answer.
38:03>> Yeah. So, the models like this is like
38:06at least currently, things may change,
38:08they like their own traces more. Even if
38:11they're the traces are coming from a
38:13better model, they tend to like their
38:16traces their own generated traces more.
38:19And we're going to have talk about
38:20swirl, this is one of the papers that uh
38:24>> the multi-step reasoning
38:24>> the multi-step reasoning that we talked
38:26about. What if we bring a different
38:28model, not in the context of generating
38:31reason traces, but in in the context of
38:33like evaluating and giving feedback. And
38:36we we're going to see that as well. But
38:38models, surprisingly or not
38:41surprisingly, like their own traces a
38:42lot more.
38:44>> For reasoning models, how are they
38:47taught to reason? Is it a hardcoded set
38:49of sequences of or taught to do traces
38:51or they somehow fine-tuned to say, "You
38:54need to do this much or this kind of
38:56thing that will cause the traces to
38:58generate."
38:59>> So, I don't think there's a published
39:00piece of work that really covers this
39:02bit. Uh
39:03>> It's both, yeah.
39:05>> But it's it's a bit of both. At the same
39:06time, the base model did have the
39:09thinking capability to begin with,
39:10right? It would go do in some go do some
39:13amount of thinking. One thing that we
39:15will cover in the classes this notion of
39:17outcome rewards model and process reward
39:19model and how you can use that feedback
39:22to get the model back.
39:25>> Yeah, there's always some bootstrapping.
39:27Like, okay, here are different ways.
39:29Like, chain of thought itself, which is
39:31heavily used in the instruction tuning
39:33of data set,
39:35tell shows the model some ways of like
39:37how to think.
39:39And there could be like some templates
39:41showing
39:42for the model fine-tuning. But
39:45the things that the model, if you just
39:46ask them any ask it any question and it
39:48comes up with that, there's a whole lot
39:50of generalization that. So, models have
39:52gone way above like what the data or
39:55instructions that are used for training
39:57them.
40:00Yes.
40:01>> Uh can you repeat some of Is there a way
40:03to make the number of samples dependent
40:05on the problem
40:07>> Please speak up.
40:08>> The number of samples, can they be
40:10dependent on problem difficulty?
40:13>> I don't think there's a published piece
40:14of work that does that. Uh but
40:17yeah,
40:18>> we just So,
40:20So, when So, there is
40:23follow-up work to that where we you use
40:25a reward model and then you can use
40:27that. So, assuming the reward model has
40:29some notion of complexity, if you
40:30haven't solved it, it can guide like
40:32more sampling or whether it's repeated
40:35or parallel.
40:36Uh but that's definitely an interesting
40:39direction to explore.
40:41Yeah.
40:42>> [snorts]
40:43>> Okay, we need to get to the Oh, we
40:46haven't. Right.
40:47>> Yeah, let me cover these.
40:49>> Okay, so we you now know about large
40:52language models and then you heard about
40:54how they learned how to think and then
40:56they learned how to reason.
40:58So, what's next and why does this Why is
41:00this course relevant? So, a lot of what
41:02this course is about is that large
41:04language models as chatbots or as
41:07reasoning models are basically still
41:08single turn like or or just like in the
41:11chat format. So, they're not
41:12accomplishing a task for you. They're
41:14fun to interact with, but they're not
41:16necessarily accomplishing a task for
41:18you. What has happened this year
41:20almost in the last few months and it's
41:22been surprising is that
41:24agents like Claude Code or Deep Research
41:26have really enabled people to do
41:28real-world workflows.
41:30So, they're agentic workflows which can
41:32achieve tasks that you ask them to do
41:34end-to-end. So, for example, if you want
41:36to go research and figure out where
41:38should I
41:40rent a home perhaps for the entire year
41:44for if I want to take a class at
41:46Stanford.
41:47The model would actually be able to do a
41:49whole bunch of analysis and like go look
41:51at a lot of different websites and
41:53actually summarize the results and give
41:55you pros and cons of different places.
41:57This used to not be possible before. So,
41:59this
42:00will cover a little bit of how that that
42:01becomes possible and how this course
42:03allows you to learn about that. And then
42:05Claude Code for example, any of you are
42:08Codex which is the coding agent from
42:10OpenAI, if you're using that, what you
42:12will see is that just by giving
42:14instructions in English, you can modify
42:16files or you can like figure out test
42:18cases and and whatnot. So, it really
42:21uh is become a coding productivity tool
42:23in in the day-to-day workflows for
42:25software engineers at this point in
42:26time.
42:28So, what what is the transition from LLM
42:30to agents? So, as I was giving you the
42:32example of Deep Research and of uh
42:35coding agents, basically the goal
42:38basically now the uh model can be given
42:41a goal and it will plan out the steps,
42:43then it will go interact with uh the
42:45environment, and based on the feedback,
42:48it will correct its steps until it
42:49achieves the goals or it will come back
42:51and say, "I cannot achieve the goal."
42:53So, this notion of like having some
42:54notion of a goal, uh taking actions
42:58towards that goal, getting the feedback,
43:00and then deciding when to stop, that's
43:03what makes agents uh different from the
43:05chatbot times of what you were doing
43:07before. And um
43:10this also might require interacting with
43:12tools uh which are external to what the
43:15model itself is doing, getting some
43:17inputs from there, but it still stays on
43:19track of like whatever task it's uh
43:22choosing to accomplish, which might mean
43:23that it has to have some form of memory
43:26to keep track of the task that it's
43:28trying to accomplish.
43:30So, what has been uh accomplished? So,
43:32like in in in several cases in simpler
43:34cases, you can achieve these uh um
43:37these end-to-end goals like Deep
43:39Research can like be accomplished
43:40end-to-end. But, in most scenarios,
43:42you're still having very static
43:44workflows. So, today what exists is
43:46closer to agentic workflows where you
43:48have um some sort of an input where
43:50you're giving a model the goal and then
43:52uh
43:53what this slide is showing you is that
43:55one model is perhaps uh giving the
43:57output like a solution and then another
43:59model is judging it and then deciding
44:01based on that whether the solution
44:03should be accepted. So, that's like one
44:05orchestration framework that's possible.
44:07Uh Deep Research will be more of the
44:09second one where you're basically
44:10generating
44:12you're calling the LLM on multiple
44:15possible inputs and then aggregating
44:17these
44:18things to get the output which will be a
44:20summary of the deep research results.
44:22So, this is a very
44:23cartoonish abstraction of what agentic
44:26workflows might look like and you can
44:27generalize from there, but instead of
44:29having like very open-ended loop that I
44:32was showing you here where you're
44:34supposed to go take an action from the
44:35environment and come back and do some
44:37feedback.
44:38Um,
44:39it's easier for open-ended problems to
44:41like construct this graph by hand of
44:43like how a human would do it and then
44:44like get this feedback perhaps from an
44:46LLM which is the LLM evaluator here. So,
44:50a lot of the real world workflows still
44:52have this paradigm, but in in certain
44:54cases we are starting to see signs of
44:57life for what's possible in this slide
44:59encoding being one and research being
45:01another one.
45:03So, typically the workflows will have
45:05things like LLM calls. So, you would
45:07basically call an LLM with an
45:09instruction or some form of like input
45:11and then you're asking the LLM to give
45:13you an output. It will have some form of
45:15verifiers. We'll cover verifiers in a
45:16whole lecture by itself. It will have
45:18some form of critics or judges which is
45:21effectively LLM as a judge paradigm.
45:24There might be tool calls. For example,
45:26for deep research you have to actually
45:27go search the web to figure out what
45:30exactly is the content that you should
45:32be looking at. So, there might be tool
45:33calls of that form or
45:35getting some value around
45:38what exactly is the weather.
45:40Um,
45:41and similarly search would be another
45:43tool call. And then you would
45:44orchestrate these in in some form of a
45:46workflow. So, a simplest workflow here
45:49would be prompt chaining. So, similar to
45:51reasoning models where the task gets
45:53decomposed into subtasks, prompt
45:55chaining would simply be like you have
45:56chained a bunch of different subtasks
45:58that you're supposed to go accomplish to
46:00achieve the end-to-end goal. There might
46:02be routing. Routing works for complex
46:04tasks where you say, "Okay, if this is
46:07very complex, then go do this more
46:09complicated set of LLM calls, and if
46:11it's less complicated, then do this
46:13simpler workflow."
46:14Um there might be parallelization where
46:16you like deep research is one example
46:18where you you can have multiple LLM
46:20calls work simultaneously on researching
46:23different keywords that you give it as
46:25input to the LLM, and then uh finally
46:27you aggregate the output, or you might
46:29break the task into independent
46:31subtasks, and then uh combine uh the
46:33solution. Or you might have this notion
46:36of orchestrator where you basically are
46:38actually using some sort of a
46:41In simple words, LLM managers where
46:43central LLM is actually doing the
46:45planning. So, in Claude code, you
46:47actually start to see that that there is
46:48some notion of like a plan that it comes
46:50up with, and then based on that plan, it
46:52will like make subsequent LLM calls.
46:55You might have an evaluator or a judge.
46:57So, instead of like getting feedback
46:59from real-world like user or from some
47:01sort of a actual running a unit test, it
47:03actually might just use LLM as a judge,
47:05and we will have uh some homeworks that
47:08will cover this aspect. And then you
47:09might have verifiers. Verifiers are
47:11things where you can actually verify the
47:13output. So, in code for example, if you
47:16run the code, how do you know that this
47:18code is correct? So, typically as
47:20software developers, we write unit
47:21tests. Uh so, similarly, verifiers might
47:24be running some kind of unit test to
47:26check whether what LLM generated is
47:28correct. And in domains which are
47:30verifiable, uh math, code, um and other
47:33domains that are more rule-based, uh
47:35this verification is a good way to give
47:37feedback back to the model so that um it
47:40can correct its steps.
47:42So, most of these uh
47:44workflows mean that the
47:46uh the LLM needs to be better at
47:48planning. It needs to be better at
47:49multi-step reasoning, and it needs to be
47:51better at self-improvement. Like when it
47:53makes mistakes, it needs to be able to
47:54correct itself. So, effectively, these
47:57are sort of new paradigms that the
47:59current set of LLMs just with reasoning
48:02were not quite accomplishing, and these
48:04are some of the topics we do plan to
48:06cover in subsequent lectures.
48:09And just to drive the point home around
48:12coding agents, so this is a very simple
48:14example. This is actually
48:17I should have updated the slide to four
48:18cloud code, but basically, what this is
48:20showing is that you have an LM agent,
48:22and
48:23it's interacting with the computer
48:25these days just with the terminal. You
48:27give it an instruction where you might
48:28tell it that you want to implement a
48:30task,
48:31and then
48:32has some navigation of repository,
48:34searching of files, all of these tool
48:36calls around like viewing files, editing
48:38lines, and then it will go execute a
48:40bunch of commands in the terminal, and
48:42based on the output, it might be like,
48:44"Okay, I need to go edit this other
48:46file, or I need to go look at this other
48:48file." So, if you look at this loop,
48:50this very much mirrors uh
48:52what was So, this was not quite reliable
48:54last year, and it's just starting to get
48:56reliable in what we're seeing in the
48:58realm of coding agents now.
49:00Yes.
49:11I mean, I think the paradigm is very
49:13much the same. It's it's mostly a matter
49:15of more powerful models, and then better
49:17RL.
49:19RL with verifiable rewards is working.
49:22The train time scaling is working well.
49:37I mean, it's computer data, right? One
49:40of those two, but like just also, once
49:42the models start to get better, there's
49:44the self-improvement loop that kicks in.
49:47Uh because you can now generate tasks,
49:49and the tasks become more reliable. So,
49:51in the published pieces of work, if you
49:52look at say Code Monkeys, which we coded
49:54last year,
49:56if you can generate unit tests
49:58for whatever code the model generated
50:00and then see if the
50:02generated unit tests are
50:05making things better, then that that
50:06that's a very good way to verify things,
50:08right?
50:09>> Yeah, like model-based. It seems like to
50:12some extent, if we have good verifiers,
50:15um
50:16we can make the model generate the the
50:19right code.
50:21Um but the question is how do we do
50:23that? And there are ways to do it. And
50:25we
50:26we would be happy to
50:28for you guys to take research projects
50:30in that domain, too. But obviously, it's
50:32an active research area, but there are
50:35ways to make improvements that we can
50:38discuss.
50:38>> what you're alluding to is this notion
50:40of generator-verifier gap. So, it's easy
50:42for models to generate a whole bunch of
50:44nonsense or sensible set of like
50:46reasoning traces or like useful set of
50:48content. But at the end of the day,
50:50whether that's useful or not, we need
50:52feedback loop for that. And if you're
50:54creative writing, how much feedback can
50:55you get? So, human feedback ends up
50:57becoming a bottleneck. In domains where
50:59you can have good feedback, that's where
51:01it's possible to continue to improve the
51:03model. And that like but where getting
51:05robust verification, a lot of robust
51:07verification is hard. So, uh Azalia will
51:10cover a paper that she did in her her
51:11lab about like how to combine verifiers.
51:14And but what verification continues to
51:16be one of the bottlenecks in this space
51:18to make them better.
51:20>> Yes.
51:21>> Um just like intuitively, like in
51:23pre-training, you get
51:25abilities to find looking at the same
51:27logical abilities.
51:28>> Mhm.
51:29>> So, just like to me, intuitively, I'm
51:30trying to figure out why there's a
51:31significant drop.
51:33>> Mhm.
51:33>> Because like we're sort of dividing up
51:35what they know.
51:36>> Mhm.
51:36>> And we have logical abilities that I
51:39guess shouldn't exist
51:40>> Mhm.
51:40>> in pre-training.
51:41>> Mhm.
51:42>> Um
51:43So, yeah, I I guess like I'm I'm not
51:46completely resonating with like
51:48>> I I think this is still an active area
51:50of research. So, there is different set
51:52of opinions around what really improves
51:54the model. Is it RL or is it like the
51:57the pre-training diverse data by itself?
51:59And I I don't think there's single point
52:02of consensus at this point in time. Both
52:04processes help. Um what you just said,
52:07uh which I'll repeat for the class, is
52:09that if pre-training is the place where
52:12uh the model after repeated sampling
52:13should be able to at least have one
52:15solution correct out of if it was
52:16generating a large number of samples,
52:18then uh this notion of giving feedback
52:20should improve the pass at one accuracy,
52:22but should not improve the model. Why is
52:24it like such a big jump? But I think um
52:27that whole loop is not completely well
52:29understood. It's like the first signs of
52:31life and it starts to get
52:32commercialized, but I think there's a
52:34lot more research still open in this
52:36area.
52:37So, that's only one uh way of thinking
52:39about it. There are other um at least
52:41some signs of life that say that you can
52:44continue to do RL and then that will
52:46continue to improve the model. Yeah.
52:50>> Okay.
52:51>> Yeah.
52:52So, uh so this is basically covering a
52:55lot of the same set of points, but uh
52:57one abstraction that is worth taking
52:59away from this particular chart is that
53:02even when you give the model a goal, it
53:04has to clarify the user intent. So, it's
53:06like not always obvious to it what the
53:08user wants. And then it might go search
53:10for relevant files and then whatever um
53:13set of uh actions it chooses to take,
53:16often times it needs some form of
53:17verification. In this particular case,
53:19it's based on um
53:21passing the tests and it might actually
53:22generate the tests that needs to pass,
53:25which um
53:27which is also uh mentioned here. But the
53:30the key idea is that
53:32if you give it a task, how does it come
53:33back and how do you know that it's going
53:34to complete what you asked it for? Often
53:37times the users will not specify the
53:39problem well enough. So, asking
53:41clarifying the user intent so that it
53:43knows what to go look for and how to
53:45verify it starts to become important and
53:48in models like 03, I think they have
53:49seen a lot of these traces end to end.
53:51So, they are able to like do the
53:52planning, the reasoning, the multi-step
53:54reasoning and like come back with the
53:55conclusion, but
53:57a lot of this is needed for um
54:00completing tasks end to end or having an
54:02end to end goal.
54:03And where this has been super useful is
54:06if you have repeated repetitive tasks
54:08like code migrations or version upgrades
54:10or if you need to restructure the code
54:12base.
54:13Or if you have tasks that involve data
54:15engineering where you have to basically
54:17do some sort of extract the data and
54:19then do some sort of clean clean up on
54:21the data or if you have to do some sort
54:23[snorts] of data warehouse migration.
54:25So, a lot of this work tends to be
54:27extremely
54:28repetitive and it's much easier to just
54:31delegate it to coding agents.
54:33Same for unit tasks often times like
54:35much easier to like start to give it to
54:37coding agents.
54:38Another area where agents have become
54:41extremely prevalent is in customer
54:43support. It's one of the most thankless
54:45jobs that
54:46when when you are on the customer
54:48support side of things you have to do
54:49and using LLMs to do that definitely
54:51makes
54:53streamlines the experience in
54:54interesting ways. So, one simple set of
54:56examples ends up being like you can use
54:58them to do live transcriptions. So, that
54:59gives you a very nice record and and
55:01you're seeing that even for meetings
55:02these days.
55:04Another simple example is that you can
55:06have knowledge assist. So, if you have a
55:08database of information, the customer
55:11support agent does not need to know
55:12everything. They can consult with the
55:14LLM and and get an answer and surface
55:17the relevant article which is better
55:20than having just search
55:22like an index and search by itself.
55:25Smart replies. So, in in chat often
55:28times this has existed for a while, but
55:30you can use an agent to like give chat
55:33responses and then having a call summary
55:36can help you um
55:38you can use the call summary to really
55:40improve the customer experience by
55:42itself as well. So, overall, I think in
55:43the customer support area, there are
55:46[snorts] multiple companies that are
55:48going after this area, but uh
55:50there are different segments of the
55:51problem that can be addressed using LLMs
55:53and they have been super useful and then
55:55there's also the end-to-end stuff that's
55:56starting to happen here.
55:59And then the third example that we will
56:01use and actually some of the homeworks
56:03is that if you have very complex topics
56:05and you want to provide a comprehensive
56:07report, um
56:08earlier it used to be that you had to do
56:10a literature review and then you had to
56:11summarize each paper and then you had to
56:13synthesize these things and these days
56:15you can just give it to an LLM and
56:16actually do it for you and um study
56:19maybe more articles than you would. So,
56:21typically, if you give it an example
56:23like uh say 2022 Winter Olympics opening
56:26ceremony, it will identify what
56:28references to go look at, then it will
56:30construct an outline of like, okay,
56:32these references are relevant or not,
56:35and then summarize each references as to
56:38what is the relevant content and then
56:39combine them to create a full-length
56:41article, um which is
56:44uh very impressive in certain cases and
56:46and you'll get to try it in one of the
56:47homeworks.
56:49And then even uh more uh forward-looking
56:52um these research agents are starting to
56:55be used for uh as AI scientists. So,
56:58basically, they're starting to assist
57:00scientists in say uh solving math
57:02problems or solving science problems.
57:04So, here the LLM is used as a
57:07brainstorming thing to come up with
57:08ideas to the idea generation phase, then
57:11the experiment iteration phase, it might
57:12actually help you iterate on the
57:14experiments that you are hoping to work
57:17on. This is from the AI scientist paper.
57:19And then in the paper write-up phase, it
57:21will help you improve uh
57:23the the paper write-up by itself.
57:27And what's interesting uh is that
57:31even even though these models
57:32hallucinate just I just the notion that
57:35they can come up with so many different
57:36set of ideas
57:38as an AI scientist it might actually
57:40brainstorm or like give you ideas
57:42outside what you would have thought if
57:44you had just like taken a bunch of
57:45courses or or if you're a researcher and
57:47you've been in the field for a long time
57:49sometimes the reading the web allows
57:52these LLMs to come up with ideas that
57:54are way outside the box and can be a
57:55very good brainstorming um
57:58way uh so these have been super useful
58:00for in the AI [clears throat] scientist
58:02style of work.
58:04Okay, so
58:05let's move to course logistics but
58:07before that do we have any questions?
58:09Yes.
58:11>> To clarify so is reasoning and chain of
58:14thought baked in training or is it just
58:17prompt engineering that was accidentally
58:19discovering these kind of capabilities
58:20in larger models? And if so are there
58:23other emergent behaviors besides our
58:26reasoning and chain of thought that
58:28you think might be discovered?
58:30>> So uh
58:32I mean it was not baked in by
58:34by design it was it was basically
58:37discovered as in like it was
58:39I mean we gave it hard problems and then
58:41we saw that it was basically by having
58:43reasoning chains it was doing better. So
58:45the GSM8K was the first paper that kind
58:47of showed signs of life of this and then
58:50with larger model like Palm we actually
58:53saw that this was a very big deal and
58:55one of the examples there was that it
58:57could explain jokes that which was very
58:58impressive and then from there the
59:00reasoning models have emerged. So it was
59:02not
59:03but it has read all of the web so it's
59:05it has definitely seen data which is
59:07like more
59:09methodical and systematic.
59:11>> But the reasoning models are trained to
59:13be reasoning more and more, right? So
59:16it's they didn't it entire reasoning is
59:19not an emergent behavior. The reasoning
59:21models they're trained to be to be
59:24thinking but like the models are
59:26convergent so the models are going to be
59:28trained such that they know when to they
59:30need a lot of reasoning and when they
59:32don't and and to generate the answers.
59:35But chain of thought originally was an
59:38emergent behavior. They noticed that oh,
59:40if we
59:41explain things the model, it's better.
59:46>> of emergent behaviors in large models.
59:48>> I don't think it's I would see it as
59:50emergent behaviors per se, but I think
59:52as I was mentioning, we're usually going
59:54looking for certain things, right? So in
59:56the agentic workflows, what we're
59:57looking for is planning,
59:59which is a form of reasoning. We're
1:00:00looking for multi-step reasoning, which
1:00:02we will cover in class as well. And then
1:00:04we're looking for self-improvement or
1:00:06self-correction. So all of these
1:00:07capabilities would be nice to have in
1:00:09like what gets the models there is a set
1:00:12of questions that are worth like there
1:00:14are papers on this kind of thing.
1:00:16>> Like self-correction,
1:00:17backtracking, things like that is like
1:00:20yes, you could call it emergent, but you
1:00:23could also say think about it that they
1:00:26have seen this kind of behavior and it's
1:00:28kind of like it's reinforced in the way
1:00:30that they're fine-tuned. So it's
1:00:34Yeah, it's hard to say.
1:00:38Okay. Um so let's go through the the the
1:00:42logistics for the class. Um here is a
1:00:45list of all the amazing topics that
1:00:47you're going to learn about. Um they are
1:00:50um
1:00:52yeah.
1:00:53This is very cool. [laughter]
1:00:55>> I think the main thing to remember here
1:00:57is that the overall theme stays the same
1:00:59and then we'll also have guest lectures
1:01:01around from folks in Frontier AI Labs
1:01:04and covering things all the way from say
1:01:07how have has how has post-training
1:01:09evolved or say even multimodal agents
1:01:11and in robotics. So it's it's a it's
1:01:15going to be a mix of lectures and guest
1:01:17lectures and
1:01:19and of course you have project
1:01:20presentations.
1:01:23>> So, here is some logistics that you all
1:01:26know the the pre-record there. Just make
1:01:29sure that you you are comfortable with
1:01:31these pre-reqs as you
1:01:33uh register then in the class. So, we
1:01:36have the external website, but uh
1:01:39make sure that you're
1:01:41uh you check Canvas. That's where we
1:01:44send out the latest updates at all
1:01:45times.
1:01:46Um and um
1:01:49we will also upload the We try to upload
1:01:51the lectures before we start every every
1:01:54in every class so you have access to the
1:01:56lectures.
1:01:57Uh and we will also have We already have
1:01:59the due dates for all the assignments
1:02:01and all the projects, so you can check
1:02:02it out.
1:02:04Um
1:02:05So, this quarter we're going to have
1:02:07three homeworks. That's a difference
1:02:09from between this class and that's one
1:02:11of the differences. So, we have one more
1:02:14amazing homework for you all that we've
1:02:16designed and the TAs have done a great
1:02:18job
1:02:19and that's helps you
1:02:21um hopefully learn
1:02:23uh learn these topics better and more in
1:02:25depth.
1:02:27Um
1:02:27And then we also have a course project.
1:02:29So, the course project is where you can
1:02:31kind of unleash your creativity and like
1:02:34your your way of like building agentic
1:02:37systems or going deeper into a question
1:02:41uh and and designing experiments around
1:02:44it and see what works and what doesn't.
1:02:46Um we will provide some um kind of
1:02:50examples or suggestions, but it could be
1:02:53completely on you how to design this
1:02:55project. We'll have some examples from
1:02:57last year like the successful projects
1:02:59from last year as well.
1:03:01Uh it's going to be uploaded to
1:03:04um Canvas or Actually, we'll display
1:03:07them on the website.
1:03:08>> Oh, on the website. Oh, cool.
1:03:09>> Yeah. The public website, so you can all
1:03:11see that. Um
1:03:13So, yeah.
1:03:14>> the course project, I think the main
1:03:16thing to remember is that uh
1:03:18we will have API credits and you can
1:03:20work in teams of two to four people.
1:03:24Right?
1:03:24>> Yeah. Or
1:03:25up to four, I
1:03:27>> to four. Okay.
1:03:28>> It can be one if you really want that.
1:03:30We we we suggest you team up with
1:03:33others.
1:03:34Um at the very least, you have more
1:03:36credits collectively, so you can run
1:03:38more experiments. But then you might
1:03:40also uh find a friend collaborator along
1:03:43the way and you can um
1:03:46um do something bigger. Um
1:03:48and we are going to have this um
1:03:52What is this? The course project. Yeah,
1:03:54we're we're going to uh we're going to
1:03:55let you know about some
1:03:57uh some of the previous years and some
1:03:59ideas that you can take. Um here are
1:04:01some course project examples that are
1:04:04acceptable. For example, a new
1:04:06evaluation data set or a new benchmark.
1:04:09Um you can design a project around the
1:04:13reliability of a of an agentic agentic
1:04:16system that already exists. You can take
1:04:18a benchmark that exists and try to hill
1:04:20climb on it with whatever idea great
1:04:23idea that you have. Um and then or you
1:04:26can just be You're going to have a whole
1:04:28lot of papers covered in this course and
1:04:30we have we are showing all of that to
1:04:32you. They're all on the website. So as
1:04:34you as we give these lectures, we kind
1:04:37of expect you to also have read those
1:04:39papers or read the papers along the way
1:04:42and develop better and deeper intuitions
1:04:44on how these methods work. And you your
1:04:47project could always be improving those
1:04:49or like questioning one of the decisions
1:04:52that they have made and trying to uh
1:04:54change that or uh or evaluate that. Um
1:04:58Negative examples are like a survey
1:05:01paper or like We just don't want We want
1:05:03something researchy here. So we just
1:05:05don't want like just an app that you put
1:05:08together and you just show us something.
1:05:09It has to be like Here is like, here is
1:05:13the hypothesis.
1:05:14Here is the question that we wanted to
1:05:16answer. Here is the type of like
1:05:18improvement we wanted to see or the type
1:05:20of like properties we wanted to analyze
1:05:23and then we have built that, right? We
1:05:24want something something that
1:05:27something more than live coding
1:05:29basically in this.
1:05:31>> Um I mean in terms of milestones, I
1:05:33think that it's worthwhile to remember
1:05:35that you want to start early.
1:05:37So it's you need to have a project
1:05:39proposal
1:05:41somewhere around I think we we put it
1:05:43early October.
1:05:44>> Yeah.
1:05:44>> And then uh
1:05:45>> It's on the website, yeah.
1:05:46>> It's on the website and then for the
1:05:48midterm project presentation, we do want
1:05:50you to have some progress. So 2 weeks
1:05:52after the project proposal, we do expect
1:05:54you to have like So start thinking about
1:05:56like what experiments you want to run by
1:05:58the time you get to the project proposal
1:06:00so that
1:06:01you actually go run them for the midterm
1:06:03projects. And so we do expect you to
1:06:05have made some progress in the midterm
1:06:06project presentation. It should not just
1:06:09be
1:06:10here is a proposal and then the final
1:06:12report has a lot of weight and then the
1:06:13final poster presentation uh will be the
1:06:16end of the quarter.
1:06:20>> Um yes.
1:06:40I
1:06:41I think so. If you add to it, right?
1:06:46>> See this one name I want to name it in
1:06:48the past study, right?
1:06:50Yeah, yeah, that's fine.
1:06:51>> I I think it's
1:06:52>> I mean as long as you're doing actual
1:06:54work and it's not just like reusing the
1:06:55work from
1:06:56>> [laughter]
1:07:00>> Um
1:07:01>> We had We had students publishing papers
1:07:05out of their projects
1:07:07uh in the last quarter. So in the last
1:07:09last time we taught the class. So that's
1:07:11something you could it could be one of
1:07:13you or many of you this time again
1:07:16turning your research project, working
1:07:18more on it and then
1:07:20turning it into a publication and
1:07:24conferences.
1:07:26And so here is a save the date for for
1:07:31our poster presentation. It's going to
1:07:33be December 12th 4:00 to 6:00 p.m. So
1:07:37we we would want you to be there and
1:07:39present the posters. We'll have people
1:07:41from industry joining and you can that's
1:07:44where you can brag about what you've
1:07:45done and just meet new people. Here is
1:07:49the course grading rubric. So we have
1:07:53three homeworks. That's 50% of your
1:07:56of of your entire grade and then another
1:07:5950% is on the project. And the duration
1:08:02of these are kind of like a line with
1:08:03how much how much the grading grading is
1:08:07allocated to each of these. And we of
1:08:11course expect you to honor
1:08:15the honor code.
1:08:17And let's see.
1:08:20Office hours will be posted on Canvas.
1:08:23You can ask your questions. We really
1:08:25really encourage you to ask questions on
1:08:27Ed Stem
1:08:28and make it public so others probably if
1:08:31you have a question
1:08:33it's very likely others have the same
1:08:35question. So please like do broadcast
1:08:37this to everybody.
1:08:40Ed Stem is
1:08:43again you can ask questions and then
1:08:45GradeScope is where you submit your
1:08:46project milestones and everything else.
1:08:50Yeah, the homeworks and everything.
1:08:52We have the following late policy that
1:08:54we think it's kind of like
1:08:56between the generous kind of like
1:09:00So, I I am generous side of things and
1:09:03because the class is
1:09:05big this quarter, we really can't make
1:09:08any exceptions. So, try to use these
1:09:11kind of late days wisely.
1:09:14Uh
1:09:15Audits are not allowed, but we will have
1:09:17the the course
1:09:20the we will have the videos videos on
1:09:22YouTube eventually. Eventually.
1:09:26And that's it. Any questions?
1:09:32No?
1:09:33All right.
1:09:34>> Okay. Thanks everyone.

About this transcript

This page contains the full transcript of YouTube transcript (6YnLB0XbTnI) , generated from the public captions YouTube serves with the video. The transcript has 11,603 words across 1,840 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.