CLAUDE AI ile sıfırdan Ajan Kurma — Transcript
Full transcript
- 0:00Um so we'll introduce ourselves. So I'm
- 0:03Akansa. I have been working in large
- 0:05language models for a while and I'm
- 0:06adjunk professor at Stanford and I'm
- 0:08also in research uh at a startup called
- 0:11reflection AI.
- 0:12>> Hi I'm Aalia Mir Hoseni. I'm a I'm an
- 0:16assistant professor in the CS department
- 0:19and um uh Akansa and I met each other
- 0:23when we were at Google Google Brain.
- 0:26Google Brain. Yeah. Back in the day. and
- 0:29then Google deep mind and I also worked
- 0:32on cloud at entropic and gemini at
- 0:34Google uh deep mind and we are so
- 0:37excited to uh teach this course again
- 0:40this is the second time we are teaching
- 0:41of course we have made some updates both
- 0:44to the lectures and also the way we are
- 0:47uh kind of conducting everything during
- 0:49the the for the course uh and we are we
- 0:53were just so amazed by the amount of
- 0:54interest uh for people to join this
- 0:57course So uh we are excited to have you
- 1:00here.
- 1:01>> Uh the course website is at
- 1:02cs329a.stanford.edu.
- 1:05We have updated the website in terms of
- 1:07all the lecture materials and and the
- 1:10papers that you'll be reading as well as
- 1:11uh what's the homework schedule to
- 1:13expect um the project proposal and
- 1:15project structures as well.
- 1:18So in today's lecture we'll start with
- 1:20an overview of large language models uh
- 1:23just uh how the scaling trends have
- 1:24evolved over the last uh five years and
- 1:27then we'll focus more on what we will
- 1:29cover in this class and then from there
- 1:32uh in the second half of the uh this
- 1:34lecture we'll cover the course logistics
- 1:36and that part you should pay a lot of
- 1:38attention to because that influences
- 1:39your grades.
- 1:43So let's take a look at an overview of
- 1:45the trends for large language models. So
- 1:48one of the aspects that has been
- 1:49mind-blowing since uh GP3 came out is
- 1:52that
- 1:54uh there's a uh that as you scale up the
- 1:57number of parameters in a model uh they
- 1:59get better and what this means is uh
- 2:01that language models like BERT and T5
- 2:05were good but as we increase the number
- 2:06of parameters they got really better.
- 2:08they they perform much better and what
- 2:10this led to is something called a
- 2:12scaling loss for large language models
- 2:14uh which are used in building the base
- 2:16model for the pre-trained base model and
- 2:20uh as you increase the amount of compute
- 2:21so there are three graphs here as you
- 2:23increase the amount of compute on the
- 2:24x-axis the loss function on the y-axis
- 2:28goes down and and the more compute you
- 2:31have put in the more uh the test loss
- 2:34goes down which leads to a better model.
- 2:36So that was one big aspect that led to
- 2:39GP3 and then all the subsequent chat GPT
- 2:42palm um Gemini and so on. And then
- 2:45similarly as you increase the data set
- 2:47size on X-axis uh the test loss goes
- 2:50down further on uh Y-axis. So that's
- 2:53another aspect of like another axis of
- 2:55scaling that leads to better language
- 2:57models. And uh the fourth one is of the
- 3:00third one is of course parameter count.
- 3:02So if you increase the number of layers
- 3:04in a transformer or the amount of
- 3:05parameters that are there in the
- 3:07language model that improves the loss uh
- 3:10value and that that leads to a better
- 3:11model and and this has been the
- 3:13foundation for a lot of the uh this has
- 3:15been the foundation for a long time uh
- 3:17until last year where this was starting
- 3:19to hit some kind of a saturation point.
- 3:24So from 2018 to almost 2024 um this
- 3:28graph is a little bit outdated but uh
- 3:31from 2018 to 2024 the at the model size
- 3:34has gone up um almost consistently we
- 3:37have been increasing the the size of the
- 3:38model so you started with something like
- 3:40B was 340 million parameters GBD2 was
- 3:431.5 billion parameters and then GB3 was
- 3:46175 billion parameters and then u say
- 3:49palm was 540 billion and then uh there
- 3:52was a bunch of subsequent models and
- 3:54GBD4 is estimated to be uh there which
- 3:57is like trillions of parameters. So you
- 3:59can imagine that there has been an
- 4:00exponential growth in just the model
- 4:02size for large language models. The
- 4:04large stands for that they're growing in
- 4:06size and that has been one of the key
- 4:09areas in which u the models have been
- 4:11getting better.
- 4:13So why do we care about the models uh
- 4:15getting bigger? What does this lead to?
- 4:17So as I was explaining that as the
- 4:20models get bigger uh they continuously
- 4:22improve in performance on uh natural
- 4:24language benchmarks on various other
- 4:26benchmarks reasoning benchmarks and so
- 4:28on and then other interesting aspect
- 4:30that I uh will show you in a moment is
- 4:33that they learn how to do few short
- 4:34learning. So earlier you have to
- 4:36fine-tune the model for specific domain
- 4:38but just by giving a few examples the
- 4:41model can follow that template and and
- 4:43reason in that same vein which makes it
- 4:46extremely easy to prototype uh things.
- 4:49And then the third aspect is that as the
- 4:52models become bigger they have emerging
- 4:54behavior where the capabilities like
- 4:55reasoning only emerged in larger models.
- 4:59So just to understand few short learning
- 5:01or zero short learning here. So if if
- 5:04the model is given a question like or a
- 5:07task like translate English to French uh
- 5:10it's able to give uh so this is called a
- 5:12prompt like it's given a task translate
- 5:14English to French and then it's asked to
- 5:15translate cheese to whatever is the
- 5:17French word if the model can predict the
- 5:19answer without uh actually being trained
- 5:22on the specific task then that would be
- 5:24called zeros short learning and then few
- 5:26short learning in addition to giving the
- 5:28in uh description of the task you're
- 5:30also giving a few examples in this
- 5:32particular case it's given translate
- 5:34English to French and it's given a few
- 5:35examples of the translation from English
- 5:38to French and then it's asked to
- 5:40translate cheese and it's able to
- 5:41translate that and you can see that this
- 5:43capability shows up in the large
- 5:46language models across a large number of
- 5:48tasks and that has been one of the key
- 5:49areas that GPD3 and pal and um other
- 5:53models enabled and then that uh
- 5:55subsequent innovations uh came from
- 5:57reasoning aspects of this. So on the
- 5:59reasoning side um as the models have
- 6:02become bigger they have this emerging
- 6:04behavior of reasoning which Aalia will
- 6:06tell us a little bit more about.
- 6:08So as the models become bigger not only
- 6:11we have this kind of a predictive
- 6:13scaling loss property that we kind of
- 6:16know how the loss functions going to go
- 6:18down as we throw in more compute data
- 6:20and parameters in the model. There are
- 6:23new kind of behaviors that appear in the
- 6:27model that they didn't have before and
- 6:28we never could predict that until we had
- 6:31these bigger models and saw these kind
- 6:32of behavior in them. And one of the most
- 6:36important
- 6:37uh one of those behaviors is the
- 6:40emergent is the chain of thought
- 6:42behavior. And for those of you who are
- 6:45not familiar with this, here is how it
- 6:47works. So in a usual prompt prompting of
- 6:51a model, we can ask a model in this case
- 6:53a a math problem um and then we can give
- 6:57it a oneshot example. Here is a math
- 6:59problem. Here is the answer. Now I ask
- 7:02you a similar but not exact not exactly
- 7:05the same math problem and the model is
- 7:08expected to answer that. Um and the
- 7:11model might be able to use the the
- 7:13oneshot example or it might not be. But
- 7:15we can do better here by not only give
- 7:19an example in but also provide the
- 7:22reasoning or the chain of thought on how
- 7:25we got to that answer. So in this case
- 7:27the question is a very simple question.
- 7:30Roger has five tennis balls. He buys two
- 7:33more cans of tennis balls. Each can each
- 7:36can has three tennis balls. How many how
- 7:39many total tennis balls does he have
- 7:41now? And instead of just saying 11, like
- 7:43we're walking the model through how to
- 7:45get to this. Um, Roger started with five
- 7:48balls. Two cans of three tennis balls is
- 7:50six. And uh, 5 + 6 is 11. Uh, and with
- 7:55this example, then the model has become
- 7:58familiar with the process to get to the
- 8:01answer and it can leverage that to
- 8:04answer new problems. Well, obviously
- 8:07this is a very simple problem. any like
- 8:091B parameter model nowadays can solve
- 8:11this doesn't need a an example or a few
- 8:15shot or a chain of thought but the chain
- 8:17of thought property itself is a property
- 8:19that is holding to this day and it's a
- 8:22very very important property for the
- 8:24reasoning models and all the thinking
- 8:26models and all the prog and many much of
- 8:29the progress that we have seen um in the
- 8:32past year or so
- 8:34now um this property again appears As we
- 8:38have larger and larger models, here are
- 8:40three models, lambda, GPT, and palm. And
- 8:44as you can see, the chain of thought,
- 8:45and this is some results on the math,
- 8:48some math data set. And these models
- 8:52when they're smaller like 8 billion
- 8:55parameter for lambda or uh G for GPT
- 8:58around 7 billion, they can't really
- 9:00benefit from chain of thought like it
- 9:03doesn't do anything for them. But as
- 9:05they become larger uh or for a palm
- 9:08model again at 8 billion it can leverage
- 9:11this chain of thought property and learn
- 9:14from the reasoning that is and the than
- 9:16the process that is provided in the
- 9:18context in order to solve problems
- 9:20better. Um
- 9:23and not only that there are other kind
- 9:25of abilities to solve new task as we
- 9:28increase the size of the models like for
- 9:30examples models can all of a sudden
- 9:33solve mod arithmetics better or solve um
- 9:37a word unscramble problem just by having
- 9:40this larger size and this property. You
- 9:43see that all of a sudden it appears at a
- 9:45certain size. Um and because of all of
- 9:48this, we still as in frontier companies
- 9:52and labs are still interested in pushing
- 9:54the the scaling laws and not only get
- 9:57the natural kind of progression but also
- 10:00we we will see more and more of these
- 10:03kind of emergent behavior that we are uh
- 10:05that is going to be very important.
- 10:09Now um a little bit of history here. So
- 10:12chat GPT was launched in uh November
- 10:162022. Obviously it was one of the most
- 10:19successful
- 10:21uh kind of [snorts] apps or software
- 10:23ever created. Uh the time to reach 1
- 10:26million users for it was um 5 days which
- 10:31is significantly
- 10:32um faster than any that many of the
- 10:35other famous um software that you can
- 10:38see there or services that you can see
- 10:39there. And there were pieces of of kind
- 10:42of innovation on top of just scaling up
- 10:46the parameter size of the model that
- 10:48were very important to make Chad GPT
- 10:51what it was and to make that kind of
- 10:53leaprog and over the previous models
- 10:57like GPT3 and so on. And two p two
- 11:00important piece here were um the
- 11:03instruction tuning and um the
- 11:06reinforcement learning from human
- 11:08feedback we which which we are going to
- 11:10briefly mention in the next slides but
- 11:12throughout this course we are going to
- 11:14learn more and more about them. Um so
- 11:18let's
- 11:20first and then um we're going to learn
- 11:22about all of these in more details but
- 11:25let's just take a step back and see how
- 11:28the process works. So the pre-training
- 11:30step is the step with the the beginning
- 11:34of any language model training. Uh
- 11:37that's the easiest step and we just
- 11:39train the model to predict the next
- 11:41token out of all sort of text and data
- 11:44and so on. And then this is the
- 11:46finetuning step that was a key
- 11:48differentiating factor for a model like
- 11:51chat GBT. Um and let's start with um
- 11:56uh the first aspect of fine finetuning
- 11:59which is kind of alignment with human
- 12:01preferences.
- 12:03So when a model is pre-trained it has no
- 12:06sense. It has seen all data on the
- 12:08internet on the in the books everywhere
- 12:11but it has no sense of like what is
- 12:13right and wrong and what is it just have
- 12:16this statistically knows about the state
- 12:18of the world but does it it doesn't
- 12:20exactly know things or know how to
- 12:21follow instructions. So the next step in
- 12:25training these models uh is the the the
- 12:28the step where we try to steer the AI
- 12:31models to follow the goals and
- 12:33preferences uh and values of humans. Um
- 12:38and this is still a big problem. It's a
- 12:40big issue. We we haven't like mastered
- 12:43it yet. But as the models become as we
- 12:46progress models are becoming better and
- 12:48much more uh kind of like uh powerful in
- 12:53the things that they can do. Um so it
- 12:55turns out that we can fine-tune the
- 12:57models to be aligned. For example, uh
- 13:00here are some graphs that show what if
- 13:02we take our base model that's just
- 13:05pretrain and fine-tune it for things
- 13:08like sensibleness or safety. And these
- 13:12are like these are data that we have
- 13:15collected. These are highquality data
- 13:17for kind of showing the model what what
- 13:20we mean but by what's safe and what's
- 13:22unsafe or what's sensible and what
- 13:24doesn't make sense. And these are like
- 13:26more cur like more highly curated data
- 13:29that we pass to the model for
- 13:31fine-tuning and to make the model more
- 13:33aligned with the values or the or the
- 13:35preferences of humans.
- 13:38Um
- 13:41so going back let's just walk through
- 13:44these process again. So
- 13:48the first stop when we take a
- 13:50pre-trained language model is uh
- 13:53fine-tuning with next token prediction
- 13:56which follows the same process as
- 13:58pre-training but this time on data that
- 14:01is much more higher quality. Uh for
- 14:04example, this could be data from books,
- 14:08creative essays, um data that companies
- 14:11pay a whole lot, millions of dollars or
- 14:13hundreds of millions of dollars
- 14:15potentially to buy this data that's
- 14:17really high quality and then we train
- 14:19the we fine-tune the model on those and
- 14:21the model becomes much better as a
- 14:24result of that.
- 14:26Um
- 14:27the the next step is instruction tuning
- 14:31and by instruction tuning what we mean
- 14:33is that we show the model again u tra
- 14:37traditionally in the past two years this
- 14:39in instruction tuning data is like a
- 14:42combination of human generated data or
- 14:44templates and then and then and the
- 14:46synthetic data. But the way this
- 14:48instruction t fine-tuning works is that
- 14:51we do have uh instruction and then
- 14:54question answer pairs uh where the model
- 14:58learns how to follow the questions and
- 15:01answers them. For example, here an
- 15:03instruction uh tuning data set could be
- 15:06about the weather like please answer the
- 15:09following question. What is the boiling
- 15:11point of nitrogen and then this is the
- 15:13answer. Um or we can have chain of
- 15:16thought uh fine-tuning where we um show
- 15:19the model how it can walk through a
- 15:21process to get to the answer and then uh
- 15:24we have the label we have the results
- 15:26over the here and this is like these
- 15:28pairs of like questions and instructions
- 15:30and answers and again this this data set
- 15:33is very um is is there's a lot of effort
- 15:37that goes into this data set and the
- 15:39quality of data set and the uh
- 15:41generality of it uh has a lot of impact
- 15:44on the quality of the model that we get
- 15:46at this stage. Um and after this point
- 15:49then the model becomes more and more
- 15:51like um like the models we see today
- 15:55like you can ask them questions you can
- 15:58go back and forth with them. They they
- 16:00have this sense of like how to uh
- 16:02perceive a question and how to walk
- 16:04through kind of uh some sort of like
- 16:07process to get to to an answer.
- 16:11And the next step after instruction
- 16:13fine-tuning is this process of um using
- 16:17human preferences and fine-tuning the
- 16:20model based on that. And this process is
- 16:22called RLHF or reinforce reinforcement
- 16:24learning from human feedback. Um the way
- 16:28this is different from the previous
- 16:30system is the way we create the
- 16:32objective function and uh create the
- 16:34data. So in this case instead of like
- 16:37just creating some supervised prompt and
- 16:40labels we are basically creating a
- 16:43reward model out of human preferences.
- 16:46So companies spend a lot of money uh
- 16:51asking humans, sometimes experts,
- 16:53sometimes just humans, normal humans, uh
- 16:57about questions and the answers
- 16:59generated by the model and we ask the
- 17:02the humans to rate them to see which one
- 17:05is correct and which one is not correct
- 17:07and that's how we can create some sort
- 17:09of a reward model here. Um and this
- 17:12reward model is instead instead of human
- 17:15then we can use the reward model in
- 17:17order to kind of like guide the
- 17:19parameters of our LLM towards uh
- 17:22generating answers that are in line with
- 17:25this reward model. So basically we want
- 17:27the the generations by the model uh to
- 17:30be such that this reward model says yes
- 17:32these are good generations. Um we're
- 17:34going to learn more about this here but
- 17:36I wanted we wanted you to have um the
- 17:39high level idea of what RLHF means. Uh
- 17:42and we can collect we can have different
- 17:45reward type different types of reward.
- 17:47For example, we could ask the model to
- 17:49reward a pair of answers based on their
- 17:52correctness or helpfulness or specity or
- 17:55harmlessness. all sort of reward
- 17:57functions that we can define and
- 17:58depending on how we want the models to
- 18:01be and what we care most about we can
- 18:04weight these different reward models and
- 18:06kind of use that in the RLHF process.
- 18:10So these step at a at a high level of
- 18:13course there's a lot of effort goes into
- 18:15that but like pre-training and then um
- 18:19fine-tuning on higher quality data
- 18:21instruction tuning and RLHF are were the
- 18:24core kind of components that made um a
- 18:29model like chat GPT over over something
- 18:32that existed over all the large models
- 18:35um like GPT3 and prior models. So that
- 18:38made a huge difference in how good and
- 18:41capable the models were. Um
- 18:45uh so so that was very cool up until
- 18:48last year or a year year and a half ago
- 18:52pre-training and then fine-tuning were
- 18:53the big pieces. Uh but since a year and
- 18:57a half ago, it turns out inference is
- 18:59also a frontier for making the models
- 19:02much more capable. Um so
- 19:06and that o opened like a whole set of
- 19:09whole lot of new research and directions
- 19:11and ways that we can make the model
- 19:13better. Um so let's think let's talk
- 19:17about it a little bit like this is a
- 19:19work we did in my lab last year uh last
- 19:24summer it was called large language
- 19:26monkeys um and it's kind of inspired by
- 19:29the infinite monkey theorem and it's
- 19:31related to in inference scaling um how
- 19:34many of you have heard of the infinite
- 19:36monkey theorem
- 19:38okay that's a good amount so the idea of
- 19:41the infinite monkey theorem uh what it
- 19:43says is that it's not proven, but what
- 19:47it says is that if you have a monkey and
- 19:49a kind of a typing machine and then we
- 19:52keep letting the monkey type all day
- 19:54forever, um after some time we will have
- 19:57the works of William Shakespeare in in
- 20:00the things that the monkey types. Um so
- 20:03we were inspired by that to name our
- 20:05project. Uh but the idea here was that
- 20:08let's let's have the LLM be the monkey
- 20:12and uh let's uh ask the model over and
- 20:16over again to solve a given input
- 20:19problem. Uh you can think of it as
- 20:22parallel sampling or parallel
- 20:24generation. So instead of asking the
- 20:26model once to solve the input problem,
- 20:27you can ask it many times. And also
- 20:30let's assume you have some sort of a
- 20:32verifier or selection mechanism. So you
- 20:35can pick which one of these generated
- 20:38responses were correct. Um and then you
- 20:42output that in your system that becomes
- 20:44the out the final output. So the model
- 20:47instead of generating one answer
- 20:48generates many many answers and then
- 20:50your verifier selects which one of them
- 20:52is correct and then outputs one of the
- 20:55correct ones. Um you can think of a
- 20:57verifier as say some unit test. If the
- 21:00problem is a generating code, the
- 21:02verifier can could be the unit test that
- 21:04we run against the code and we see which
- 21:07which one of the code generations passes
- 21:09all the unit tests and so on.
- 21:12Um the reason this is very this is a
- 21:16this can be a possible possibility here
- 21:18is that models are um do not are not uh
- 21:23there's some variance in the way the
- 21:25models generate responses. it's not
- 21:27deterministic and you can even control
- 21:29the variation in the model responses
- 21:32with this thing called temperature. So
- 21:34you can force the model or encourage the
- 21:36model to generate more different
- 21:38responses as you ask a question.
- 21:42Um and here now let's see some of the
- 21:46results of this repeated sampling for a
- 21:49bunch of tasks like math and coding
- 21:52benchmarks. we increase the number of
- 21:56samples per problem from one which is
- 21:59the normal one uh to 10,000
- 22:03and here we are showing the coverage or
- 22:05the fraction of problems that are solved
- 22:08by at least one of these samples and
- 22:11what we are seeing here is that um so
- 22:14the red line the red dash one was the
- 22:16GPT40
- 22:18model and while these blue and green
- 22:22lines which were the llama 8B, lama 3B
- 22:25and 7B were worse than GPT 40 model with
- 22:29one sample. But if we increase the
- 22:31number of samples from these models in
- 22:35all of these cases, they do better than
- 22:38the GPT40 model. And what it says is
- 22:43that it kind of seems like the model
- 22:45models already know a whole lot more
- 22:49than what you get out of them when you
- 22:51just ask them once, right? So this is
- 22:53the kind of the underlying property of
- 22:57inference scaling because it seems like
- 22:59we can get a whole lot of a whole lot
- 23:01more capability and problem solving kind
- 23:04of ability from these models if we just
- 23:07bring in this inference scaling. Here we
- 23:10are showing repeated sampling but there
- 23:11are many other ways to do that and we
- 23:14are so the reason it's called inference
- 23:16scaling is that we are not touching any
- 23:18of the parameters of the model. the
- 23:20model is fixed and we are just like at
- 23:22inference time we are creating different
- 23:24ways to generate to kind of produce
- 23:27generations from the model and use them
- 23:30to generate higher quality answers or
- 23:32answers to harder problems for the
- 23:34model. Um the interesting part here is
- 23:37that for some of these problems out of
- 23:39these 10,000 solutions maybe like three
- 23:42or four of them were correct for a
- 23:45single problem. So it shows that how how
- 23:48kind of um
- 23:50how important it is to to scale the
- 23:53inference to really get to the core
- 23:56capability of the models.
- 23:58But what is the difference like um with
- 24:01like traversing all the possible answers
- 24:03and just pick up like the correct like
- 24:07if the number of samples are here like I
- 24:10said like you need like 10,000 times
- 24:13sampling
- 24:16>> sorry what's the question
- 24:17>> what is the difference of things done
- 24:19with like traversing all the possible
- 24:21answers and and and
- 24:24>> so the possible answers need to be
- 24:26generated somehow. by the model, right?
- 24:30So, so this is like the space of all
- 24:33possible answers to the model is way
- 24:35bigger like like right we can't like if
- 24:38you do like a tree search or a random
- 24:40search is like way bigger right there's
- 24:42no way to like hit even like imagine
- 24:45like we are asking a human like just sit
- 24:48down and solve this problem right there
- 24:50is a limit to um of course there's no
- 24:54limit to human creativity humans
- 24:56creativity but there's a limit to how
- 24:58many generations we can have right or if
- 25:01we randomize thing there's a whole lot
- 25:03set of answers that can be generated and
- 25:06it's and this is like highly still
- 25:08highly sample efficient right 10,000 is
- 25:10not a whole lot especially like for
- 25:12example for the math um F2F some of
- 25:16these problem are really hard like IMO
- 25:19level problems like these are really
- 25:21hard problems that a tiny model like a
- 25:23seven and a llama 8p can solve
- 25:28>> I was wondering How
- 25:29>> how does the latency compare
- 25:32like when you do actually generate all
- 25:34those samples like compared to the GP4
- 25:37right like how does the latency
- 25:40is there a trade-off there
- 25:41>> yeah of course there's a trade-off um
- 25:44and we we have we touch on that in the
- 25:46in the lecture that I go more details
- 25:48towards this the parallel samples the
- 25:50good thing about them that they're they
- 25:52can be run in parallel um so from a
- 25:55latency perspective that's less of an
- 25:57issue but of There is the frontier of
- 25:59cost tradeoffs between how much compute
- 26:02you're spending and and we will look
- 26:04into that and it varies it varies
- 26:06depending on the type of and complexity
- 26:08of the problems.
- 26:10>> Yes.
- 26:12>> When you have a verifiable domain like
- 26:15mapping coding, how do you handle it
- 26:16when you don't have?
- 26:19>> Yes, great question again. It's a very
- 26:21hard when you have verifiers it's much
- 26:24easier when you don't have verifiers.
- 26:26There's this whole set of research on
- 26:28how to train LLM as judge or LLM as
- 26:32reward functions or LLMs with tools
- 26:34becoming and we're going to learn about
- 26:36all we have a lecture just dedicated to
- 26:38verifiers. So
- 26:42>> yes a quick question sorry so um does
- 26:45this generalize to any positive
- 26:48temperature? Um, so like as the like the
- 26:52temperature goes to infinity.
- 26:57>> Yeah, we can't like have the temperature
- 26:58being too high because it's just this
- 27:00going to be gibberish like the model
- 27:02doesn't generate. So we there's actually
- 27:05like a research project last year was
- 27:07like on tuning that or like figuring out
- 27:10what temperature usually if you go
- 27:12beyond like 1.2 or so it's it's not
- 27:15great. Yeah. Yeah. But there there other
- 27:19tricks that you can do to kind of kind
- 27:22of uh make the model be more um like
- 27:26have more diversity in the type of
- 27:28answers and that can be useful. Uh all
- 27:32right. Um
- 27:36so
- 27:38let's look at this. So now again going
- 27:42back to the events of last year year and
- 27:45a half um we had the deepseek model.
- 27:48Well, I think it came out in
- 27:49>> December
- 27:50>> December Christmas.
- 27:52>> Deeply came out December 2024.
- 27:55>> Uh oh. Wow. [laughter]
- 27:58>> Short time or long time I don't know
- 28:00both. U so so when models like deepse
- 28:04came last year. So basically a core kind
- 28:07of innovation in deepseek and then the
- 28:1001 series Gemini thinking and such were
- 28:12that now we are kind of bringing this
- 28:15fine-tuning and test time scaling
- 28:16together because with this test time
- 28:19scaling and the models themselves we
- 28:21have this new like kind of engine that
- 28:24we can generate a whole lot of synthetic
- 28:26data um for for example for problems
- 28:31like for math problems where we know the
- 28:33answer we can have the model generate
- 28:35different like kind of answers that
- 28:36leads to that final uh final kind of uh
- 28:39golden answer. Or for the coding
- 28:42problems, we can use the model to
- 28:44generate tons of data during test time
- 28:46scaling uh tons of like quality data to
- 28:49solve coding problems and that becomes
- 28:51part of our training set. Now we can use
- 28:53that to fine-tune the model um to to
- 28:56become better. So that kind of bringing
- 28:59these two together became a big piece in
- 29:02thinking models and reasoning models. Um
- 29:06and this this is like a this is like a
- 29:08an area that is like really really
- 29:11important. There's just so much to be
- 29:13explored in this because again the
- 29:15there's no it's kind of open-ended,
- 29:17right? There's no boundary in how good
- 29:19the models can become with test time
- 29:21scaling and then bringing that back to
- 29:24the process of training the model or
- 29:26fine-tuning the model to become better
- 29:28is very exciting and that's kind of the
- 29:31self-improving piece that we are very
- 29:33excited about.
- 29:36Um so a little bit about reasoning
- 29:38models. Um in the language monkeys what
- 29:41we were showing was the coverage like if
- 29:44we had access to a verifiers how uh we
- 29:47we can see the kind of a log linear kind
- 29:49of scaling logs with the number of
- 29:50samples. Um and open AAI when they
- 29:55released 01 uh last September um they
- 29:58showed a the log linear relationship
- 30:01this time with passad one for these like
- 30:04hard hard Amy uh benchmark which is like
- 30:08a set of really difficult math
- 30:10questions. So as they're show what
- 30:13they're showing here is that as they're
- 30:15increasing test time compute and this is
- 30:17in log scale um uh the the accuracy the
- 30:22pass had one accuracy of the model goes
- 30:24up and this is for test time alone. Uh
- 30:27previously uh it this has been shown for
- 30:30training but it's very interesting that
- 30:32this kind of scaling applies to test
- 30:34time as well um without changing the
- 30:37parameter count of the model. Um
- 30:42um I'm going to walk you through like
- 30:44one a few examples of how this kind of
- 30:47reasoning or test time uh scaling uh
- 30:50kind of is works with models like 03 and
- 30:54or Gemini. So one motivation here is
- 30:58that for difficult problems just like
- 31:00humans think a lot more spend a lot more
- 31:03time or consider many different
- 31:05strategies uh thinking models do the
- 31:07same and they may use a chain of thought
- 31:11or other techniques to just go go about
- 31:14like solving a problem. Um and there are
- 31:18different steps to that. For example, um
- 31:22problem analysis is one step like the
- 31:24model first sees a problem and tries to
- 31:26analyze it. Um then it can do the task
- 31:29decomposition breaking a task into kind
- 31:32of simpler tasks that are more
- 31:35addressable. Um then there is the self
- 31:38evolution strategies where the model
- 31:41kind of tries something sees the
- 31:43feedback for example runs some tests on
- 31:47the code or use a uses a calculator or
- 31:51just judges the answer and it can use
- 31:53that feedback to optimize itself. Um
- 31:55there's selfcorrection and there are
- 31:58alternative proposals. If something
- 32:00doesn't work, the model can backtrack
- 32:02and kind of um try a different approach.
- 32:05These are some key principle principles
- 32:08that the model uh probably at some point
- 32:12uh pieces of it were trained as part of
- 32:14the training kind of data set were
- 32:16curated by humans. But a big part of it
- 32:19is also the model kind of acquiring
- 32:22these skills during this fine-tuning
- 32:25process and RL like RL from synthetic
- 32:28data process that the model does. Um
- 32:31here is an example of like how one does
- 32:35analysis. Um the model is asked to write
- 32:38a bash script um that takes a matrix and
- 32:43then outputs the transpose of the that
- 32:45matrix. So the model starts with ju just
- 32:48like humans like starts like thinking
- 32:52what are the pieces important pieces
- 32:53here. So the user is requesting a bash
- 32:56script. Um let's understand the input
- 32:58and output uh formats of this matrix. Um
- 33:03and and so on. So the model just thinks
- 33:05itself. Um and amazingly this kind of
- 33:09like it's like a progression of chain of
- 33:12thought. This is like helping the model
- 33:14finding the answer. But the difference
- 33:16between chain of thought and this is
- 33:17that the model itself is producing this
- 33:20chain of thought.
- 33:22Um there's of course task decomposition
- 33:25like in this case for the approach uh
- 33:28the model thinks about parsing the input
- 33:31building the matrix as an array of as as
- 33:34an array of arrays and so on. Um there's
- 33:37also selfcorrection which is a big
- 33:39important piece of um this is an
- 33:42important piece here and I don't know
- 33:44how many of you have seen that when
- 33:45trying with the models that you can see
- 33:48the thinking traces uh and this is
- 33:50actually pretty common like the model
- 33:52starts something and then in the middle
- 33:53says wait the correct for like there's
- 33:56something wrong maybe um or and maybe I
- 34:00need to fix that and [laughter] it's
- 34:02very very cool that the model can do
- 34:03that itself um So reasoning models when
- 34:06they came out like models like 01
- 34:09compared to a model like GPT40 which
- 34:11wasn't a reasoning model um they tend to
- 34:15be better in obviously the reasoning
- 34:17task for example in math calculation
- 34:20data analysis programming and such they
- 34:24all kind of outperform GPT40 but not
- 34:28necessarily in personal writing or
- 34:30editing text and so on. Yeah. Question.
- 34:35[clears throat]
- 34:35>> Do you think the gains due to this
- 34:38asking for a reasoning choice is due to
- 34:40actually the act of [clears throat]
- 34:41generating reasoning trace or just the
- 34:44act of asking the model to think or
- 34:47these decomposition at all? Like for
- 34:49example, if you said like think step by
- 34:51step but don't generate out loud like
- 34:54don't actually thought about it without
- 34:58saying.
- 34:59>> Yeah. Um I repeat the question like do
- 35:03you're saying that is it the do you
- 35:05think the reasoning models are becoming
- 35:07what they are is based on the is it
- 35:10based on the loud way of thinking
- 35:13>> I I think the question is that is it
- 35:15because of the um reasoning choices or
- 35:18is it because of things step by step the
- 35:19chain of thought is it [clears throat]
- 35:20like is it the data that's making a
- 35:22difference or is it the the instruction
- 35:25um
- 35:25>> it's I mean so the data basically has
- 35:28made the model kind of become a
- 35:30generalized thinker, right? Like the
- 35:33model can and the all of it all of these
- 35:36steps like breaking down a problem,
- 35:39being able to backtrack, being being
- 35:41able to do analysis, all of these are
- 35:43these skills that the model has learned
- 35:46during this thinking optimization
- 35:48process that now generalize to other
- 35:50things and they're helping the model
- 35:53right now.
- 35:53>> So I think the other way to look at it
- 35:55is Zalia presented repeated sampling. So
- 35:57if you had just the base model right
- 36:00after pre-training it would be able to
- 36:02do some reasonings but at the same time
- 36:04it will generate different kinds of
- 36:06reasoning chains and it doesn't quite
- 36:08know which one is correct. So a lot of
- 36:10the what we will cover in train time or
- 36:13test time scaling really comes down to
- 36:15it learns which is correct. So pass one
- 36:18accuracy goes up which is what you were
- 36:20seeing in the reasoning models as
- 36:21opposed to like passet k or coverage uh
- 36:24which is what you were seeing in the
- 36:25repeated sampling results.
- 36:27>> Yeah. And and one other way to think
- 36:30about like yes these kind of like kind
- 36:32of deep thinking and things that the
- 36:35model generates is actually helping the
- 36:38model every time to solve a new problem.
- 36:41Um and although it's like very expensive
- 36:45for the model to generate those things
- 36:47still like we want them to generate
- 36:50those because it leads to better
- 36:52answers. Um
- 36:55any other questions? Yes.
- 36:57>> We use a different model for the
- 36:59reasoning step versus generating the
- 37:01final answer like maybe a smaller or
- 37:03like a purposeful one just the final
- 37:05answer is a large model. Um
- 37:09so typically the reasoning capabilities
- 37:11have gone up with the model size. So if
- 37:13if anything uh you would use the larger
- 37:17model reasoning traces and then maybe
- 37:19you collect a bunch of reasoning traces
- 37:21and then have a smaller model summarize
- 37:23the answer.
- 37:23>> Yeah. So the models like this is like at
- 37:26least currently things may change. They
- 37:28like their own traces more even if the
- 37:31the traces are coming from a better
- 37:34model. they tend to like their traces
- 37:37their own generated traces more and
- 37:39we're going to have talk about swirl.
- 37:41This is one of the papers that uh
- 37:43>> the multi-step reasoning
- 37:44>> the multi-step reasoning that we talk
- 37:46about what if we bring a different model
- 37:49not in the context of generating reason
- 37:51interests but in in the context of like
- 37:53evaluating and giving feedback and we
- 37:56we're going to see that as well but
- 37:58models surprisingly or not surprisingly
- 38:01they like their own traces a lot more.
- 38:04>> For reasoning models how are they taught
- 38:07to reason? Is it a hardcoded set of
- 38:09sequences they're taught to do traces or
- 38:11they somehow fine-tune to say you need
- 38:14to do this much or this kind of thing
- 38:16which will cause the traces to generate.
- 38:19>> So I don't think there's a published
- 38:20piece of work that really covers this
- 38:22bit. Um
- 38:23>> both yeah
- 38:24>> but it's a bit of both and at the same
- 38:26time the base model did have the
- 38:29thinking capability to begin with right
- 38:31it would go do in some go do some amount
- 38:34of thinking. One thing that we will
- 38:35cover in the class is uh this notion of
- 38:37outcome rewards model and process reward
- 38:39model in how you can use that feedback
- 38:42to get the model back. Yeah, there's
- 38:45always some bootstrapping like okay here
- 38:48are different ways like chain of thought
- 38:50itself which is heavily used in the
- 38:52instruction tuning of data set tell
- 38:55shows the model some ways of like how to
- 38:58think um and there could be like some
- 39:00templates showing uh for the model
- 39:03fine-tuning but the things that a model
- 39:06if you just ask them any ask it any
- 39:08question and it comes up with that
- 39:09there's a whole lot of generalization
- 39:11that so models have gone way above like
- 39:14what the data or instructions that are
- 39:16used for training them.
- 39:20[snorts]
- 39:20>> Yes.
- 39:22>> Is there a way to make the number of
- 39:24samples dependent on problem?
- 39:27>> Please uh speak up the number of
- 39:28samples. Can they be dependent on
- 39:30problem difficulty?
- 39:33>> I don't think there's a published piece
- 39:34of work that does that. uh but uh
- 39:37>> yeah we just so
- 39:40so when so there is follow-up work to
- 39:43that where we you use a reward model and
- 39:46then you can use that so assuming the
- 39:48reward model has some notion of
- 39:50complexity if you haven't solved it it
- 39:51can guide like more sampling or whether
- 39:54it's repeated or or parallel uh but
- 39:57that's definitely an interesting uh
- 39:59direction to explore yeah
- 40:03okay we need to get to the Oh, you
- 40:06haven't, right?
- 40:07>> Yeah, let me record this. Okay, so we
- 40:10you now know about large language models
- 40:13and then you heard about how they
- 40:15learned how to think and then they
- 40:16learned how to reason. So, what's next
- 40:19and why does this why is this course
- 40:20relevant? So, a lot of what this course
- 40:22is about is that large language models
- 40:25as chat bots or as uh reasoning models
- 40:28are basically still single turn like or
- 40:30or just like in the chat format. So,
- 40:32they're not accomplishing a task for
- 40:33you. They're fun to interact with, but
- 40:36they're not necessarily accomplishing a
- 40:37task for you. What has happened this
- 40:39year uh almost in the last few months
- 40:41and it's been surprising is that agents
- 40:44like cloud code or deep research have
- 40:46really enabled people to do real world
- 40:49workflows. Um so there are agentic
- 40:51workflows which can achieve tasks that
- 40:53you ask them to do end to end. So for
- 40:55example, if you want to go research and
- 40:57figure out where should I uh rent a home
- 41:01uh perhaps for uh the entire uh year for
- 41:04uh if I want to take a class at
- 41:06Stanford, the model would actually be
- 41:08able to do a whole bunch of analysis and
- 41:10like go look at a lot of different
- 41:12websites and actually summarize the
- 41:14results and give you pros and cons of
- 41:16different places. This used to not be
- 41:18possible before. So this uh we'll cover
- 41:20a little bit of how that that becomes
- 41:22possible and how this course allows you
- 41:23to learn about that and then cloud code
- 41:26for example if any of you are codeex uh
- 41:29which is the coding agent from openai if
- 41:31you're using that what you'll see is
- 41:33that just by giving instructions in
- 41:35English you can modify files or you can
- 41:37like figure out test cases and and
- 41:39whatnot so it really uh is become a
- 41:42coding productivity tool in in the
- 41:44day-to-day workflows for software
- 41:45engineers at this point in time so what
- 41:48what is the transition from LM to
- 41:50agents. So as I was giving you the
- 41:52example of deep research and of uh
- 41:55coding agents basically the goal
- 41:58basically now the uh model can be given
- 42:00a goal and it will plan out the steps
- 42:03then it will go interact with uh the
- 42:05environment and based on the feedback it
- 42:08will correct its steps until it achieves
- 42:09the goals or it will come back and say I
- 42:11cannot achieve the goal. So this notion
- 42:13of like having some notion of a goal uh
- 42:17taking actions towards that goal getting
- 42:19the feedback and then deciding when to
- 42:22stop that's what makes agents uh
- 42:24different from the chatbot times of what
- 42:27we were doing before and um this also
- 42:30might require interacting with tools uh
- 42:33which are external to what the model
- 42:36itself is doing getting some inputs from
- 42:37there but it still stays on track of
- 42:40like whatever task it's uh choosing to
- 42:42accomplish which might mean that it has
- 42:44to have some form of memory to keep
- 42:46track of the task that it's trying to
- 42:48accomplish.
- 42:50So what has been uh accomplished? So
- 42:52like in in in several cases in simpler
- 42:54cases you can achieve these uh um these
- 42:57end to-end goals like deep research can
- 43:00like be accomplished end to end but in
- 43:01most scenarios you still having very
- 43:03static workflows. So today what exists
- 43:06is closer to agentic workflows where you
- 43:08have um some sort of an input where
- 43:10you're giving a model the goal and then
- 43:12um what this slide is showing you is
- 43:14that one model is perhaps uh giving the
- 43:17output like a solution and then another
- 43:19model is judging it and then deciding
- 43:21based on that whether the solution
- 43:23should be accepted. So that's like one
- 43:25orchestration framework that's possible.
- 43:27uh deep research would be more of the
- 43:29second one where you're basically
- 43:30generating mult uh you're calling the
- 43:33LLM on multiple possible inputs and then
- 43:36aggregating the um things to get the
- 43:39output which will be a summary of the
- 43:41deep research results. So this is a very
- 43:43cartoonish abstraction of what agentic
- 43:46workflows might look like and you can
- 43:47generalize from there. But instead of
- 43:49having like very open-ended loop that I
- 43:52was showing you here where you're
- 43:54supposed to go take an action from the
- 43:55environment and come back and do some
- 43:57feedback. Um it's easier for open-ended
- 44:01problems to like construct this graph by
- 44:02hand of like how a human would do it and
- 44:04then like get this feedback perhaps from
- 44:06an LLM which is the LM evaluator here.
- 44:09So a lot of the real world workflows
- 44:12still have this paradigm but in in
- 44:14certain cases we are starting to see
- 44:16signs of life for what's possible in
- 44:18this slide and coding being one and
- 44:20research being another one.
- 44:23So typically the workflows will have
- 44:25things like LM calls. So you would
- 44:27basically call an LM with an instruction
- 44:29or some form of like input and then
- 44:31you're asking the LM to give you an
- 44:33output. It will have some form of
- 44:35verifiers. We'll cover verifiers in a
- 44:36whole lecture by itself. It will have
- 44:38some form of critics or judges which is
- 44:41effectively LLM as a judge uh paradigm.
- 44:44Uh there might be tool calls for example
- 44:46for deep research you have to actually
- 44:47go search the web to figure out what
- 44:50exactly is the content that you should
- 44:52be looking at. So there might be tool
- 44:54calls of that form or uh getting uh some
- 44:56value around what exactly is the weather
- 45:00um and similarly search would be another
- 45:03tool call and then you would orchestrate
- 45:05these in in some form of a workflow. So
- 45:08a simplest workflow here would be prompt
- 45:10chaining. So similar to reasoning models
- 45:12where the task gets decomposed into
- 45:14subtasks. Prompt chaining will simply be
- 45:16like you have chained a bunch of
- 45:17different subtasks that you're supposed
- 45:19to go accomplish to achieve the end to
- 45:21end goal. There might be routing.
- 45:23Routing works for complex tasks where
- 45:25you say okay if this is very complex
- 45:27then go to this more complicated set of
- 45:30LLM calls and it's less complicated then
- 45:32do this simpler workflow. Um there might
- 45:35be parallelization where you like deep
- 45:37research is one example where you you
- 45:39can have multiple LLM calls work
- 45:41simultaneously on researching different
- 45:43keywords that you give as input to the
- 45:45LLM and then uh finally you aggregate
- 45:48the output or you might break the task
- 45:50into independent subtasks and then uh
- 45:52combine uh the solution or you might
- 45:55have this notion of orchestrator where
- 45:57you basically are actually using some
- 45:59sort of a
- 46:02in simple words LM manager so a central
- 46:04LM is actually doing the planning. So in
- 46:06cloud code you actually start to see
- 46:07that that there's some notion of like a
- 46:09plan that it comes up with and then
- 46:11based on that plan it will like make
- 46:13subsequent LM calls. You might have an
- 46:16evaluator or a judge. So instead of like
- 46:18getting feedback from real world like
- 46:20user or from some sort of a actual
- 46:22running a unit test, it actually might
- 46:24just use LLM as a judge and we will have
- 46:27uh some homeworks that will cover this
- 46:28aspect. And then you might have
- 46:30verifiers. Verifiers are things where
- 46:32you can actually verify the output. So
- 46:34in code for example, if you run the
- 46:37code, how do you know that this code is
- 46:38correct? So typically as software
- 46:40developers, we write unit tests. Uh so
- 46:42similarly verifiers might be running
- 46:44some kind of unit test to check whether
- 46:47what LM generated is correct and in
- 46:49domains which are verifiable uh math
- 46:52code um and other domains that are more
- 46:54rule-based uh this verification is a
- 46:56good way to give feedback back to the
- 46:58model so that um it can correct its
- 47:01steps. So most of these u workflows mean
- 47:05that the uh the LM needs to be better at
- 47:08planning. It needs to be better at
- 47:09multi-step reasoning and it needs to be
- 47:11better at self-improvement like when it
- 47:13makes mistakes it needs to be able to
- 47:14correct itself. So effectively these are
- 47:17sort of new paradigms that the current
- 47:19set of LLMs just with reasoning were not
- 47:22quite accomplishing and these are some
- 47:24of the topics we do plan to cover in
- 47:26subsequent lectures
- 47:29and just to uh drive the point home
- 47:31around uh coding agents. So this is a
- 47:34very simple example. This is actually um
- 47:36I should have updated the slide to for
- 47:38cloud code but basically what this is
- 47:40showing is that you have an LM agent and
- 47:43uh it's interacting with the computer
- 47:45these days just with a terminal. You
- 47:47give it an instruction where you might
- 47:48tell it that you want to implement a
- 47:50test. Um and then it has some navigation
- 47:53of repositories, searching of files, all
- 47:55of these tool calls around like viewing
- 47:57files, editing lines and then it will go
- 47:59execute a bunch of commands in the
- 48:01terminal and based on the output it
- 48:03might be like okay I need to go edit
- 48:05this other file or I need to go look at
- 48:07this other file. So if you look at this
- 48:09loop this very much mirrors u what was
- 48:12so this was not quite reliable last year
- 48:15and it's just starting to get reliable
- 48:16in what we are seeing in the uh realm of
- 48:18coding agents now.
- 48:20Yes.
- 48:21>> Can I ask why you think it's more
- 48:24reliable with I guess the general
- 48:27I mean I would say not change that much.
- 48:31>> I mean I think the paradigm is very much
- 48:33the same. It's it's mostly a matter of
- 48:35more powerful models and then better RL
- 48:41verifiable rewards is working. The train
- 48:42time scaling is working well.
- 48:47I guess the most
- 48:49improve I guess working on performance
- 48:51without necessarily just throwing more
- 48:55comput power
- 48:57>> I mean computer data right one of those
- 49:00two but like just also once the models
- 49:03start to get better there's the
- 49:04self-improvement loop that kicks in uh
- 49:07because you can now generate tests and
- 49:09the tests become more reliable so in the
- 49:11published pieces of work if you look at
- 49:12say code monkeys which we covered last
- 49:14year um if you can generate unit tests
- 49:17um for whatever code the model generated
- 49:20and then see if the generated unit tests
- 49:23are
- 49:25making things better then that that
- 49:26that's a very good way to verify things
- 49:28right
- 49:29>> yeah like model based it seems like to
- 49:32some extent if we have good verifiers
- 49:35um we can make the model generate the
- 49:38the right code
- 49:41uh but the question is how do we do that
- 49:43and there are ways to do it and we we we
- 49:46would be happy to for you guys to take
- 49:49research projects in that domain too.
- 49:51Obviously, it's an active um research
- 49:54area, but there are ways to make um
- 49:56improvements that you can
- 49:58>> I think what you're alluding to is this
- 49:59notion of generator verifier gap. So,
- 50:01it's easy for models to generate a whole
- 50:03bunch of nonsense or sensible set of
- 50:06like reasoning traces or like useful set
- 50:08of content. But at the end of the day,
- 50:10whether that's useful or not, we need
- 50:12feedback loop for that. And if you're
- 50:14creative writing, how much feedback can
- 50:15you get? So, human feedback ends up
- 50:17becoming a bottleneck. In domains where
- 50:19you can have good feedback, that's where
- 50:21it's possible to continue to improve the
- 50:23model and that like but ver getting
- 50:25robust verification a lot of robust
- 50:27verification is hard. So Aalia will
- 50:30cover a paper that she did in her lab
- 50:32about like how to combine verifiers and
- 50:34but verification continues to be one of
- 50:36the bottlenecks in this space to make
- 50:38them better.
- 50:40>> Yes. Um just like intuitively I
- 50:43pre-training you get ability
- 50:47such a large
- 50:49>> so just like to me intuitively I'm
- 50:50trying to figure out why there's such a
- 50:52big job
- 50:53>> because like we're sort of providing a
- 50:55signal and we're trying that I guess
- 50:59shouldn't exist.
- 51:03So yeah, I I guess like I'm just not
- 51:07re like
- 51:08>> I I think this is still an active area
- 51:10of research. So there's different set of
- 51:12opinions around what really improves the
- 51:14model. Is it RL or is it like the the
- 51:17pre-training the diverse data by itself
- 51:20and I I don't think there's single point
- 51:22of consensus at this point in time. Both
- 51:24processes help. um what you just said uh
- 51:27which I'll repeat for the class is that
- 51:29if pre-training is the place where uh
- 51:32the model after repeated sampling should
- 51:33be able to at least have one solution
- 51:35correct out of if it was generating a
- 51:37large number of samples then uh this
- 51:39notion of giving feedback should improve
- 51:41the pass at one accuracy but should not
- 51:43improve the model why is it like such a
- 51:45big jump but I think um that whole loop
- 51:48is not completely well understood it's
- 51:50like the first signs of life and it
- 51:51starts to get commercialized but I think
- 51:53there's a lot more research still open
- 51:55in this area.
- 51:56>> Yeah.
- 51:57>> So that's only one uh way of thinking
- 51:59about it. There are other um at least
- 52:01some signs of life which say that you
- 52:03can continue to do RL and then that will
- 52:06continue to improve the model.
- 52:07>> Yeah.
- 52:10>> Okay.
- 52:10>> Yeah.
- 52:12So uh so this is uh basically covering a
- 52:15lot of uh the same set of points but uh
- 52:17one abstraction that is worth taking
- 52:19away from this particular chart is that
- 52:22even when you give the model a goal it
- 52:24has to clarify the user intent. So it's
- 52:26like not always obvious to it what the
- 52:28user wants and then it might go search
- 52:30for relevant files and then whatever um
- 52:33set of actions it chooses to take often
- 52:36times it needs some form of
- 52:37verification. In this particular case,
- 52:39it's based on um passing the test and it
- 52:42might actually generate the test that it
- 52:44needs to pass which um which is also uh
- 52:48mentioned here. But the the key idea is
- 52:50that if you give it a task, how does it
- 52:53come back and how do you know that it's
- 52:54going to complete what you asked it for.
- 52:57Often times the users will not specify
- 52:59the problem well enough. So asking
- 53:01clarifying the user intent so that it
- 53:03knows what to go look for and how to
- 53:05verify starts to become uh important.
- 53:07And in models like 03 I think they have
- 53:09seen a lot of these traces end to end.
- 53:11So they are able to like do the planning
- 53:13the reasoning the multi-step reasoning
- 53:14and like come back with the conclusion.
- 53:16But uh a lot of this um is needed for um
- 53:20completing tasks end to end or having an
- 53:22end to end goal. And where this has been
- 53:24super useful is uh if you have
- 53:26repetitive uh repetitive tasks like code
- 53:29migrations or version upgrades or if you
- 53:31need to restructure the codebase or if
- 53:33you have tasks that involve data
- 53:35engineering where you have to basically
- 53:37do some sort of extract uh the data and
- 53:39then do some sort of clean cleanup on
- 53:41the data or if you have to do some sort
- 53:43of data warehouse migration. So a lot of
- 53:45this work tends to be extremely um
- 53:48repetitive and it's much easier to just
- 53:51delegate it to coding agents. uh same
- 53:53for unit tests often times like much
- 53:55easier to like start to give it to uh
- 53:57coding agents. Another area where agents
- 54:00have become uh extremely prevalent is in
- 54:03customer support. It's one of the most
- 54:05thankless jobs that u when when you are
- 54:07on the customer support side of things
- 54:09you have to do and using LLMs to do that
- 54:11definitely makes um streamlines the
- 54:13experience in interesting ways. So one
- 54:15simple set of examples ends up being
- 54:17like you can use them to do live
- 54:18transcription. So that gives you a very
- 54:20nice record and and you're seeing that
- 54:22even for meetings these days. Another
- 54:24simple is example is that you can have
- 54:27knowledge assist. So if you have a
- 54:28database of information, the the
- 54:30customer support agent does not need to
- 54:32know everything. They can consult with
- 54:34the LLM and and get an answer and
- 54:36surface the relevant article. Uh which
- 54:39is better than having um just search um
- 54:42like an index and a search by itself.
- 54:45Uh smart reply. So in in chat often
- 54:48times this has existed for a while but
- 54:50you can use an agent to like give uh
- 54:52chat responses and then having a call
- 54:55summary can help you um you can use uh
- 54:58the call summary to really uh improve
- 55:01the customer experience by itself as
- 55:02well. So overall I think in the customer
- 55:04support area there's multiple uh
- 55:07companies that are going after this area
- 55:09but uh there are different segments of
- 55:11the problem that can be addressed using
- 55:12LLMs and they have been super useful and
- 55:15then there's also the end to end stuff
- 55:16that's starting to happen here
- 55:19and then the third example that we will
- 55:21use and actually some of the homeworks
- 55:23is that if you have very complex topics
- 55:25and uh you want to provide a
- 55:26comprehensive report um earlier it used
- 55:29to be that you had to do a lit review
- 55:30and then you had to summarize each paper
- 55:32and then you have to synthesize these
- 55:33things and these days you can just give
- 55:35it to an LLM and actually do it for you
- 55:37and um study maybe more articles than
- 55:40you would. So typically if you give it
- 55:42an example like u say 2022 winter
- 55:45Olympics opening ceremony it will
- 55:48identify what references to go look at
- 55:50then it will construct an outline of
- 55:51like okay these references are relevant
- 55:53or not and then summarize each of the
- 55:56references uh as to what is the relevant
- 55:59content and then combine them to create
- 56:01a fulllength article um which is u very
- 56:04impressive in certain cases and and
- 56:06you'll get to try it in one of the
- 56:07homeworks
- 56:09and then even more forwardlooking um
- 56:13these research agents are starting to be
- 56:16used for uh as AI scientists. So
- 56:18basically they're starting to assist
- 56:20scientists in say uh solving math
- 56:22problems or solving science problems. So
- 56:24here the LLM is used as a brainstorming
- 56:27thing to come up with ideas. So the idea
- 56:29generation phase then the experiment
- 56:31iteration phase it might actually help
- 56:33you iterate on the experiments that you
- 56:35are hoping to work on. This is from the
- 56:37AI scientist paper. And then in the
- 56:40paper write up phase it will help you
- 56:42improve uh the the paper write up by
- 56:44itself.
- 56:47And what's interesting uh is that
- 56:51even even though these models
- 56:52hallucinate just the just the notion
- 56:55that they can come up with so many
- 56:56different set of ideas as an AI
- 56:59scientist it might actually brainstorm
- 57:00or like give you ideas outside what you
- 57:03would have thought if you had just like
- 57:05taken a bunch of courses or or if you're
- 57:06a researcher and you've been in the
- 57:08field for a long time. Sometimes the
- 57:10reading the web uh allows these LLMs to
- 57:13come up with ideas that are way outside
- 57:14the box and can be a very good
- 57:16brainstorming um way. Uh so this these
- 57:19have been super useful for in the AI
- 57:21scientist style of work. Okay. So let's
- 57:25move to course logistics. But before
- 57:27that do we have any questions?
- 57:29>> Yes.
- 57:31>> I just want to clarify. So um is
- 57:33reasoning and chain of thought baked in
- 57:35training or is it just prompt
- 57:37engineering that was accidentally
- 57:39discovering these kind of capabilities
- 57:40in larger models and if so are there
- 57:43other emerging behaviors besides reason
- 57:47that you think might be discovered?
- 57:50>> So uh I mean it was not baked in by uh
- 57:54by design. uh it was it was basically
- 57:57discovered as in like it was I mean we
- 57:59gave it hard problems and then we saw
- 58:01that it was basically by having
- 58:03reasoning chains it was doing better. So
- 58:05the GSM8K was the first paper that kind
- 58:07of showed signs of life of this and then
- 58:10uh with larger model uh like palm we
- 58:12actually saw that this was a very big
- 58:14deal and one of the examples there was
- 58:16that could explain jokes which was very
- 58:18impressive and then from there the
- 58:20reasoning models have emerged so it was
- 58:22not but it has read all of the web so
- 58:25it's it has definitely seen data which
- 58:27is like more um methodical and
- 58:29systematic
- 58:30>> but the reasoning models are trained to
- 58:33be reasoning more and more, right? So,
- 58:36it's they didn't the entire reasoning is
- 58:39not an emergent behavior. The reasoning
- 58:41models, they're trained to be to be
- 58:44thinking, but like the models are
- 58:46convergent. So, the models are going to
- 58:48be trained such that they know when to
- 58:50they need a lot of reasoning and when
- 58:51they don't and and to generate answers.
- 58:55But chain of thought originally was an
- 58:58emerging behavior. they noticed that oh
- 59:00if we explain things a model it's better
- 59:05>> any kind of other emerging behaviors and
- 59:07larger models
- 59:08>> I don't think it's I would say it as
- 59:10immersion behaviors per se but I think
- 59:12as I was mentioning we usually go
- 59:14looking for certain things right so in
- 59:16the agentic workflows what we're looking
- 59:17for is planning which is a form of
- 59:20reasoning we're looking for multi-step
- 59:21reasoning which we will cover in class
- 59:23as well and then we're looking for
- 59:25self-improvement or selfcorrection so
- 59:27all of these capabilities would be nice
- 59:29to have and like what gets the models
- 59:31there is a set of questions that are
- 59:33worth like there are papers on this kind
- 59:34of things
- 59:36>> like selfcorrection
- 59:37backtracking things like that is like
- 59:40yes you could call it emergent but you
- 59:43could also say think about it that they
- 59:46have seen this kind of behavior and it's
- 59:48kind of like it's reinforced in the way
- 59:50that they're fine-tuned so it's
- 59:54yeah it's hard to
- 59:58Okay. Um, so let's go through the the
- 1:00:01the logistics for the class. Um, here's
- 1:00:05a list of all the amazing topics that
- 1:00:07you're going to learn about. Um, they
- 1:00:09are um,
- 1:00:12yeah, this is very cool.
- 1:00:15>> I think the main thing to remember here
- 1:00:16is that the overall theme stays the
- 1:00:18same. uh and then we'll also have guest
- 1:00:20lectures around from u folks in frontier
- 1:00:23AI labs and covering things all the way
- 1:00:26from say how have uh has how has post
- 1:00:29training evolved or say even multimodal
- 1:00:31agents and uh in robotics. So it's it's
- 1:00:34a it's going to be a mix of lectures and
- 1:00:37guest lectures and um and of course your
- 1:00:40project presentations.
- 1:00:42[snorts]
- 1:00:43So here is some logistics that you all
- 1:00:46know the the prerec are there just make
- 1:00:48sure that you you are comfortable with
- 1:00:51these prerexs as you uh register them in
- 1:00:54the class. So we have the external
- 1:00:57website but uh make sure that you're uh
- 1:01:02you check canvas that's where we send
- 1:01:04out the latest updates at all times. Um
- 1:01:07and um we will also upload the we try to
- 1:01:11upload the lectures before we start
- 1:01:13every every in every class so you have
- 1:01:16access to the lectures. Uh and we will
- 1:01:18also have we already have the due dates
- 1:01:20for all the assignments and all the
- 1:01:21projects. So you can check it out. Um
- 1:01:25so this quarter we are going to have
- 1:01:27three homeworks. That's a difference
- 1:01:29from between this class and that's one
- 1:01:31of the differences. So we have one more
- 1:01:34amazing homework for you all that we
- 1:01:36have uh designed and the TAs have done a
- 1:01:38great job and that's helps you um
- 1:01:41hopefully learn uh learn these topics
- 1:01:44better and more in depth. Um and then we
- 1:01:48also have a course project. So the
- 1:01:50course project is where you can kind of
- 1:01:51unleash your creativity and like your
- 1:01:55your way of like building agentic
- 1:01:57systems or going deeper into a question
- 1:02:01uh and and designing experiments around
- 1:02:04it and see what works and what doesn't.
- 1:02:06Um we will provide some um kind of
- 1:02:10examples or suggestions but it could be
- 1:02:13completely on you how to design this
- 1:02:15project. We'll have some examples from
- 1:02:17last year like the successful projects
- 1:02:19from last year as well. Uh it's going to
- 1:02:22be uploaded to
- 1:02:24um canvas or
- 1:02:26>> actually we'll display them on the
- 1:02:27website.
- 1:02:28>> Oh, on the website. Oh, cool. The public
- 1:02:30website. So you can also see that. Um so
- 1:02:34>> so for the course project I think the
- 1:02:36main thing to remember is that uh we
- 1:02:38will have API credits and you can work
- 1:02:41in teams of two to four people,
- 1:02:43>> right?
- 1:02:44>> Yeah. or up to four. I guess
- 1:02:48>> it can be one if you really want that.
- 1:02:50We we we suggest you team up with
- 1:02:53others. Um at the very least you have
- 1:02:56more credit collectively so you can run
- 1:02:58more experiments but then you might also
- 1:03:01uh find a friend collaborator along the
- 1:03:03way and you can um um do something
- 1:03:06bigger. Um and we are going to have this
- 1:03:11um what is this? The course project.
- 1:03:14Yeah, we we're going to we're going to
- 1:03:15let you know about some uh some of the
- 1:03:18previous years and some ideas that you
- 1:03:20can take. Um here are some course
- 1:03:22project examples that are acceptable.
- 1:03:25For example, a new evaluation data set
- 1:03:28or a new benchmark. Um you can design a
- 1:03:32project around the reliability of a of
- 1:03:34an agent agentic system that already
- 1:03:36exists. You can take a benchmark that
- 1:03:39exists and try to hill climb on it with
- 1:03:41whatever idea great idea that you have.
- 1:03:45Um and then or you can just we we are
- 1:03:47going to have a whole lot of papers
- 1:03:49covered in this course and we have we
- 1:03:51are showing all of that to you. They're
- 1:03:53all on the website. So as you as we give
- 1:03:56this lectures, we kind of expect you to
- 1:03:58also have read those papers or read the
- 1:04:00papers along the way and develop better
- 1:04:03and deeper intuitions on how these
- 1:04:05methods work. and your project could
- 1:04:08always be improving those or like
- 1:04:10questioning one of the decisions that
- 1:04:12they have made and trying to uh change
- 1:04:14that or or evaluate that. Um negative
- 1:04:19examples are like a survey paper or like
- 1:04:22we just don't we want something
- 1:04:24researchy here. So we just don't want
- 1:04:26like just an app that you put together
- 1:04:28and you just show us something. It has
- 1:04:30to be like here is like here is the
- 1:04:33hypothesis here is the question that we
- 1:04:36wanted to answer here is the type of
- 1:04:37like improvement we wanted to see or the
- 1:04:40type of like uh properties we wanted to
- 1:04:42analyze and then we have built that
- 1:04:44right we want something something that
- 1:04:46um uh something more than wipe coding
- 1:04:49basically in this um
- 1:04:52>> in terms of milestones I think the it's
- 1:04:54worthwhile to remember that you want to
- 1:04:56start early so it's you need to have a
- 1:04:59project proposal um somewhere around I
- 1:05:02think we we put it early October.
- 1:05:04>> Yeah.
- 1:05:04>> And then uh
- 1:05:05>> it's on the website. Yeah,
- 1:05:06>> it's on the website. And then for the
- 1:05:08midterm project presentation, we do want
- 1:05:10you to have some progress. So two weeks
- 1:05:12after the project proposal, we do expect
- 1:05:14you to have like so start thinking about
- 1:05:16like what experiments you want to run by
- 1:05:18the time you get to the project proposal
- 1:05:20so that you actually go run them for the
- 1:05:22midterm projects. And so we do expect
- 1:05:25you to have made some progress in the
- 1:05:26midterm project presentation. It should
- 1:05:28not just be here is a proposal and then
- 1:05:31the final report has a lot of weight and
- 1:05:33then the final poster presentation uh
- 1:05:35will be the end of the quarter.
- 1:05:40Um yes.
- 1:06:00>> I I think so if you add to it, right?
- 1:06:06>> Okay. Okay. when I was independent
- 1:06:08study, right?
- 1:06:09>> Yeah.
- 1:06:10>> Yeah. Yeah, that's fine.
- 1:06:11>> I I think
- 1:06:12>> I mean as long as you're doing actual
- 1:06:14work and it's not just like reusing the
- 1:06:16work from
- 1:06:19>> Yeah.
- 1:06:20>> Um we had uh we had students publishing
- 1:06:24papers out of their projects uh in the
- 1:06:27last quarter. So uh in the last last
- 1:06:30time we taught the class. So that's
- 1:06:31something you could it could be one of
- 1:06:33you or many of you this time again uh
- 1:06:36turning your research project working
- 1:06:38more on it and then uh turning it into a
- 1:06:41publication at at um uh conferences.
- 1:06:46Um and so here is a save the date for
- 1:06:51for our poster presentation. It's going
- 1:06:53to be December 12, 4 to 6 p.m. So um we
- 1:06:57we would want you to be there and
- 1:06:59present the posters. you'll have people
- 1:07:01from industry joining and you can that's
- 1:07:04where you can brag about what you've
- 1:07:06done and just meet [clears throat] new
- 1:07:07people. Um here is the course uh grading
- 1:07:11rubric. So uh we have three homeworks
- 1:07:14that's 50% of your uh of of your entire
- 1:07:18grade and then another 50% is on the
- 1:07:20project and the duration of these are
- 1:07:22kind of like align with how much uh how
- 1:07:25much the grading um grading is allocated
- 1:07:28to each of these. Um and we of course
- 1:07:32expect you to uh honor
- 1:07:35the honor code. Um and let's see
- 1:07:40office hours will be posted on Canvas.
- 1:07:43Um you can ask your questions. We really
- 1:07:45really encourage you to ask questions on
- 1:07:47Edson uh and make it public. So others
- 1:07:50probably if you have a question um it's
- 1:07:53very likely others have the same
- 1:07:55question. So please help us like do
- 1:07:57broadcast this to everybody. Um um at
- 1:08:00SSEM is um again you can ask questions
- 1:08:04and then grade scope is where you submit
- 1:08:06your project milestones and everything
- 1:08:09else. Yeah, the homeworks and
- 1:08:11everything. Um we have the following
- 1:08:13late policy that we think is kind of
- 1:08:15like
- 1:08:16within the generous kind of like um side
- 1:08:20on the gener generous side of things and
- 1:08:23because the class is big this quarter we
- 1:08:27really can't make any exceptions. So try
- 1:08:29to use these um kind of late days uh
- 1:08:32wisely. Uh audits are not allowed but we
- 1:08:36will have the the course
- 1:08:39the we will have the videos
- 1:08:42>> videos on YouTube eventually.
- 1:08:44>> Eventually
- 1:08:46and that's it. Any questions?
- 1:08:52>> No. All right.
- 1:08:54>> Okay. Thanks everyone.
About this transcript
This page contains the full transcript of CLAUDE AI ile sıfırdan Ajan Kurma by Kickooo, generated from the public captions YouTube serves with the video. The transcript has 11,786 words across 1,676 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.