YouTube2Text

CLAUDE AI ile sıfırdan Ajan Kurma — Transcript

by Kickooo · 11,786 words · 1,676 segments · language en · Watch on YouTube

Full transcript

  1. 0:00Um so we'll introduce ourselves. So I'm
  2. 0:03Akansa. I have been working in large
  3. 0:05language models for a while and I'm
  4. 0:06adjunk professor at Stanford and I'm
  5. 0:08also in research uh at a startup called
  6. 0:11reflection AI.
  7. 0:12>> Hi I'm Aalia Mir Hoseni. I'm a I'm an
  8. 0:16assistant professor in the CS department
  9. 0:19and um uh Akansa and I met each other
  10. 0:23when we were at Google Google Brain.
  11. 0:26Google Brain. Yeah. Back in the day. and
  12. 0:29then Google deep mind and I also worked
  13. 0:32on cloud at entropic and gemini at
  14. 0:34Google uh deep mind and we are so
  15. 0:37excited to uh teach this course again
  16. 0:40this is the second time we are teaching
  17. 0:41of course we have made some updates both
  18. 0:44to the lectures and also the way we are
  19. 0:47uh kind of conducting everything during
  20. 0:49the the for the course uh and we are we
  21. 0:53were just so amazed by the amount of
  22. 0:54interest uh for people to join this
  23. 0:57course So uh we are excited to have you
  24. 1:00here.
  25. 1:01>> Uh the course website is at
  26. 1:02cs329a.stanford.edu.
  27. 1:05We have updated the website in terms of
  28. 1:07all the lecture materials and and the
  29. 1:10papers that you'll be reading as well as
  30. 1:11uh what's the homework schedule to
  31. 1:13expect um the project proposal and
  32. 1:15project structures as well.
  33. 1:18So in today's lecture we'll start with
  34. 1:20an overview of large language models uh
  35. 1:23just uh how the scaling trends have
  36. 1:24evolved over the last uh five years and
  37. 1:27then we'll focus more on what we will
  38. 1:29cover in this class and then from there
  39. 1:32uh in the second half of the uh this
  40. 1:34lecture we'll cover the course logistics
  41. 1:36and that part you should pay a lot of
  42. 1:38attention to because that influences
  43. 1:39your grades.
  44. 1:43So let's take a look at an overview of
  45. 1:45the trends for large language models. So
  46. 1:48one of the aspects that has been
  47. 1:49mind-blowing since uh GP3 came out is
  48. 1:52that
  49. 1:54uh there's a uh that as you scale up the
  50. 1:57number of parameters in a model uh they
  51. 1:59get better and what this means is uh
  52. 2:01that language models like BERT and T5
  53. 2:05were good but as we increase the number
  54. 2:06of parameters they got really better.
  55. 2:08they they perform much better and what
  56. 2:10this led to is something called a
  57. 2:12scaling loss for large language models
  58. 2:14uh which are used in building the base
  59. 2:16model for the pre-trained base model and
  60. 2:20uh as you increase the amount of compute
  61. 2:21so there are three graphs here as you
  62. 2:23increase the amount of compute on the
  63. 2:24x-axis the loss function on the y-axis
  64. 2:28goes down and and the more compute you
  65. 2:31have put in the more uh the test loss
  66. 2:34goes down which leads to a better model.
  67. 2:36So that was one big aspect that led to
  68. 2:39GP3 and then all the subsequent chat GPT
  69. 2:42palm um Gemini and so on. And then
  70. 2:45similarly as you increase the data set
  71. 2:47size on X-axis uh the test loss goes
  72. 2:50down further on uh Y-axis. So that's
  73. 2:53another aspect of like another axis of
  74. 2:55scaling that leads to better language
  75. 2:57models. And uh the fourth one is of the
  76. 3:00third one is of course parameter count.
  77. 3:02So if you increase the number of layers
  78. 3:04in a transformer or the amount of
  79. 3:05parameters that are there in the
  80. 3:07language model that improves the loss uh
  81. 3:10value and that that leads to a better
  82. 3:11model and and this has been the
  83. 3:13foundation for a lot of the uh this has
  84. 3:15been the foundation for a long time uh
  85. 3:17until last year where this was starting
  86. 3:19to hit some kind of a saturation point.
  87. 3:24So from 2018 to almost 2024 um this
  88. 3:28graph is a little bit outdated but uh
  89. 3:31from 2018 to 2024 the at the model size
  90. 3:34has gone up um almost consistently we
  91. 3:37have been increasing the the size of the
  92. 3:38model so you started with something like
  93. 3:40B was 340 million parameters GBD2 was
  94. 3:431.5 billion parameters and then GB3 was
  95. 3:46175 billion parameters and then u say
  96. 3:49palm was 540 billion and then uh there
  97. 3:52was a bunch of subsequent models and
  98. 3:54GBD4 is estimated to be uh there which
  99. 3:57is like trillions of parameters. So you
  100. 3:59can imagine that there has been an
  101. 4:00exponential growth in just the model
  102. 4:02size for large language models. The
  103. 4:04large stands for that they're growing in
  104. 4:06size and that has been one of the key
  105. 4:09areas in which u the models have been
  106. 4:11getting better.
  107. 4:13So why do we care about the models uh
  108. 4:15getting bigger? What does this lead to?
  109. 4:17So as I was explaining that as the
  110. 4:20models get bigger uh they continuously
  111. 4:22improve in performance on uh natural
  112. 4:24language benchmarks on various other
  113. 4:26benchmarks reasoning benchmarks and so
  114. 4:28on and then other interesting aspect
  115. 4:30that I uh will show you in a moment is
  116. 4:33that they learn how to do few short
  117. 4:34learning. So earlier you have to
  118. 4:36fine-tune the model for specific domain
  119. 4:38but just by giving a few examples the
  120. 4:41model can follow that template and and
  121. 4:43reason in that same vein which makes it
  122. 4:46extremely easy to prototype uh things.
  123. 4:49And then the third aspect is that as the
  124. 4:52models become bigger they have emerging
  125. 4:54behavior where the capabilities like
  126. 4:55reasoning only emerged in larger models.
  127. 4:59So just to understand few short learning
  128. 5:01or zero short learning here. So if if
  129. 5:04the model is given a question like or a
  130. 5:07task like translate English to French uh
  131. 5:10it's able to give uh so this is called a
  132. 5:12prompt like it's given a task translate
  133. 5:14English to French and then it's asked to
  134. 5:15translate cheese to whatever is the
  135. 5:17French word if the model can predict the
  136. 5:19answer without uh actually being trained
  137. 5:22on the specific task then that would be
  138. 5:24called zeros short learning and then few
  139. 5:26short learning in addition to giving the
  140. 5:28in uh description of the task you're
  141. 5:30also giving a few examples in this
  142. 5:32particular case it's given translate
  143. 5:34English to French and it's given a few
  144. 5:35examples of the translation from English
  145. 5:38to French and then it's asked to
  146. 5:40translate cheese and it's able to
  147. 5:41translate that and you can see that this
  148. 5:43capability shows up in the large
  149. 5:46language models across a large number of
  150. 5:48tasks and that has been one of the key
  151. 5:49areas that GPD3 and pal and um other
  152. 5:53models enabled and then that uh
  153. 5:55subsequent innovations uh came from
  154. 5:57reasoning aspects of this. So on the
  155. 5:59reasoning side um as the models have
  156. 6:02become bigger they have this emerging
  157. 6:04behavior of reasoning which Aalia will
  158. 6:06tell us a little bit more about.
  159. 6:08So as the models become bigger not only
  160. 6:11we have this kind of a predictive
  161. 6:13scaling loss property that we kind of
  162. 6:16know how the loss functions going to go
  163. 6:18down as we throw in more compute data
  164. 6:20and parameters in the model. There are
  165. 6:23new kind of behaviors that appear in the
  166. 6:27model that they didn't have before and
  167. 6:28we never could predict that until we had
  168. 6:31these bigger models and saw these kind
  169. 6:32of behavior in them. And one of the most
  170. 6:36important
  171. 6:37uh one of those behaviors is the
  172. 6:40emergent is the chain of thought
  173. 6:42behavior. And for those of you who are
  174. 6:45not familiar with this, here is how it
  175. 6:47works. So in a usual prompt prompting of
  176. 6:51a model, we can ask a model in this case
  177. 6:53a a math problem um and then we can give
  178. 6:57it a oneshot example. Here is a math
  179. 6:59problem. Here is the answer. Now I ask
  180. 7:02you a similar but not exact not exactly
  181. 7:05the same math problem and the model is
  182. 7:08expected to answer that. Um and the
  183. 7:11model might be able to use the the
  184. 7:13oneshot example or it might not be. But
  185. 7:15we can do better here by not only give
  186. 7:19an example in but also provide the
  187. 7:22reasoning or the chain of thought on how
  188. 7:25we got to that answer. So in this case
  189. 7:27the question is a very simple question.
  190. 7:30Roger has five tennis balls. He buys two
  191. 7:33more cans of tennis balls. Each can each
  192. 7:36can has three tennis balls. How many how
  193. 7:39many total tennis balls does he have
  194. 7:41now? And instead of just saying 11, like
  195. 7:43we're walking the model through how to
  196. 7:45get to this. Um, Roger started with five
  197. 7:48balls. Two cans of three tennis balls is
  198. 7:50six. And uh, 5 + 6 is 11. Uh, and with
  199. 7:55this example, then the model has become
  200. 7:58familiar with the process to get to the
  201. 8:01answer and it can leverage that to
  202. 8:04answer new problems. Well, obviously
  203. 8:07this is a very simple problem. any like
  204. 8:091B parameter model nowadays can solve
  205. 8:11this doesn't need a an example or a few
  206. 8:15shot or a chain of thought but the chain
  207. 8:17of thought property itself is a property
  208. 8:19that is holding to this day and it's a
  209. 8:22very very important property for the
  210. 8:24reasoning models and all the thinking
  211. 8:26models and all the prog and many much of
  212. 8:29the progress that we have seen um in the
  213. 8:32past year or so
  214. 8:34now um this property again appears As we
  215. 8:38have larger and larger models, here are
  216. 8:40three models, lambda, GPT, and palm. And
  217. 8:44as you can see, the chain of thought,
  218. 8:45and this is some results on the math,
  219. 8:48some math data set. And these models
  220. 8:52when they're smaller like 8 billion
  221. 8:55parameter for lambda or uh G for GPT
  222. 8:58around 7 billion, they can't really
  223. 9:00benefit from chain of thought like it
  224. 9:03doesn't do anything for them. But as
  225. 9:05they become larger uh or for a palm
  226. 9:08model again at 8 billion it can leverage
  227. 9:11this chain of thought property and learn
  228. 9:14from the reasoning that is and the than
  229. 9:16the process that is provided in the
  230. 9:18context in order to solve problems
  231. 9:20better. Um
  232. 9:23and not only that there are other kind
  233. 9:25of abilities to solve new task as we
  234. 9:28increase the size of the models like for
  235. 9:30examples models can all of a sudden
  236. 9:33solve mod arithmetics better or solve um
  237. 9:37a word unscramble problem just by having
  238. 9:40this larger size and this property. You
  239. 9:43see that all of a sudden it appears at a
  240. 9:45certain size. Um and because of all of
  241. 9:48this, we still as in frontier companies
  242. 9:52and labs are still interested in pushing
  243. 9:54the the scaling laws and not only get
  244. 9:57the natural kind of progression but also
  245. 10:00we we will see more and more of these
  246. 10:03kind of emergent behavior that we are uh
  247. 10:05that is going to be very important.
  248. 10:09Now um a little bit of history here. So
  249. 10:12chat GPT was launched in uh November
  250. 10:162022. Obviously it was one of the most
  251. 10:19successful
  252. 10:21uh kind of [snorts] apps or software
  253. 10:23ever created. Uh the time to reach 1
  254. 10:26million users for it was um 5 days which
  255. 10:31is significantly
  256. 10:32um faster than any that many of the
  257. 10:35other famous um software that you can
  258. 10:38see there or services that you can see
  259. 10:39there. And there were pieces of of kind
  260. 10:42of innovation on top of just scaling up
  261. 10:46the parameter size of the model that
  262. 10:48were very important to make Chad GPT
  263. 10:51what it was and to make that kind of
  264. 10:53leaprog and over the previous models
  265. 10:57like GPT3 and so on. And two p two
  266. 11:00important piece here were um the
  267. 11:03instruction tuning and um the
  268. 11:06reinforcement learning from human
  269. 11:08feedback we which which we are going to
  270. 11:10briefly mention in the next slides but
  271. 11:12throughout this course we are going to
  272. 11:14learn more and more about them. Um so
  273. 11:18let's
  274. 11:20first and then um we're going to learn
  275. 11:22about all of these in more details but
  276. 11:25let's just take a step back and see how
  277. 11:28the process works. So the pre-training
  278. 11:30step is the step with the the beginning
  279. 11:34of any language model training. Uh
  280. 11:37that's the easiest step and we just
  281. 11:39train the model to predict the next
  282. 11:41token out of all sort of text and data
  283. 11:44and so on. And then this is the
  284. 11:46finetuning step that was a key
  285. 11:48differentiating factor for a model like
  286. 11:51chat GBT. Um and let's start with um
  287. 11:56uh the first aspect of fine finetuning
  288. 11:59which is kind of alignment with human
  289. 12:01preferences.
  290. 12:03So when a model is pre-trained it has no
  291. 12:06sense. It has seen all data on the
  292. 12:08internet on the in the books everywhere
  293. 12:11but it has no sense of like what is
  294. 12:13right and wrong and what is it just have
  295. 12:16this statistically knows about the state
  296. 12:18of the world but does it it doesn't
  297. 12:20exactly know things or know how to
  298. 12:21follow instructions. So the next step in
  299. 12:25training these models uh is the the the
  300. 12:28the step where we try to steer the AI
  301. 12:31models to follow the goals and
  302. 12:33preferences uh and values of humans. Um
  303. 12:38and this is still a big problem. It's a
  304. 12:40big issue. We we haven't like mastered
  305. 12:43it yet. But as the models become as we
  306. 12:46progress models are becoming better and
  307. 12:48much more uh kind of like uh powerful in
  308. 12:53the things that they can do. Um so it
  309. 12:55turns out that we can fine-tune the
  310. 12:57models to be aligned. For example, uh
  311. 13:00here are some graphs that show what if
  312. 13:02we take our base model that's just
  313. 13:05pretrain and fine-tune it for things
  314. 13:08like sensibleness or safety. And these
  315. 13:12are like these are data that we have
  316. 13:15collected. These are highquality data
  317. 13:17for kind of showing the model what what
  318. 13:20we mean but by what's safe and what's
  319. 13:22unsafe or what's sensible and what
  320. 13:24doesn't make sense. And these are like
  321. 13:26more cur like more highly curated data
  322. 13:29that we pass to the model for
  323. 13:31fine-tuning and to make the model more
  324. 13:33aligned with the values or the or the
  325. 13:35preferences of humans.
  326. 13:38Um
  327. 13:41so going back let's just walk through
  328. 13:44these process again. So
  329. 13:48the first stop when we take a
  330. 13:50pre-trained language model is uh
  331. 13:53fine-tuning with next token prediction
  332. 13:56which follows the same process as
  333. 13:58pre-training but this time on data that
  334. 14:01is much more higher quality. Uh for
  335. 14:04example, this could be data from books,
  336. 14:08creative essays, um data that companies
  337. 14:11pay a whole lot, millions of dollars or
  338. 14:13hundreds of millions of dollars
  339. 14:15potentially to buy this data that's
  340. 14:17really high quality and then we train
  341. 14:19the we fine-tune the model on those and
  342. 14:21the model becomes much better as a
  343. 14:24result of that.
  344. 14:26Um
  345. 14:27the the next step is instruction tuning
  346. 14:31and by instruction tuning what we mean
  347. 14:33is that we show the model again u tra
  348. 14:37traditionally in the past two years this
  349. 14:39in instruction tuning data is like a
  350. 14:42combination of human generated data or
  351. 14:44templates and then and then and the
  352. 14:46synthetic data. But the way this
  353. 14:48instruction t fine-tuning works is that
  354. 14:51we do have uh instruction and then
  355. 14:54question answer pairs uh where the model
  356. 14:58learns how to follow the questions and
  357. 15:01answers them. For example, here an
  358. 15:03instruction uh tuning data set could be
  359. 15:06about the weather like please answer the
  360. 15:09following question. What is the boiling
  361. 15:11point of nitrogen and then this is the
  362. 15:13answer. Um or we can have chain of
  363. 15:16thought uh fine-tuning where we um show
  364. 15:19the model how it can walk through a
  365. 15:21process to get to the answer and then uh
  366. 15:24we have the label we have the results
  367. 15:26over the here and this is like these
  368. 15:28pairs of like questions and instructions
  369. 15:30and answers and again this this data set
  370. 15:33is very um is is there's a lot of effort
  371. 15:37that goes into this data set and the
  372. 15:39quality of data set and the uh
  373. 15:41generality of it uh has a lot of impact
  374. 15:44on the quality of the model that we get
  375. 15:46at this stage. Um and after this point
  376. 15:49then the model becomes more and more
  377. 15:51like um like the models we see today
  378. 15:55like you can ask them questions you can
  379. 15:58go back and forth with them. They they
  380. 16:00have this sense of like how to uh
  381. 16:02perceive a question and how to walk
  382. 16:04through kind of uh some sort of like
  383. 16:07process to get to to an answer.
  384. 16:11And the next step after instruction
  385. 16:13fine-tuning is this process of um using
  386. 16:17human preferences and fine-tuning the
  387. 16:20model based on that. And this process is
  388. 16:22called RLHF or reinforce reinforcement
  389. 16:24learning from human feedback. Um the way
  390. 16:28this is different from the previous
  391. 16:30system is the way we create the
  392. 16:32objective function and uh create the
  393. 16:34data. So in this case instead of like
  394. 16:37just creating some supervised prompt and
  395. 16:40labels we are basically creating a
  396. 16:43reward model out of human preferences.
  397. 16:46So companies spend a lot of money uh
  398. 16:51asking humans, sometimes experts,
  399. 16:53sometimes just humans, normal humans, uh
  400. 16:57about questions and the answers
  401. 16:59generated by the model and we ask the
  402. 17:02the humans to rate them to see which one
  403. 17:05is correct and which one is not correct
  404. 17:07and that's how we can create some sort
  405. 17:09of a reward model here. Um and this
  406. 17:12reward model is instead instead of human
  407. 17:15then we can use the reward model in
  408. 17:17order to kind of like guide the
  409. 17:19parameters of our LLM towards uh
  410. 17:22generating answers that are in line with
  411. 17:25this reward model. So basically we want
  412. 17:27the the generations by the model uh to
  413. 17:30be such that this reward model says yes
  414. 17:32these are good generations. Um we're
  415. 17:34going to learn more about this here but
  416. 17:36I wanted we wanted you to have um the
  417. 17:39high level idea of what RLHF means. Uh
  418. 17:42and we can collect we can have different
  419. 17:45reward type different types of reward.
  420. 17:47For example, we could ask the model to
  421. 17:49reward a pair of answers based on their
  422. 17:52correctness or helpfulness or specity or
  423. 17:55harmlessness. all sort of reward
  424. 17:57functions that we can define and
  425. 17:58depending on how we want the models to
  426. 18:01be and what we care most about we can
  427. 18:04weight these different reward models and
  428. 18:06kind of use that in the RLHF process.
  429. 18:10So these step at a at a high level of
  430. 18:13course there's a lot of effort goes into
  431. 18:15that but like pre-training and then um
  432. 18:19fine-tuning on higher quality data
  433. 18:21instruction tuning and RLHF are were the
  434. 18:24core kind of components that made um a
  435. 18:29model like chat GPT over over something
  436. 18:32that existed over all the large models
  437. 18:35um like GPT3 and prior models. So that
  438. 18:38made a huge difference in how good and
  439. 18:41capable the models were. Um
  440. 18:45uh so so that was very cool up until
  441. 18:48last year or a year year and a half ago
  442. 18:52pre-training and then fine-tuning were
  443. 18:53the big pieces. Uh but since a year and
  444. 18:57a half ago, it turns out inference is
  445. 18:59also a frontier for making the models
  446. 19:02much more capable. Um so
  447. 19:06and that o opened like a whole set of
  448. 19:09whole lot of new research and directions
  449. 19:11and ways that we can make the model
  450. 19:13better. Um so let's think let's talk
  451. 19:17about it a little bit like this is a
  452. 19:19work we did in my lab last year uh last
  453. 19:24summer it was called large language
  454. 19:26monkeys um and it's kind of inspired by
  455. 19:29the infinite monkey theorem and it's
  456. 19:31related to in inference scaling um how
  457. 19:34many of you have heard of the infinite
  458. 19:36monkey theorem
  459. 19:38okay that's a good amount so the idea of
  460. 19:41the infinite monkey theorem uh what it
  461. 19:43says is that it's not proven, but what
  462. 19:47it says is that if you have a monkey and
  463. 19:49a kind of a typing machine and then we
  464. 19:52keep letting the monkey type all day
  465. 19:54forever, um after some time we will have
  466. 19:57the works of William Shakespeare in in
  467. 20:00the things that the monkey types. Um so
  468. 20:03we were inspired by that to name our
  469. 20:05project. Uh but the idea here was that
  470. 20:08let's let's have the LLM be the monkey
  471. 20:12and uh let's uh ask the model over and
  472. 20:16over again to solve a given input
  473. 20:19problem. Uh you can think of it as
  474. 20:22parallel sampling or parallel
  475. 20:24generation. So instead of asking the
  476. 20:26model once to solve the input problem,
  477. 20:27you can ask it many times. And also
  478. 20:30let's assume you have some sort of a
  479. 20:32verifier or selection mechanism. So you
  480. 20:35can pick which one of these generated
  481. 20:38responses were correct. Um and then you
  482. 20:42output that in your system that becomes
  483. 20:44the out the final output. So the model
  484. 20:47instead of generating one answer
  485. 20:48generates many many answers and then
  486. 20:50your verifier selects which one of them
  487. 20:52is correct and then outputs one of the
  488. 20:55correct ones. Um you can think of a
  489. 20:57verifier as say some unit test. If the
  490. 21:00problem is a generating code, the
  491. 21:02verifier can could be the unit test that
  492. 21:04we run against the code and we see which
  493. 21:07which one of the code generations passes
  494. 21:09all the unit tests and so on.
  495. 21:12Um the reason this is very this is a
  496. 21:16this can be a possible possibility here
  497. 21:18is that models are um do not are not uh
  498. 21:23there's some variance in the way the
  499. 21:25models generate responses. it's not
  500. 21:27deterministic and you can even control
  501. 21:29the variation in the model responses
  502. 21:32with this thing called temperature. So
  503. 21:34you can force the model or encourage the
  504. 21:36model to generate more different
  505. 21:38responses as you ask a question.
  506. 21:42Um and here now let's see some of the
  507. 21:46results of this repeated sampling for a
  508. 21:49bunch of tasks like math and coding
  509. 21:52benchmarks. we increase the number of
  510. 21:56samples per problem from one which is
  511. 21:59the normal one uh to 10,000
  512. 22:03and here we are showing the coverage or
  513. 22:05the fraction of problems that are solved
  514. 22:08by at least one of these samples and
  515. 22:11what we are seeing here is that um so
  516. 22:14the red line the red dash one was the
  517. 22:16GPT40
  518. 22:18model and while these blue and green
  519. 22:22lines which were the llama 8B, lama 3B
  520. 22:25and 7B were worse than GPT 40 model with
  521. 22:29one sample. But if we increase the
  522. 22:31number of samples from these models in
  523. 22:35all of these cases, they do better than
  524. 22:38the GPT40 model. And what it says is
  525. 22:43that it kind of seems like the model
  526. 22:45models already know a whole lot more
  527. 22:49than what you get out of them when you
  528. 22:51just ask them once, right? So this is
  529. 22:53the kind of the underlying property of
  530. 22:57inference scaling because it seems like
  531. 22:59we can get a whole lot of a whole lot
  532. 23:01more capability and problem solving kind
  533. 23:04of ability from these models if we just
  534. 23:07bring in this inference scaling. Here we
  535. 23:10are showing repeated sampling but there
  536. 23:11are many other ways to do that and we
  537. 23:14are so the reason it's called inference
  538. 23:16scaling is that we are not touching any
  539. 23:18of the parameters of the model. the
  540. 23:20model is fixed and we are just like at
  541. 23:22inference time we are creating different
  542. 23:24ways to generate to kind of produce
  543. 23:27generations from the model and use them
  544. 23:30to generate higher quality answers or
  545. 23:32answers to harder problems for the
  546. 23:34model. Um the interesting part here is
  547. 23:37that for some of these problems out of
  548. 23:39these 10,000 solutions maybe like three
  549. 23:42or four of them were correct for a
  550. 23:45single problem. So it shows that how how
  551. 23:48kind of um
  552. 23:50how important it is to to scale the
  553. 23:53inference to really get to the core
  554. 23:56capability of the models.
  555. 23:58But what is the difference like um with
  556. 24:01like traversing all the possible answers
  557. 24:03and just pick up like the correct like
  558. 24:07if the number of samples are here like I
  559. 24:10said like you need like 10,000 times
  560. 24:13sampling
  561. 24:16>> sorry what's the question
  562. 24:17>> what is the difference of things done
  563. 24:19with like traversing all the possible
  564. 24:21answers and and and
  565. 24:24>> so the possible answers need to be
  566. 24:26generated somehow. by the model, right?
  567. 24:30So, so this is like the space of all
  568. 24:33possible answers to the model is way
  569. 24:35bigger like like right we can't like if
  570. 24:38you do like a tree search or a random
  571. 24:40search is like way bigger right there's
  572. 24:42no way to like hit even like imagine
  573. 24:45like we are asking a human like just sit
  574. 24:48down and solve this problem right there
  575. 24:50is a limit to um of course there's no
  576. 24:54limit to human creativity humans
  577. 24:56creativity but there's a limit to how
  578. 24:58many generations we can have right or if
  579. 25:01we randomize thing there's a whole lot
  580. 25:03set of answers that can be generated and
  581. 25:06it's and this is like highly still
  582. 25:08highly sample efficient right 10,000 is
  583. 25:10not a whole lot especially like for
  584. 25:12example for the math um F2F some of
  585. 25:16these problem are really hard like IMO
  586. 25:19level problems like these are really
  587. 25:21hard problems that a tiny model like a
  588. 25:23seven and a llama 8p can solve
  589. 25:28>> I was wondering How
  590. 25:29>> how does the latency compare
  591. 25:32like when you do actually generate all
  592. 25:34those samples like compared to the GP4
  593. 25:37right like how does the latency
  594. 25:40is there a trade-off there
  595. 25:41>> yeah of course there's a trade-off um
  596. 25:44and we we have we touch on that in the
  597. 25:46in the lecture that I go more details
  598. 25:48towards this the parallel samples the
  599. 25:50good thing about them that they're they
  600. 25:52can be run in parallel um so from a
  601. 25:55latency perspective that's less of an
  602. 25:57issue but of There is the frontier of
  603. 25:59cost tradeoffs between how much compute
  604. 26:02you're spending and and we will look
  605. 26:04into that and it varies it varies
  606. 26:06depending on the type of and complexity
  607. 26:08of the problems.
  608. 26:10>> Yes.
  609. 26:12>> When you have a verifiable domain like
  610. 26:15mapping coding, how do you handle it
  611. 26:16when you don't have?
  612. 26:19>> Yes, great question again. It's a very
  613. 26:21hard when you have verifiers it's much
  614. 26:24easier when you don't have verifiers.
  615. 26:26There's this whole set of research on
  616. 26:28how to train LLM as judge or LLM as
  617. 26:32reward functions or LLMs with tools
  618. 26:34becoming and we're going to learn about
  619. 26:36all we have a lecture just dedicated to
  620. 26:38verifiers. So
  621. 26:42>> yes a quick question sorry so um does
  622. 26:45this generalize to any positive
  623. 26:48temperature? Um, so like as the like the
  624. 26:52temperature goes to infinity.
  625. 26:57>> Yeah, we can't like have the temperature
  626. 26:58being too high because it's just this
  627. 27:00going to be gibberish like the model
  628. 27:02doesn't generate. So we there's actually
  629. 27:05like a research project last year was
  630. 27:07like on tuning that or like figuring out
  631. 27:10what temperature usually if you go
  632. 27:12beyond like 1.2 or so it's it's not
  633. 27:15great. Yeah. Yeah. But there there other
  634. 27:19tricks that you can do to kind of kind
  635. 27:22of uh make the model be more um like
  636. 27:26have more diversity in the type of
  637. 27:28answers and that can be useful. Uh all
  638. 27:32right. Um
  639. 27:36so
  640. 27:38let's look at this. So now again going
  641. 27:42back to the events of last year year and
  642. 27:45a half um we had the deepseek model.
  643. 27:48Well, I think it came out in
  644. 27:49>> December
  645. 27:50>> December Christmas.
  646. 27:52>> Deeply came out December 2024.
  647. 27:55>> Uh oh. Wow. [laughter]
  648. 27:58>> Short time or long time I don't know
  649. 28:00both. U so so when models like deepse
  650. 28:04came last year. So basically a core kind
  651. 28:07of innovation in deepseek and then the
  652. 28:1001 series Gemini thinking and such were
  653. 28:12that now we are kind of bringing this
  654. 28:15fine-tuning and test time scaling
  655. 28:16together because with this test time
  656. 28:19scaling and the models themselves we
  657. 28:21have this new like kind of engine that
  658. 28:24we can generate a whole lot of synthetic
  659. 28:26data um for for example for problems
  660. 28:31like for math problems where we know the
  661. 28:33answer we can have the model generate
  662. 28:35different like kind of answers that
  663. 28:36leads to that final uh final kind of uh
  664. 28:39golden answer. Or for the coding
  665. 28:42problems, we can use the model to
  666. 28:44generate tons of data during test time
  667. 28:46scaling uh tons of like quality data to
  668. 28:49solve coding problems and that becomes
  669. 28:51part of our training set. Now we can use
  670. 28:53that to fine-tune the model um to to
  671. 28:56become better. So that kind of bringing
  672. 28:59these two together became a big piece in
  673. 29:02thinking models and reasoning models. Um
  674. 29:06and this this is like a this is like a
  675. 29:08an area that is like really really
  676. 29:11important. There's just so much to be
  677. 29:13explored in this because again the
  678. 29:15there's no it's kind of open-ended,
  679. 29:17right? There's no boundary in how good
  680. 29:19the models can become with test time
  681. 29:21scaling and then bringing that back to
  682. 29:24the process of training the model or
  683. 29:26fine-tuning the model to become better
  684. 29:28is very exciting and that's kind of the
  685. 29:31self-improving piece that we are very
  686. 29:33excited about.
  687. 29:36Um so a little bit about reasoning
  688. 29:38models. Um in the language monkeys what
  689. 29:41we were showing was the coverage like if
  690. 29:44we had access to a verifiers how uh we
  691. 29:47we can see the kind of a log linear kind
  692. 29:49of scaling logs with the number of
  693. 29:50samples. Um and open AAI when they
  694. 29:55released 01 uh last September um they
  695. 29:58showed a the log linear relationship
  696. 30:01this time with passad one for these like
  697. 30:04hard hard Amy uh benchmark which is like
  698. 30:08a set of really difficult math
  699. 30:10questions. So as they're show what
  700. 30:13they're showing here is that as they're
  701. 30:15increasing test time compute and this is
  702. 30:17in log scale um uh the the accuracy the
  703. 30:22pass had one accuracy of the model goes
  704. 30:24up and this is for test time alone. Uh
  705. 30:27previously uh it this has been shown for
  706. 30:30training but it's very interesting that
  707. 30:32this kind of scaling applies to test
  708. 30:34time as well um without changing the
  709. 30:37parameter count of the model. Um
  710. 30:42um I'm going to walk you through like
  711. 30:44one a few examples of how this kind of
  712. 30:47reasoning or test time uh scaling uh
  713. 30:50kind of is works with models like 03 and
  714. 30:54or Gemini. So one motivation here is
  715. 30:58that for difficult problems just like
  716. 31:00humans think a lot more spend a lot more
  717. 31:03time or consider many different
  718. 31:05strategies uh thinking models do the
  719. 31:07same and they may use a chain of thought
  720. 31:11or other techniques to just go go about
  721. 31:14like solving a problem. Um and there are
  722. 31:18different steps to that. For example, um
  723. 31:22problem analysis is one step like the
  724. 31:24model first sees a problem and tries to
  725. 31:26analyze it. Um then it can do the task
  726. 31:29decomposition breaking a task into kind
  727. 31:32of simpler tasks that are more
  728. 31:35addressable. Um then there is the self
  729. 31:38evolution strategies where the model
  730. 31:41kind of tries something sees the
  731. 31:43feedback for example runs some tests on
  732. 31:47the code or use a uses a calculator or
  733. 31:51just judges the answer and it can use
  734. 31:53that feedback to optimize itself. Um
  735. 31:55there's selfcorrection and there are
  736. 31:58alternative proposals. If something
  737. 32:00doesn't work, the model can backtrack
  738. 32:02and kind of um try a different approach.
  739. 32:05These are some key principle principles
  740. 32:08that the model uh probably at some point
  741. 32:12uh pieces of it were trained as part of
  742. 32:14the training kind of data set were
  743. 32:16curated by humans. But a big part of it
  744. 32:19is also the model kind of acquiring
  745. 32:22these skills during this fine-tuning
  746. 32:25process and RL like RL from synthetic
  747. 32:28data process that the model does. Um
  748. 32:31here is an example of like how one does
  749. 32:35analysis. Um the model is asked to write
  750. 32:38a bash script um that takes a matrix and
  751. 32:43then outputs the transpose of the that
  752. 32:45matrix. So the model starts with ju just
  753. 32:48like humans like starts like thinking
  754. 32:52what are the pieces important pieces
  755. 32:53here. So the user is requesting a bash
  756. 32:56script. Um let's understand the input
  757. 32:58and output uh formats of this matrix. Um
  758. 33:03and and so on. So the model just thinks
  759. 33:05itself. Um and amazingly this kind of
  760. 33:09like it's like a progression of chain of
  761. 33:12thought. This is like helping the model
  762. 33:14finding the answer. But the difference
  763. 33:16between chain of thought and this is
  764. 33:17that the model itself is producing this
  765. 33:20chain of thought.
  766. 33:22Um there's of course task decomposition
  767. 33:25like in this case for the approach uh
  768. 33:28the model thinks about parsing the input
  769. 33:31building the matrix as an array of as as
  770. 33:34an array of arrays and so on. Um there's
  771. 33:37also selfcorrection which is a big
  772. 33:39important piece of um this is an
  773. 33:42important piece here and I don't know
  774. 33:44how many of you have seen that when
  775. 33:45trying with the models that you can see
  776. 33:48the thinking traces uh and this is
  777. 33:50actually pretty common like the model
  778. 33:52starts something and then in the middle
  779. 33:53says wait the correct for like there's
  780. 33:56something wrong maybe um or and maybe I
  781. 34:00need to fix that and [laughter] it's
  782. 34:02very very cool that the model can do
  783. 34:03that itself um So reasoning models when
  784. 34:06they came out like models like 01
  785. 34:09compared to a model like GPT40 which
  786. 34:11wasn't a reasoning model um they tend to
  787. 34:15be better in obviously the reasoning
  788. 34:17task for example in math calculation
  789. 34:20data analysis programming and such they
  790. 34:24all kind of outperform GPT40 but not
  791. 34:28necessarily in personal writing or
  792. 34:30editing text and so on. Yeah. Question.
  793. 34:35[clears throat]
  794. 34:35>> Do you think the gains due to this
  795. 34:38asking for a reasoning choice is due to
  796. 34:40actually the act of [clears throat]
  797. 34:41generating reasoning trace or just the
  798. 34:44act of asking the model to think or
  799. 34:47these decomposition at all? Like for
  800. 34:49example, if you said like think step by
  801. 34:51step but don't generate out loud like
  802. 34:54don't actually thought about it without
  803. 34:58saying.
  804. 34:59>> Yeah. Um I repeat the question like do
  805. 35:03you're saying that is it the do you
  806. 35:05think the reasoning models are becoming
  807. 35:07what they are is based on the is it
  808. 35:10based on the loud way of thinking
  809. 35:13>> I I think the question is that is it
  810. 35:15because of the um reasoning choices or
  811. 35:18is it because of things step by step the
  812. 35:19chain of thought is it [clears throat]
  813. 35:20like is it the data that's making a
  814. 35:22difference or is it the the instruction
  815. 35:25um
  816. 35:25>> it's I mean so the data basically has
  817. 35:28made the model kind of become a
  818. 35:30generalized thinker, right? Like the
  819. 35:33model can and the all of it all of these
  820. 35:36steps like breaking down a problem,
  821. 35:39being able to backtrack, being being
  822. 35:41able to do analysis, all of these are
  823. 35:43these skills that the model has learned
  824. 35:46during this thinking optimization
  825. 35:48process that now generalize to other
  826. 35:50things and they're helping the model
  827. 35:53right now.
  828. 35:53>> So I think the other way to look at it
  829. 35:55is Zalia presented repeated sampling. So
  830. 35:57if you had just the base model right
  831. 36:00after pre-training it would be able to
  832. 36:02do some reasonings but at the same time
  833. 36:04it will generate different kinds of
  834. 36:06reasoning chains and it doesn't quite
  835. 36:08know which one is correct. So a lot of
  836. 36:10the what we will cover in train time or
  837. 36:13test time scaling really comes down to
  838. 36:15it learns which is correct. So pass one
  839. 36:18accuracy goes up which is what you were
  840. 36:20seeing in the reasoning models as
  841. 36:21opposed to like passet k or coverage uh
  842. 36:24which is what you were seeing in the
  843. 36:25repeated sampling results.
  844. 36:27>> Yeah. And and one other way to think
  845. 36:30about like yes these kind of like kind
  846. 36:32of deep thinking and things that the
  847. 36:35model generates is actually helping the
  848. 36:38model every time to solve a new problem.
  849. 36:41Um and although it's like very expensive
  850. 36:45for the model to generate those things
  851. 36:47still like we want them to generate
  852. 36:50those because it leads to better
  853. 36:52answers. Um
  854. 36:55any other questions? Yes.
  855. 36:57>> We use a different model for the
  856. 36:59reasoning step versus generating the
  857. 37:01final answer like maybe a smaller or
  858. 37:03like a purposeful one just the final
  859. 37:05answer is a large model. Um
  860. 37:09so typically the reasoning capabilities
  861. 37:11have gone up with the model size. So if
  862. 37:13if anything uh you would use the larger
  863. 37:17model reasoning traces and then maybe
  864. 37:19you collect a bunch of reasoning traces
  865. 37:21and then have a smaller model summarize
  866. 37:23the answer.
  867. 37:23>> Yeah. So the models like this is like at
  868. 37:26least currently things may change. They
  869. 37:28like their own traces more even if the
  870. 37:31the traces are coming from a better
  871. 37:34model. they tend to like their traces
  872. 37:37their own generated traces more and
  873. 37:39we're going to have talk about swirl.
  874. 37:41This is one of the papers that uh
  875. 37:43>> the multi-step reasoning
  876. 37:44>> the multi-step reasoning that we talk
  877. 37:46about what if we bring a different model
  878. 37:49not in the context of generating reason
  879. 37:51interests but in in the context of like
  880. 37:53evaluating and giving feedback and we
  881. 37:56we're going to see that as well but
  882. 37:58models surprisingly or not surprisingly
  883. 38:01they like their own traces a lot more.
  884. 38:04>> For reasoning models how are they taught
  885. 38:07to reason? Is it a hardcoded set of
  886. 38:09sequences they're taught to do traces or
  887. 38:11they somehow fine-tune to say you need
  888. 38:14to do this much or this kind of thing
  889. 38:16which will cause the traces to generate.
  890. 38:19>> So I don't think there's a published
  891. 38:20piece of work that really covers this
  892. 38:22bit. Um
  893. 38:23>> both yeah
  894. 38:24>> but it's a bit of both and at the same
  895. 38:26time the base model did have the
  896. 38:29thinking capability to begin with right
  897. 38:31it would go do in some go do some amount
  898. 38:34of thinking. One thing that we will
  899. 38:35cover in the class is uh this notion of
  900. 38:37outcome rewards model and process reward
  901. 38:39model in how you can use that feedback
  902. 38:42to get the model back. Yeah, there's
  903. 38:45always some bootstrapping like okay here
  904. 38:48are different ways like chain of thought
  905. 38:50itself which is heavily used in the
  906. 38:52instruction tuning of data set tell
  907. 38:55shows the model some ways of like how to
  908. 38:58think um and there could be like some
  909. 39:00templates showing uh for the model
  910. 39:03fine-tuning but the things that a model
  911. 39:06if you just ask them any ask it any
  912. 39:08question and it comes up with that
  913. 39:09there's a whole lot of generalization
  914. 39:11that so models have gone way above like
  915. 39:14what the data or instructions that are
  916. 39:16used for training them.
  917. 39:20[snorts]
  918. 39:20>> Yes.
  919. 39:22>> Is there a way to make the number of
  920. 39:24samples dependent on problem?
  921. 39:27>> Please uh speak up the number of
  922. 39:28samples. Can they be dependent on
  923. 39:30problem difficulty?
  924. 39:33>> I don't think there's a published piece
  925. 39:34of work that does that. uh but uh
  926. 39:37>> yeah we just so
  927. 39:40so when so there is follow-up work to
  928. 39:43that where we you use a reward model and
  929. 39:46then you can use that so assuming the
  930. 39:48reward model has some notion of
  931. 39:50complexity if you haven't solved it it
  932. 39:51can guide like more sampling or whether
  933. 39:54it's repeated or or parallel uh but
  934. 39:57that's definitely an interesting uh
  935. 39:59direction to explore yeah
  936. 40:03okay we need to get to the Oh, you
  937. 40:06haven't, right?
  938. 40:07>> Yeah, let me record this. Okay, so we
  939. 40:10you now know about large language models
  940. 40:13and then you heard about how they
  941. 40:15learned how to think and then they
  942. 40:16learned how to reason. So, what's next
  943. 40:19and why does this why is this course
  944. 40:20relevant? So, a lot of what this course
  945. 40:22is about is that large language models
  946. 40:25as chat bots or as uh reasoning models
  947. 40:28are basically still single turn like or
  948. 40:30or just like in the chat format. So,
  949. 40:32they're not accomplishing a task for
  950. 40:33you. They're fun to interact with, but
  951. 40:36they're not necessarily accomplishing a
  952. 40:37task for you. What has happened this
  953. 40:39year uh almost in the last few months
  954. 40:41and it's been surprising is that agents
  955. 40:44like cloud code or deep research have
  956. 40:46really enabled people to do real world
  957. 40:49workflows. Um so there are agentic
  958. 40:51workflows which can achieve tasks that
  959. 40:53you ask them to do end to end. So for
  960. 40:55example, if you want to go research and
  961. 40:57figure out where should I uh rent a home
  962. 41:01uh perhaps for uh the entire uh year for
  963. 41:04uh if I want to take a class at
  964. 41:06Stanford, the model would actually be
  965. 41:08able to do a whole bunch of analysis and
  966. 41:10like go look at a lot of different
  967. 41:12websites and actually summarize the
  968. 41:14results and give you pros and cons of
  969. 41:16different places. This used to not be
  970. 41:18possible before. So this uh we'll cover
  971. 41:20a little bit of how that that becomes
  972. 41:22possible and how this course allows you
  973. 41:23to learn about that and then cloud code
  974. 41:26for example if any of you are codeex uh
  975. 41:29which is the coding agent from openai if
  976. 41:31you're using that what you'll see is
  977. 41:33that just by giving instructions in
  978. 41:35English you can modify files or you can
  979. 41:37like figure out test cases and and
  980. 41:39whatnot so it really uh is become a
  981. 41:42coding productivity tool in in the
  982. 41:44day-to-day workflows for software
  983. 41:45engineers at this point in time so what
  984. 41:48what is the transition from LM to
  985. 41:50agents. So as I was giving you the
  986. 41:52example of deep research and of uh
  987. 41:55coding agents basically the goal
  988. 41:58basically now the uh model can be given
  989. 42:00a goal and it will plan out the steps
  990. 42:03then it will go interact with uh the
  991. 42:05environment and based on the feedback it
  992. 42:08will correct its steps until it achieves
  993. 42:09the goals or it will come back and say I
  994. 42:11cannot achieve the goal. So this notion
  995. 42:13of like having some notion of a goal uh
  996. 42:17taking actions towards that goal getting
  997. 42:19the feedback and then deciding when to
  998. 42:22stop that's what makes agents uh
  999. 42:24different from the chatbot times of what
  1000. 42:27we were doing before and um this also
  1001. 42:30might require interacting with tools uh
  1002. 42:33which are external to what the model
  1003. 42:36itself is doing getting some inputs from
  1004. 42:37there but it still stays on track of
  1005. 42:40like whatever task it's uh choosing to
  1006. 42:42accomplish which might mean that it has
  1007. 42:44to have some form of memory to keep
  1008. 42:46track of the task that it's trying to
  1009. 42:48accomplish.
  1010. 42:50So what has been uh accomplished? So
  1011. 42:52like in in in several cases in simpler
  1012. 42:54cases you can achieve these uh um these
  1013. 42:57end to-end goals like deep research can
  1014. 43:00like be accomplished end to end but in
  1015. 43:01most scenarios you still having very
  1016. 43:03static workflows. So today what exists
  1017. 43:06is closer to agentic workflows where you
  1018. 43:08have um some sort of an input where
  1019. 43:10you're giving a model the goal and then
  1020. 43:12um what this slide is showing you is
  1021. 43:14that one model is perhaps uh giving the
  1022. 43:17output like a solution and then another
  1023. 43:19model is judging it and then deciding
  1024. 43:21based on that whether the solution
  1025. 43:23should be accepted. So that's like one
  1026. 43:25orchestration framework that's possible.
  1027. 43:27uh deep research would be more of the
  1028. 43:29second one where you're basically
  1029. 43:30generating mult uh you're calling the
  1030. 43:33LLM on multiple possible inputs and then
  1031. 43:36aggregating the um things to get the
  1032. 43:39output which will be a summary of the
  1033. 43:41deep research results. So this is a very
  1034. 43:43cartoonish abstraction of what agentic
  1035. 43:46workflows might look like and you can
  1036. 43:47generalize from there. But instead of
  1037. 43:49having like very open-ended loop that I
  1038. 43:52was showing you here where you're
  1039. 43:54supposed to go take an action from the
  1040. 43:55environment and come back and do some
  1041. 43:57feedback. Um it's easier for open-ended
  1042. 44:01problems to like construct this graph by
  1043. 44:02hand of like how a human would do it and
  1044. 44:04then like get this feedback perhaps from
  1045. 44:06an LLM which is the LM evaluator here.
  1046. 44:09So a lot of the real world workflows
  1047. 44:12still have this paradigm but in in
  1048. 44:14certain cases we are starting to see
  1049. 44:16signs of life for what's possible in
  1050. 44:18this slide and coding being one and
  1051. 44:20research being another one.
  1052. 44:23So typically the workflows will have
  1053. 44:25things like LM calls. So you would
  1054. 44:27basically call an LM with an instruction
  1055. 44:29or some form of like input and then
  1056. 44:31you're asking the LM to give you an
  1057. 44:33output. It will have some form of
  1058. 44:35verifiers. We'll cover verifiers in a
  1059. 44:36whole lecture by itself. It will have
  1060. 44:38some form of critics or judges which is
  1061. 44:41effectively LLM as a judge uh paradigm.
  1062. 44:44Uh there might be tool calls for example
  1063. 44:46for deep research you have to actually
  1064. 44:47go search the web to figure out what
  1065. 44:50exactly is the content that you should
  1066. 44:52be looking at. So there might be tool
  1067. 44:54calls of that form or uh getting uh some
  1068. 44:56value around what exactly is the weather
  1069. 45:00um and similarly search would be another
  1070. 45:03tool call and then you would orchestrate
  1071. 45:05these in in some form of a workflow. So
  1072. 45:08a simplest workflow here would be prompt
  1073. 45:10chaining. So similar to reasoning models
  1074. 45:12where the task gets decomposed into
  1075. 45:14subtasks. Prompt chaining will simply be
  1076. 45:16like you have chained a bunch of
  1077. 45:17different subtasks that you're supposed
  1078. 45:19to go accomplish to achieve the end to
  1079. 45:21end goal. There might be routing.
  1080. 45:23Routing works for complex tasks where
  1081. 45:25you say okay if this is very complex
  1082. 45:27then go to this more complicated set of
  1083. 45:30LLM calls and it's less complicated then
  1084. 45:32do this simpler workflow. Um there might
  1085. 45:35be parallelization where you like deep
  1086. 45:37research is one example where you you
  1087. 45:39can have multiple LLM calls work
  1088. 45:41simultaneously on researching different
  1089. 45:43keywords that you give as input to the
  1090. 45:45LLM and then uh finally you aggregate
  1091. 45:48the output or you might break the task
  1092. 45:50into independent subtasks and then uh
  1093. 45:52combine uh the solution or you might
  1094. 45:55have this notion of orchestrator where
  1095. 45:57you basically are actually using some
  1096. 45:59sort of a
  1097. 46:02in simple words LM manager so a central
  1098. 46:04LM is actually doing the planning. So in
  1099. 46:06cloud code you actually start to see
  1100. 46:07that that there's some notion of like a
  1101. 46:09plan that it comes up with and then
  1102. 46:11based on that plan it will like make
  1103. 46:13subsequent LM calls. You might have an
  1104. 46:16evaluator or a judge. So instead of like
  1105. 46:18getting feedback from real world like
  1106. 46:20user or from some sort of a actual
  1107. 46:22running a unit test, it actually might
  1108. 46:24just use LLM as a judge and we will have
  1109. 46:27uh some homeworks that will cover this
  1110. 46:28aspect. And then you might have
  1111. 46:30verifiers. Verifiers are things where
  1112. 46:32you can actually verify the output. So
  1113. 46:34in code for example, if you run the
  1114. 46:37code, how do you know that this code is
  1115. 46:38correct? So typically as software
  1116. 46:40developers, we write unit tests. Uh so
  1117. 46:42similarly verifiers might be running
  1118. 46:44some kind of unit test to check whether
  1119. 46:47what LM generated is correct and in
  1120. 46:49domains which are verifiable uh math
  1121. 46:52code um and other domains that are more
  1122. 46:54rule-based uh this verification is a
  1123. 46:56good way to give feedback back to the
  1124. 46:58model so that um it can correct its
  1125. 47:01steps. So most of these u workflows mean
  1126. 47:05that the uh the LM needs to be better at
  1127. 47:08planning. It needs to be better at
  1128. 47:09multi-step reasoning and it needs to be
  1129. 47:11better at self-improvement like when it
  1130. 47:13makes mistakes it needs to be able to
  1131. 47:14correct itself. So effectively these are
  1132. 47:17sort of new paradigms that the current
  1133. 47:19set of LLMs just with reasoning were not
  1134. 47:22quite accomplishing and these are some
  1135. 47:24of the topics we do plan to cover in
  1136. 47:26subsequent lectures
  1137. 47:29and just to uh drive the point home
  1138. 47:31around uh coding agents. So this is a
  1139. 47:34very simple example. This is actually um
  1140. 47:36I should have updated the slide to for
  1141. 47:38cloud code but basically what this is
  1142. 47:40showing is that you have an LM agent and
  1143. 47:43uh it's interacting with the computer
  1144. 47:45these days just with a terminal. You
  1145. 47:47give it an instruction where you might
  1146. 47:48tell it that you want to implement a
  1147. 47:50test. Um and then it has some navigation
  1148. 47:53of repositories, searching of files, all
  1149. 47:55of these tool calls around like viewing
  1150. 47:57files, editing lines and then it will go
  1151. 47:59execute a bunch of commands in the
  1152. 48:01terminal and based on the output it
  1153. 48:03might be like okay I need to go edit
  1154. 48:05this other file or I need to go look at
  1155. 48:07this other file. So if you look at this
  1156. 48:09loop this very much mirrors u what was
  1157. 48:12so this was not quite reliable last year
  1158. 48:15and it's just starting to get reliable
  1159. 48:16in what we are seeing in the uh realm of
  1160. 48:18coding agents now.
  1161. 48:20Yes.
  1162. 48:21>> Can I ask why you think it's more
  1163. 48:24reliable with I guess the general
  1164. 48:27I mean I would say not change that much.
  1165. 48:31>> I mean I think the paradigm is very much
  1166. 48:33the same. It's it's mostly a matter of
  1167. 48:35more powerful models and then better RL
  1168. 48:41verifiable rewards is working. The train
  1169. 48:42time scaling is working well.
  1170. 48:47I guess the most
  1171. 48:49improve I guess working on performance
  1172. 48:51without necessarily just throwing more
  1173. 48:55comput power
  1174. 48:57>> I mean computer data right one of those
  1175. 49:00two but like just also once the models
  1176. 49:03start to get better there's the
  1177. 49:04self-improvement loop that kicks in uh
  1178. 49:07because you can now generate tests and
  1179. 49:09the tests become more reliable so in the
  1180. 49:11published pieces of work if you look at
  1181. 49:12say code monkeys which we covered last
  1182. 49:14year um if you can generate unit tests
  1183. 49:17um for whatever code the model generated
  1184. 49:20and then see if the generated unit tests
  1185. 49:23are
  1186. 49:25making things better then that that
  1187. 49:26that's a very good way to verify things
  1188. 49:28right
  1189. 49:29>> yeah like model based it seems like to
  1190. 49:32some extent if we have good verifiers
  1191. 49:35um we can make the model generate the
  1192. 49:38the right code
  1193. 49:41uh but the question is how do we do that
  1194. 49:43and there are ways to do it and we we we
  1195. 49:46would be happy to for you guys to take
  1196. 49:49research projects in that domain too.
  1197. 49:51Obviously, it's an active um research
  1198. 49:54area, but there are ways to make um
  1199. 49:56improvements that you can
  1200. 49:58>> I think what you're alluding to is this
  1201. 49:59notion of generator verifier gap. So,
  1202. 50:01it's easy for models to generate a whole
  1203. 50:03bunch of nonsense or sensible set of
  1204. 50:06like reasoning traces or like useful set
  1205. 50:08of content. But at the end of the day,
  1206. 50:10whether that's useful or not, we need
  1207. 50:12feedback loop for that. And if you're
  1208. 50:14creative writing, how much feedback can
  1209. 50:15you get? So, human feedback ends up
  1210. 50:17becoming a bottleneck. In domains where
  1211. 50:19you can have good feedback, that's where
  1212. 50:21it's possible to continue to improve the
  1213. 50:23model and that like but ver getting
  1214. 50:25robust verification a lot of robust
  1215. 50:27verification is hard. So Aalia will
  1216. 50:30cover a paper that she did in her lab
  1217. 50:32about like how to combine verifiers and
  1218. 50:34but verification continues to be one of
  1219. 50:36the bottlenecks in this space to make
  1220. 50:38them better.
  1221. 50:40>> Yes. Um just like intuitively I
  1222. 50:43pre-training you get ability
  1223. 50:47such a large
  1224. 50:49>> so just like to me intuitively I'm
  1225. 50:50trying to figure out why there's such a
  1226. 50:52big job
  1227. 50:53>> because like we're sort of providing a
  1228. 50:55signal and we're trying that I guess
  1229. 50:59shouldn't exist.
  1230. 51:03So yeah, I I guess like I'm just not
  1231. 51:07re like
  1232. 51:08>> I I think this is still an active area
  1233. 51:10of research. So there's different set of
  1234. 51:12opinions around what really improves the
  1235. 51:14model. Is it RL or is it like the the
  1236. 51:17pre-training the diverse data by itself
  1237. 51:20and I I don't think there's single point
  1238. 51:22of consensus at this point in time. Both
  1239. 51:24processes help. um what you just said uh
  1240. 51:27which I'll repeat for the class is that
  1241. 51:29if pre-training is the place where uh
  1242. 51:32the model after repeated sampling should
  1243. 51:33be able to at least have one solution
  1244. 51:35correct out of if it was generating a
  1245. 51:37large number of samples then uh this
  1246. 51:39notion of giving feedback should improve
  1247. 51:41the pass at one accuracy but should not
  1248. 51:43improve the model why is it like such a
  1249. 51:45big jump but I think um that whole loop
  1250. 51:48is not completely well understood it's
  1251. 51:50like the first signs of life and it
  1252. 51:51starts to get commercialized but I think
  1253. 51:53there's a lot more research still open
  1254. 51:55in this area.
  1255. 51:56>> Yeah.
  1256. 51:57>> So that's only one uh way of thinking
  1257. 51:59about it. There are other um at least
  1258. 52:01some signs of life which say that you
  1259. 52:03can continue to do RL and then that will
  1260. 52:06continue to improve the model.
  1261. 52:07>> Yeah.
  1262. 52:10>> Okay.
  1263. 52:10>> Yeah.
  1264. 52:12So uh so this is uh basically covering a
  1265. 52:15lot of uh the same set of points but uh
  1266. 52:17one abstraction that is worth taking
  1267. 52:19away from this particular chart is that
  1268. 52:22even when you give the model a goal it
  1269. 52:24has to clarify the user intent. So it's
  1270. 52:26like not always obvious to it what the
  1271. 52:28user wants and then it might go search
  1272. 52:30for relevant files and then whatever um
  1273. 52:33set of actions it chooses to take often
  1274. 52:36times it needs some form of
  1275. 52:37verification. In this particular case,
  1276. 52:39it's based on um passing the test and it
  1277. 52:42might actually generate the test that it
  1278. 52:44needs to pass which um which is also uh
  1279. 52:48mentioned here. But the the key idea is
  1280. 52:50that if you give it a task, how does it
  1281. 52:53come back and how do you know that it's
  1282. 52:54going to complete what you asked it for.
  1283. 52:57Often times the users will not specify
  1284. 52:59the problem well enough. So asking
  1285. 53:01clarifying the user intent so that it
  1286. 53:03knows what to go look for and how to
  1287. 53:05verify starts to become uh important.
  1288. 53:07And in models like 03 I think they have
  1289. 53:09seen a lot of these traces end to end.
  1290. 53:11So they are able to like do the planning
  1291. 53:13the reasoning the multi-step reasoning
  1292. 53:14and like come back with the conclusion.
  1293. 53:16But uh a lot of this um is needed for um
  1294. 53:20completing tasks end to end or having an
  1295. 53:22end to end goal. And where this has been
  1296. 53:24super useful is uh if you have
  1297. 53:26repetitive uh repetitive tasks like code
  1298. 53:29migrations or version upgrades or if you
  1299. 53:31need to restructure the codebase or if
  1300. 53:33you have tasks that involve data
  1301. 53:35engineering where you have to basically
  1302. 53:37do some sort of extract uh the data and
  1303. 53:39then do some sort of clean cleanup on
  1304. 53:41the data or if you have to do some sort
  1305. 53:43of data warehouse migration. So a lot of
  1306. 53:45this work tends to be extremely um
  1307. 53:48repetitive and it's much easier to just
  1308. 53:51delegate it to coding agents. uh same
  1309. 53:53for unit tests often times like much
  1310. 53:55easier to like start to give it to uh
  1311. 53:57coding agents. Another area where agents
  1312. 54:00have become uh extremely prevalent is in
  1313. 54:03customer support. It's one of the most
  1314. 54:05thankless jobs that u when when you are
  1315. 54:07on the customer support side of things
  1316. 54:09you have to do and using LLMs to do that
  1317. 54:11definitely makes um streamlines the
  1318. 54:13experience in interesting ways. So one
  1319. 54:15simple set of examples ends up being
  1320. 54:17like you can use them to do live
  1321. 54:18transcription. So that gives you a very
  1322. 54:20nice record and and you're seeing that
  1323. 54:22even for meetings these days. Another
  1324. 54:24simple is example is that you can have
  1325. 54:27knowledge assist. So if you have a
  1326. 54:28database of information, the the
  1327. 54:30customer support agent does not need to
  1328. 54:32know everything. They can consult with
  1329. 54:34the LLM and and get an answer and
  1330. 54:36surface the relevant article. Uh which
  1331. 54:39is better than having um just search um
  1332. 54:42like an index and a search by itself.
  1333. 54:45Uh smart reply. So in in chat often
  1334. 54:48times this has existed for a while but
  1335. 54:50you can use an agent to like give uh
  1336. 54:52chat responses and then having a call
  1337. 54:55summary can help you um you can use uh
  1338. 54:58the call summary to really uh improve
  1339. 55:01the customer experience by itself as
  1340. 55:02well. So overall I think in the customer
  1341. 55:04support area there's multiple uh
  1342. 55:07companies that are going after this area
  1343. 55:09but uh there are different segments of
  1344. 55:11the problem that can be addressed using
  1345. 55:12LLMs and they have been super useful and
  1346. 55:15then there's also the end to end stuff
  1347. 55:16that's starting to happen here
  1348. 55:19and then the third example that we will
  1349. 55:21use and actually some of the homeworks
  1350. 55:23is that if you have very complex topics
  1351. 55:25and uh you want to provide a
  1352. 55:26comprehensive report um earlier it used
  1353. 55:29to be that you had to do a lit review
  1354. 55:30and then you had to summarize each paper
  1355. 55:32and then you have to synthesize these
  1356. 55:33things and these days you can just give
  1357. 55:35it to an LLM and actually do it for you
  1358. 55:37and um study maybe more articles than
  1359. 55:40you would. So typically if you give it
  1360. 55:42an example like u say 2022 winter
  1361. 55:45Olympics opening ceremony it will
  1362. 55:48identify what references to go look at
  1363. 55:50then it will construct an outline of
  1364. 55:51like okay these references are relevant
  1365. 55:53or not and then summarize each of the
  1366. 55:56references uh as to what is the relevant
  1367. 55:59content and then combine them to create
  1368. 56:01a fulllength article um which is u very
  1369. 56:04impressive in certain cases and and
  1370. 56:06you'll get to try it in one of the
  1371. 56:07homeworks
  1372. 56:09and then even more forwardlooking um
  1373. 56:13these research agents are starting to be
  1374. 56:16used for uh as AI scientists. So
  1375. 56:18basically they're starting to assist
  1376. 56:20scientists in say uh solving math
  1377. 56:22problems or solving science problems. So
  1378. 56:24here the LLM is used as a brainstorming
  1379. 56:27thing to come up with ideas. So the idea
  1380. 56:29generation phase then the experiment
  1381. 56:31iteration phase it might actually help
  1382. 56:33you iterate on the experiments that you
  1383. 56:35are hoping to work on. This is from the
  1384. 56:37AI scientist paper. And then in the
  1385. 56:40paper write up phase it will help you
  1386. 56:42improve uh the the paper write up by
  1387. 56:44itself.
  1388. 56:47And what's interesting uh is that
  1389. 56:51even even though these models
  1390. 56:52hallucinate just the just the notion
  1391. 56:55that they can come up with so many
  1392. 56:56different set of ideas as an AI
  1393. 56:59scientist it might actually brainstorm
  1394. 57:00or like give you ideas outside what you
  1395. 57:03would have thought if you had just like
  1396. 57:05taken a bunch of courses or or if you're
  1397. 57:06a researcher and you've been in the
  1398. 57:08field for a long time. Sometimes the
  1399. 57:10reading the web uh allows these LLMs to
  1400. 57:13come up with ideas that are way outside
  1401. 57:14the box and can be a very good
  1402. 57:16brainstorming um way. Uh so this these
  1403. 57:19have been super useful for in the AI
  1404. 57:21scientist style of work. Okay. So let's
  1405. 57:25move to course logistics. But before
  1406. 57:27that do we have any questions?
  1407. 57:29>> Yes.
  1408. 57:31>> I just want to clarify. So um is
  1409. 57:33reasoning and chain of thought baked in
  1410. 57:35training or is it just prompt
  1411. 57:37engineering that was accidentally
  1412. 57:39discovering these kind of capabilities
  1413. 57:40in larger models and if so are there
  1414. 57:43other emerging behaviors besides reason
  1415. 57:47that you think might be discovered?
  1416. 57:50>> So uh I mean it was not baked in by uh
  1417. 57:54by design. uh it was it was basically
  1418. 57:57discovered as in like it was I mean we
  1419. 57:59gave it hard problems and then we saw
  1420. 58:01that it was basically by having
  1421. 58:03reasoning chains it was doing better. So
  1422. 58:05the GSM8K was the first paper that kind
  1423. 58:07of showed signs of life of this and then
  1424. 58:10uh with larger model uh like palm we
  1425. 58:12actually saw that this was a very big
  1426. 58:14deal and one of the examples there was
  1427. 58:16that could explain jokes which was very
  1428. 58:18impressive and then from there the
  1429. 58:20reasoning models have emerged so it was
  1430. 58:22not but it has read all of the web so
  1431. 58:25it's it has definitely seen data which
  1432. 58:27is like more um methodical and
  1433. 58:29systematic
  1434. 58:30>> but the reasoning models are trained to
  1435. 58:33be reasoning more and more, right? So,
  1436. 58:36it's they didn't the entire reasoning is
  1437. 58:39not an emergent behavior. The reasoning
  1438. 58:41models, they're trained to be to be
  1439. 58:44thinking, but like the models are
  1440. 58:46convergent. So, the models are going to
  1441. 58:48be trained such that they know when to
  1442. 58:50they need a lot of reasoning and when
  1443. 58:51they don't and and to generate answers.
  1444. 58:55But chain of thought originally was an
  1445. 58:58emerging behavior. they noticed that oh
  1446. 59:00if we explain things a model it's better
  1447. 59:05>> any kind of other emerging behaviors and
  1448. 59:07larger models
  1449. 59:08>> I don't think it's I would say it as
  1450. 59:10immersion behaviors per se but I think
  1451. 59:12as I was mentioning we usually go
  1452. 59:14looking for certain things right so in
  1453. 59:16the agentic workflows what we're looking
  1454. 59:17for is planning which is a form of
  1455. 59:20reasoning we're looking for multi-step
  1456. 59:21reasoning which we will cover in class
  1457. 59:23as well and then we're looking for
  1458. 59:25self-improvement or selfcorrection so
  1459. 59:27all of these capabilities would be nice
  1460. 59:29to have and like what gets the models
  1461. 59:31there is a set of questions that are
  1462. 59:33worth like there are papers on this kind
  1463. 59:34of things
  1464. 59:36>> like selfcorrection
  1465. 59:37backtracking things like that is like
  1466. 59:40yes you could call it emergent but you
  1467. 59:43could also say think about it that they
  1468. 59:46have seen this kind of behavior and it's
  1469. 59:48kind of like it's reinforced in the way
  1470. 59:50that they're fine-tuned so it's
  1471. 59:54yeah it's hard to
  1472. 59:58Okay. Um, so let's go through the the
  1473. 1:00:01the logistics for the class. Um, here's
  1474. 1:00:05a list of all the amazing topics that
  1475. 1:00:07you're going to learn about. Um, they
  1476. 1:00:09are um,
  1477. 1:00:12yeah, this is very cool.
  1478. 1:00:15>> I think the main thing to remember here
  1479. 1:00:16is that the overall theme stays the
  1480. 1:00:18same. uh and then we'll also have guest
  1481. 1:00:20lectures around from u folks in frontier
  1482. 1:00:23AI labs and covering things all the way
  1483. 1:00:26from say how have uh has how has post
  1484. 1:00:29training evolved or say even multimodal
  1485. 1:00:31agents and uh in robotics. So it's it's
  1486. 1:00:34a it's going to be a mix of lectures and
  1487. 1:00:37guest lectures and um and of course your
  1488. 1:00:40project presentations.
  1489. 1:00:42[snorts]
  1490. 1:00:43So here is some logistics that you all
  1491. 1:00:46know the the prerec are there just make
  1492. 1:00:48sure that you you are comfortable with
  1493. 1:00:51these prerexs as you uh register them in
  1494. 1:00:54the class. So we have the external
  1495. 1:00:57website but uh make sure that you're uh
  1496. 1:01:02you check canvas that's where we send
  1497. 1:01:04out the latest updates at all times. Um
  1498. 1:01:07and um we will also upload the we try to
  1499. 1:01:11upload the lectures before we start
  1500. 1:01:13every every in every class so you have
  1501. 1:01:16access to the lectures. Uh and we will
  1502. 1:01:18also have we already have the due dates
  1503. 1:01:20for all the assignments and all the
  1504. 1:01:21projects. So you can check it out. Um
  1505. 1:01:25so this quarter we are going to have
  1506. 1:01:27three homeworks. That's a difference
  1507. 1:01:29from between this class and that's one
  1508. 1:01:31of the differences. So we have one more
  1509. 1:01:34amazing homework for you all that we
  1510. 1:01:36have uh designed and the TAs have done a
  1511. 1:01:38great job and that's helps you um
  1512. 1:01:41hopefully learn uh learn these topics
  1513. 1:01:44better and more in depth. Um and then we
  1514. 1:01:48also have a course project. So the
  1515. 1:01:50course project is where you can kind of
  1516. 1:01:51unleash your creativity and like your
  1517. 1:01:55your way of like building agentic
  1518. 1:01:57systems or going deeper into a question
  1519. 1:02:01uh and and designing experiments around
  1520. 1:02:04it and see what works and what doesn't.
  1521. 1:02:06Um we will provide some um kind of
  1522. 1:02:10examples or suggestions but it could be
  1523. 1:02:13completely on you how to design this
  1524. 1:02:15project. We'll have some examples from
  1525. 1:02:17last year like the successful projects
  1526. 1:02:19from last year as well. Uh it's going to
  1527. 1:02:22be uploaded to
  1528. 1:02:24um canvas or
  1529. 1:02:26>> actually we'll display them on the
  1530. 1:02:27website.
  1531. 1:02:28>> Oh, on the website. Oh, cool. The public
  1532. 1:02:30website. So you can also see that. Um so
  1533. 1:02:34>> so for the course project I think the
  1534. 1:02:36main thing to remember is that uh we
  1535. 1:02:38will have API credits and you can work
  1536. 1:02:41in teams of two to four people,
  1537. 1:02:43>> right?
  1538. 1:02:44>> Yeah. or up to four. I guess
  1539. 1:02:48>> it can be one if you really want that.
  1540. 1:02:50We we we suggest you team up with
  1541. 1:02:53others. Um at the very least you have
  1542. 1:02:56more credit collectively so you can run
  1543. 1:02:58more experiments but then you might also
  1544. 1:03:01uh find a friend collaborator along the
  1545. 1:03:03way and you can um um do something
  1546. 1:03:06bigger. Um and we are going to have this
  1547. 1:03:11um what is this? The course project.
  1548. 1:03:14Yeah, we we're going to we're going to
  1549. 1:03:15let you know about some uh some of the
  1550. 1:03:18previous years and some ideas that you
  1551. 1:03:20can take. Um here are some course
  1552. 1:03:22project examples that are acceptable.
  1553. 1:03:25For example, a new evaluation data set
  1554. 1:03:28or a new benchmark. Um you can design a
  1555. 1:03:32project around the reliability of a of
  1556. 1:03:34an agent agentic system that already
  1557. 1:03:36exists. You can take a benchmark that
  1558. 1:03:39exists and try to hill climb on it with
  1559. 1:03:41whatever idea great idea that you have.
  1560. 1:03:45Um and then or you can just we we are
  1561. 1:03:47going to have a whole lot of papers
  1562. 1:03:49covered in this course and we have we
  1563. 1:03:51are showing all of that to you. They're
  1564. 1:03:53all on the website. So as you as we give
  1565. 1:03:56this lectures, we kind of expect you to
  1566. 1:03:58also have read those papers or read the
  1567. 1:04:00papers along the way and develop better
  1568. 1:04:03and deeper intuitions on how these
  1569. 1:04:05methods work. and your project could
  1570. 1:04:08always be improving those or like
  1571. 1:04:10questioning one of the decisions that
  1572. 1:04:12they have made and trying to uh change
  1573. 1:04:14that or or evaluate that. Um negative
  1574. 1:04:19examples are like a survey paper or like
  1575. 1:04:22we just don't we want something
  1576. 1:04:24researchy here. So we just don't want
  1577. 1:04:26like just an app that you put together
  1578. 1:04:28and you just show us something. It has
  1579. 1:04:30to be like here is like here is the
  1580. 1:04:33hypothesis here is the question that we
  1581. 1:04:36wanted to answer here is the type of
  1582. 1:04:37like improvement we wanted to see or the
  1583. 1:04:40type of like uh properties we wanted to
  1584. 1:04:42analyze and then we have built that
  1585. 1:04:44right we want something something that
  1586. 1:04:46um uh something more than wipe coding
  1587. 1:04:49basically in this um
  1588. 1:04:52>> in terms of milestones I think the it's
  1589. 1:04:54worthwhile to remember that you want to
  1590. 1:04:56start early so it's you need to have a
  1591. 1:04:59project proposal um somewhere around I
  1592. 1:05:02think we we put it early October.
  1593. 1:05:04>> Yeah.
  1594. 1:05:04>> And then uh
  1595. 1:05:05>> it's on the website. Yeah,
  1596. 1:05:06>> it's on the website. And then for the
  1597. 1:05:08midterm project presentation, we do want
  1598. 1:05:10you to have some progress. So two weeks
  1599. 1:05:12after the project proposal, we do expect
  1600. 1:05:14you to have like so start thinking about
  1601. 1:05:16like what experiments you want to run by
  1602. 1:05:18the time you get to the project proposal
  1603. 1:05:20so that you actually go run them for the
  1604. 1:05:22midterm projects. And so we do expect
  1605. 1:05:25you to have made some progress in the
  1606. 1:05:26midterm project presentation. It should
  1607. 1:05:28not just be here is a proposal and then
  1608. 1:05:31the final report has a lot of weight and
  1609. 1:05:33then the final poster presentation uh
  1610. 1:05:35will be the end of the quarter.
  1611. 1:05:40Um yes.
  1612. 1:06:00>> I I think so if you add to it, right?
  1613. 1:06:06>> Okay. Okay. when I was independent
  1614. 1:06:08study, right?
  1615. 1:06:09>> Yeah.
  1616. 1:06:10>> Yeah. Yeah, that's fine.
  1617. 1:06:11>> I I think
  1618. 1:06:12>> I mean as long as you're doing actual
  1619. 1:06:14work and it's not just like reusing the
  1620. 1:06:16work from
  1621. 1:06:19>> Yeah.
  1622. 1:06:20>> Um we had uh we had students publishing
  1623. 1:06:24papers out of their projects uh in the
  1624. 1:06:27last quarter. So uh in the last last
  1625. 1:06:30time we taught the class. So that's
  1626. 1:06:31something you could it could be one of
  1627. 1:06:33you or many of you this time again uh
  1628. 1:06:36turning your research project working
  1629. 1:06:38more on it and then uh turning it into a
  1630. 1:06:41publication at at um uh conferences.
  1631. 1:06:46Um and so here is a save the date for
  1632. 1:06:51for our poster presentation. It's going
  1633. 1:06:53to be December 12, 4 to 6 p.m. So um we
  1634. 1:06:57we would want you to be there and
  1635. 1:06:59present the posters. you'll have people
  1636. 1:07:01from industry joining and you can that's
  1637. 1:07:04where you can brag about what you've
  1638. 1:07:06done and just meet [clears throat] new
  1639. 1:07:07people. Um here is the course uh grading
  1640. 1:07:11rubric. So uh we have three homeworks
  1641. 1:07:14that's 50% of your uh of of your entire
  1642. 1:07:18grade and then another 50% is on the
  1643. 1:07:20project and the duration of these are
  1644. 1:07:22kind of like align with how much uh how
  1645. 1:07:25much the grading um grading is allocated
  1646. 1:07:28to each of these. Um and we of course
  1647. 1:07:32expect you to uh honor
  1648. 1:07:35the honor code. Um and let's see
  1649. 1:07:40office hours will be posted on Canvas.
  1650. 1:07:43Um you can ask your questions. We really
  1651. 1:07:45really encourage you to ask questions on
  1652. 1:07:47Edson uh and make it public. So others
  1653. 1:07:50probably if you have a question um it's
  1654. 1:07:53very likely others have the same
  1655. 1:07:55question. So please help us like do
  1656. 1:07:57broadcast this to everybody. Um um at
  1657. 1:08:00SSEM is um again you can ask questions
  1658. 1:08:04and then grade scope is where you submit
  1659. 1:08:06your project milestones and everything
  1660. 1:08:09else. Yeah, the homeworks and
  1661. 1:08:11everything. Um we have the following
  1662. 1:08:13late policy that we think is kind of
  1663. 1:08:15like
  1664. 1:08:16within the generous kind of like um side
  1665. 1:08:20on the gener generous side of things and
  1666. 1:08:23because the class is big this quarter we
  1667. 1:08:27really can't make any exceptions. So try
  1668. 1:08:29to use these um kind of late days uh
  1669. 1:08:32wisely. Uh audits are not allowed but we
  1670. 1:08:36will have the the course
  1671. 1:08:39the we will have the videos
  1672. 1:08:42>> videos on YouTube eventually.
  1673. 1:08:44>> Eventually
  1674. 1:08:46and that's it. Any questions?
  1675. 1:08:52>> No. All right.
  1676. 1:08:54>> Okay. Thanks everyone.

About this transcript

This page contains the full transcript of CLAUDE AI ile sıfırdan Ajan Kurma by Kickooo, generated from the public captions YouTube serves with the video. The transcript has 11,786 words across 1,676 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.