YouTube2Text

Claude AI Full Course: Build & Automate Anything in 2 Hour I 99% Of People Are Missing Claude AI — Transcript

by frugle 1.7M · 17,587 words · 2,574 segments · language en · Watch on YouTube

Full transcript

  1. 0:00And now we're going one level beyond
  2. 0:02into what would it look like if you were
  3. 0:06building uh agentic AI systems at work
  4. 0:10in a startup in a company. Um and it's
  5. 0:15probably one of the more practical
  6. 0:17lectures. Again, the goal is not to
  7. 0:19build a product end to end in the next
  8. 0:22hour or so, but rather to tell you all
  9. 0:24the techniques that AI engineers have
  10. 0:27cracked, figured out, are exploring so
  11. 0:31that after the class, you have sort of
  12. 0:32the breath of view of different
  13. 0:34prompting techniques, different agentic
  14. 0:36workflows, multi- aent systems, eval
  15. 0:41when you want to dive deeper, you have
  16. 0:42the baggage to dive deeper and learn
  17. 0:45faster about it. Okay.
  18. 0:48Uh let's try to make it as interactive
  19. 0:50as possible as usual. Um when we look at
  20. 0:54the agenda, the agenda is going to start
  21. 0:58with the core idea behind challenges and
  22. 1:00opportunities for augmenting LLMs. So we
  23. 1:04start from a base model. How do we
  24. 1:06maximize the performance of that base
  25. 1:09model? Um then we'll dive deep into the
  26. 1:12first line of optimization which is
  27. 1:15prompting methods. and we'll see a
  28. 1:16variety of them. Then we go slightly
  29. 1:19deeper. If we were to get our hands
  30. 1:21under the hood and do some fine tuning,
  31. 1:23what would it look like? I'm not a fan
  32. 1:25of fine tuning, and I talk a lot about
  33. 1:27that, but I'll explain why. I I try to
  34. 1:30avoid fine-tuning as much as possible.
  35. 1:33Um, and then we'll do a section four on
  36. 1:36retrieval augmented generation or rag,
  37. 1:39which you've probably heard of in the
  38. 1:41news. Maybe some of you have played with
  39. 1:44rags. We're going to sort of unpack what
  40. 1:46a rag is and how it works and then the
  41. 1:48different methods within
  42. 1:51rags and then we'll talk about agentic
  43. 1:54AI workflows. Um I'll define it. Um
  44. 1:58Andrew Ang is one of the call it first
  45. 2:00ones who have called this trend a
  46. 2:03agentic AI workflows. And so we look at
  47. 2:06the definition that Enzo gives to
  48. 2:08agentic workflows and then we start
  49. 2:10seeing examples. The section six is very
  50. 2:14practical. It's a case study where we
  51. 2:17will think about an agentic workflow and
  52. 2:20we'll and I ask you to measure um if the
  53. 2:24agent actually works and we brainstorm
  54. 2:27how we can measure if an agentic
  55. 2:30workflow is working the way you want it
  56. 2:31to work. There's plenty of methods
  57. 2:33called eval
  58. 2:35um solve that problem. uh and then we'll
  59. 2:38look briefly at multi- aent of workflow
  60. 2:40and then we can have a a sort of
  61. 2:42open-ended discussion where I'll share
  62. 2:44some thoughts on what's next in AI. Um
  63. 2:47and I'm looking forward to hearing from
  64. 2:49you all as well on that one. Okay, so
  65. 2:52let's get started uh with the problem of
  66. 2:56augmenting LLM. So open-ended question
  67. 2:58for you. You are all familiar with
  68. 3:01pre-trained models like GPT3.5 Turbo or
  69. 3:05GPT40.
  70. 3:07What's the limitation of using just a
  71. 3:10base model? What are the typical issues
  72. 3:13that might arise as you're using a
  73. 3:16vanilla pre-trained model?
  74. 3:23>> Yes.
  75. 3:26>> Lacks some domain knowledge. You're
  76. 3:27perfectly right. you know, you we we had
  77. 3:30a group of students a few years ago was
  78. 3:32not LLM related, but you know, they were
  79. 3:34building a an autonomous uh farming uh
  80. 3:39device or vehicle that had a camera
  81. 3:41underneath taking pictures of crops to
  82. 3:44determine if the crop is sick or not, if
  83. 3:47it should be thrown away, like if it
  84. 3:49should be if it should be used or not.
  85. 3:51And and that data set is not a data set
  86. 3:55you find out there. And the base model
  87. 3:58or a pre-trained um computer vision
  88. 4:00model would lack that knowledge of
  89. 4:02course. What else?
  90. 4:05Yes.
  91. 4:12>> Okay. Maybe the you're saying so just to
  92. 4:15re repeat for people online you're
  93. 4:18saying the model might have been trained
  94. 4:19on high quality data but the data in the
  95. 4:22wild is actually not that high quality.
  96. 4:24And in fact, yes, the distribution of
  97. 4:26the real world might defer uh as we've
  98. 4:29seen with GANs from the training set and
  99. 4:32that might create an issue with
  100. 4:33pre-trained models. Although pre-trained
  101. 4:35LLMs are getting better at, you know,
  102. 4:37handling all sorts of data inputs. Uh
  103. 4:41yes,
  104. 4:44>> like what
  105. 4:45>> lacks current information. Uh the LLM is
  106. 4:48not up to date. And in fact, you're
  107. 4:50right. Imagine you have to retrain from
  108. 4:52scratch your LLM every couple of months.
  109. 4:55One story that I found funny um it's
  110. 4:58from probably three years ago or maybe
  111. 5:00more five years ago where um during his
  112. 5:03first presidency, President Trump one
  113. 5:06day tweeted Kov. You remember that tweet
  114. 5:10or no? Just Kof. And it was probably a
  115. 5:13typo or it was in his pocket. I don't
  116. 5:16know. But that word did not exist. the
  117. 5:19LLMs in fact that Twitter was running at
  118. 5:21the time could not recognize that word
  119. 5:24and so the recommener system sort of
  120. 5:26went wild and because suddenly everybody
  121. 5:29was making fun of that tweet using the
  122. 5:31word coe and the LM was so confused on
  123. 5:34you know what does that mean where
  124. 5:35should we show it to whom should we show
  125. 5:37it and is an example of nowadays
  126. 5:40especially on social media there's so
  127. 5:43many new trends and um it's very hard to
  128. 5:46retrain an LLM to match the new trend
  129. 5:48and understand the new words out there.
  130. 5:50I mean, you know, you often times hear
  131. 5:52Gen Z words like re or mid or whatever.
  132. 5:55I don't know all of them, but you
  133. 5:59probably want to find a way that can
  134. 6:01allow the LLM to understand those trends
  135. 6:03without retraining the LLM from scratch.
  136. 6:06Yeah. What else?
  137. 6:09It's trained to have a breath of
  138. 6:11knowledge and if you wanted to do that
  139. 6:14might limit
  140. 6:15>> yeah it might be trained on a breath of
  141. 6:17knowledge but it might fail or not
  142. 6:20perform adequately on a narrow task that
  143. 6:22is very well defined. Think about
  144. 6:25enterprise applications that yeah
  145. 6:27enterprise application you need high
  146. 6:29precision high fidelity low latency and
  147. 6:33maybe the model is not great at that
  148. 6:35specific thing. It might do fine but
  149. 6:36just not good enough and you might want
  150. 6:38to augment it in a certain way. Yeah.
  151. 6:44>> So it makes the model a lot heavier, a
  152. 6:46lot slower.
  153. 6:49>> So maybe it has a lot of broad domain
  154. 6:52knowledge that might not be needed for
  155. 6:53your application and so you're using a
  156. 6:56massive heavy model when you actually
  157. 6:57are only using 2% of the model
  158. 6:59capability. You're perfectly right. You
  159. 7:01might not need all of it. So you might
  160. 7:03find ways to prune, quantize the model,
  161. 7:05modify it. All of these are are are good
  162. 7:08points. I'm going to add a few more as
  163. 7:10well. Um, LLMs are very difficult to
  164. 7:13control. Uh, your last point is actually
  165. 7:15an example of that. You want to control
  166. 7:17the LM to use a part of its knowledge,
  167. 7:18but it's not. It's in fact getting
  168. 7:21confused. Uh, we've seen that in
  169. 7:23history. In 2016, Microsoft created a
  170. 7:27notorious Twitter bot that learned from
  171. 7:30users and it quickly became a racist
  172. 7:33jerk. Microsoft ended up removing the
  173. 7:36bot 16 hours after launching it. The
  174. 7:38community was really fast at determining
  175. 7:41that this was a racist bot. Um, and and
  176. 7:44you know, you can empathize with
  177. 7:46Microsoft in the sense that it is
  178. 7:47actually hard to control an LLM. they
  179. 7:49might have done a better job to qualify
  180. 7:52before launching, but it is really hard
  181. 7:54to control an LLM. Even more recently,
  182. 7:57this is a tweet from Sam Alman last
  183. 8:00November, um, where there was this
  184. 8:03debate between Elon Musk and Sam Alman
  185. 8:06on whose LLM is the left-wing propaganda
  186. 8:10machine or the right-wing propaganda
  187. 8:12machine. And they were hating on each
  188. 8:13other's LLMs. But that tells you at the
  189. 8:16end of the day that even those two teams
  190. 8:18Grock and OpenAI which are probably the
  191. 8:21best funded team with a lot of talent
  192. 8:24are not doing a great job at controlling
  193. 8:26their LLMs, you know.
  194. 8:29And from time to time, if you hang out
  195. 8:31on X, you might see screenshots of users
  196. 8:35interacting with LLM's and the LLM
  197. 8:37saying something really controversial or
  198. 8:40or or racist or, you know, something
  199. 8:43that would not not be considered great
  200. 8:46by social standards, I guess. And uh and
  201. 8:49that tells you that the model is really
  202. 8:51hard to to control.
  203. 8:54Um the second aspect of it is something
  204. 8:57that you've mentioned earlier. LLMs may
  205. 9:00underperform in your task. Um and that
  206. 9:03might include specific knowledge gaps
  207. 9:05such as medical diagnosis. If you're
  208. 9:07doing medical diagnosis, you would
  209. 9:08rather have a LLM that is specialized
  210. 9:10for that and is great at it. And in
  211. 9:12fact, something that we haven't
  212. 9:14mentioned as a group has sources. So the
  213. 9:16answer is sourced specifically. you have
  214. 9:19a hard time believing something unless
  215. 9:21you have the actual source of the
  216. 9:22research that backs it up. Um,
  217. 9:25inconsistencies in style and format. So,
  218. 9:28imagine you're building a legal AI
  219. 9:30agentic workflow. Uh, legal has a very
  220. 9:34specific way to write and read uh where
  221. 9:37every word counts. You know, if you're
  222. 9:38negotiating a large contract, every word
  223. 9:41on that contract might mean something
  224. 9:43else when it comes to the court. And so
  225. 9:45it's very important that you use an LLM
  226. 9:48that is very good at it. The precision
  227. 9:50matters. And then you know task specific
  228. 9:53understanding such as doing a
  229. 9:54classification on a niche field. Here I
  230. 9:56pulled an example where you know let's
  231. 9:59say a biotech product is trying to use
  232. 10:01an LLM to categorize user reviews into
  233. 10:06positive, neutral or negative. um you
  234. 10:09know maybe for that company something
  235. 10:12that's uh would be considered a negative
  236. 10:16review typically is actually considered
  237. 10:18a neutral review because the NPS of that
  238. 10:21industry tends to be way lower than
  239. 10:23other industries let's say um that's a
  240. 10:26task specific understanding and the LLM
  241. 10:28needs to be aligned to what the company
  242. 10:30believes is the categorization that it
  243. 10:32wants we will see an example of how to
  244. 10:35solve that problem in a second and then
  245. 10:37limited context handling. Um, a lot of
  246. 10:40AI applications especially in the
  247. 10:43enterprise have uh require data that has
  248. 10:46a lot of context. Like just to give you
  249. 10:49a simple example, knowledge management
  250. 10:51is an important space that enterprises
  251. 10:54buy a lot of knowledge management tool.
  252. 10:56When you go on your drive and you have
  253. 10:58all your documents, ideally you could
  254. 11:00have an LLM running on top of that
  255. 11:02drive. You can ask any question and it
  256. 11:04will read immediately thousands of
  257. 11:06document and answer what was our Q4
  258. 11:09performance in sales. Um it was X
  259. 11:12dollars. Uh it finds it super quickly.
  260. 11:15In practice because LLMs do not have a
  261. 11:18large enough context, you cannot use a
  262. 11:21standalone vanilla pre-trained LLM to
  263. 11:23solve that problem. You will have to
  264. 11:24augment it.
  265. 11:27Does that make sense? Uh the other
  266. 11:29aspect around context windows is they
  267. 11:31are in fact limited. If you look at the
  268. 11:33context windows of the models from the
  269. 11:36last 5 years um even the best models
  270. 11:41today will range in context window or
  271. 11:45number of tokens it can take as input um
  272. 11:48somewhere in the hundreds of thousands
  273. 11:50of tokens max. Just to give you a sense
  274. 11:53200,000 tokens is roughly two books.
  275. 11:57Yeah. So that's how much you can upload
  276. 12:00uh and it can read pretty much and you
  277. 12:02can imagine that when you're dealing
  278. 12:04with video understanding or heavier data
  279. 12:08files that is of course an issue.
  280. 12:12So you might have to chunk it, you might
  281. 12:14have to embed it, you might have to find
  282. 12:15other ways to get uh the LLM to handle
  283. 12:18larger contexts.
  284. 12:21Um the attention mechanism is also
  285. 12:24powerful but a problematic because it
  286. 12:26does not do a great job at attending in
  287. 12:29very large contexts. There is actually
  288. 12:32an interesting uh problem called needle
  289. 12:35in a haststack. It's an AI problem where
  290. 12:39or call it a benchmark where um in order
  291. 12:42to test if your LLM is good at attending
  292. 12:45at putting attention on a very specific
  293. 12:48fact within a large corpus, researchers
  294. 12:52might randomly insert in a book one
  295. 12:55sentence that
  296. 12:58that outlines a certain fact such as
  297. 13:02Arun and Max are having coffee at Blue
  298. 13:04Bottle in the middle of the Bible, let's
  299. 13:06or some very long texts. Um, and then
  300. 13:10you ask the LLM, um, what were Aruin and
  301. 13:14Max having, you know, at Blue Bottle?
  302. 13:18And you see if it remembers that it was
  303. 13:20coffee. It's actually a complex problem,
  304. 13:21not because the question is complex, but
  305. 13:23because you're asking the model to find
  306. 13:25a fact within a very large corpus, and
  307. 13:28that's complicated.
  308. 13:31So again, this is a a limiting factor
  309. 13:34for LLMs. We we'll talk about rag in a
  310. 13:37second but I want to preview you know
  311. 13:38the the there is debates around whether
  312. 13:41rag is the right long-term approach for
  313. 13:44AI system. So as a as a high level idea
  314. 13:47a rag is a mechanism if you will that
  315. 13:50embeds documents that an LLM can
  316. 13:53retrieve and then add as context to its
  317. 13:57initial prompt and answer a question. It
  318. 14:00has lots of application knowledge
  319. 14:01management is an example. So imagine you
  320. 14:03have your drive again but every document
  321. 14:05is sort of compressed in representation
  322. 14:09and the LLM has access to that lower
  323. 14:11dimensional representation. Um the
  324. 14:14debate that this tweet from Yu um
  325. 14:18outlines is in theory if we have
  326. 14:22infinite compute then rag is useless
  327. 14:25because you can just read a massive
  328. 14:28corpus immediately and answer your
  329. 14:30question. Uh but even in that case um
  330. 14:33latency might be an issue. Imagine the
  331. 14:35time it takes for an AI to read all your
  332. 14:37drive every single time you ask a
  333. 14:39question. It doesn't make sense. So rag
  334. 14:42has other advantages beyond even the
  335. 14:45accuracy. On top of that the sourcing
  336. 14:48matters as well. So it might rag allows
  337. 14:50you to source. We we'll talk we'll talk
  338. 14:52about all that later. But there are
  339. 14:55there's always this debate in the in the
  340. 14:57community whether a certain method is
  341. 15:00actually future proof because in
  342. 15:02practice as compute power doubles every
  343. 15:05year let's say some of the methods we're
  344. 15:07learning right now might not be relevant
  345. 15:083 years from now we don't know
  346. 15:11essentially
  347. 15:14um you know and and the analogy that
  348. 15:16that he makes on on you know context
  349. 15:19windows and why rag approaches might be
  350. 15:21relevant even a long time from now um is
  351. 15:24search. You know when you search on a
  352. 15:26search engine, you still find sources of
  353. 15:30information and in fact in the
  354. 15:31background there is very
  355. 15:34you know details traversal algorithms
  356. 15:36that rank and find uh the specific links
  357. 15:39that might be the the best to present
  358. 15:41you. um versus if you had to read
  359. 15:44imagine you had to read the entire web
  360. 15:46every single time you're doing a search
  361. 15:47query uh without being able to narrow to
  362. 15:50certain portion of the space that might
  363. 15:53again not be uh reasonable.
  364. 15:57Okay. When we're thinking of um
  365. 16:00improving LLMs, the easiest way we think
  366. 16:03of it is is two dimensions. One
  367. 16:06dimension is we are going to improve the
  368. 16:08foundation model itself. So for example,
  369. 16:11we move from GPT 3.5 turbo to GPT4 to
  370. 16:17GPT40 to GPT5. Each of that is supposed
  371. 16:21to improve the base model. GPT5 is
  372. 16:23another debate because it's sort of
  373. 16:26packaging other models within itself.
  374. 16:28But you know if you're thinking about
  375. 16:303.5 4 and 4, that's really what it is.
  376. 16:32The pre-trained model improves and so
  377. 16:34you should see your uh performance
  378. 16:36improve on your tasks. Um but the other
  379. 16:39dimension is we can actually engineer
  380. 16:42leverage the LLM in a way that makes it
  381. 16:45better. So you can prompt simply GPT40.
  382. 16:49You can
  383. 16:51chain some prompts and improve the
  384. 16:53prompt and it will improve the
  385. 16:55performance. It's shown. Uh you can even
  386. 16:57put a rag around it. You can put an
  387. 16:59agentic workflow around it. You can even
  388. 17:01put a multi- aent system around it. And
  389. 17:05that is another dimension for you to
  390. 17:07improve performance. So that's how I
  391. 17:09want you to think about it. Which LLM
  392. 17:10I'm using and then how can I maximize
  393. 17:12the performance of that LLM.
  394. 17:15This lecture is about the vertical axis.
  395. 17:18Those are the methods that we will see
  396. 17:19with it together.
  397. 17:24Sounds good for the introduction. So
  398. 17:27let's move to prompt engineering. Um I'm
  399. 17:30going to start with an interesting study
  400. 17:32just to motivate why prompt engineering
  401. 17:34matters. Um there is a a study from uh
  402. 17:38you know HPS, UPUPEN um uh as well as
  403. 17:42Harvard Business School and and and
  404. 17:45others also involve Wharton that took a
  405. 17:47subset of BCG consultants, individual
  406. 17:50contributors, split them into three
  407. 17:52groups. One group had no access to AI.
  408. 17:55One group had access to I think it was
  409. 17:57GPT 4. Um and then one group had access
  410. 18:01to the LLM but also a training on how to
  411. 18:04prompt better. Um and then they observed
  412. 18:07the performance of these consultants
  413. 18:09across a wide variety of tasks. There's
  414. 18:12a few things that they noticed that I
  415. 18:13thought was interesting. One is
  416. 18:15something they call the JAG frontier. um
  417. 18:18meaning that certain tasks that
  418. 18:20consultants are doing fall beyond the
  419. 18:24Jack frontier meaning AI is uh is not
  420. 18:29good enough. It's not it's not improving
  421. 18:32human performance. In fact, it's
  422. 18:34actually making it worse. Um and some
  423. 18:36tasks are within the frontier, meaning
  424. 18:39that uh AI is actually significantly
  425. 18:42improving the performance, the speed,
  426. 18:44the quality of the consultant. Um many
  427. 18:48tasks fell within and many tasks fell
  428. 18:50without and they shared their um
  429. 18:52insights but the TLDDR is there is a
  430. 18:55frontier within which AI is absolutely
  431. 18:57helping and one where they call out this
  432. 19:01behavior of falling asleep at the wheel
  433. 19:04where people relied on AI on a task that
  434. 19:07was beyond the frontier and in fact it
  435. 19:09ended up going worse because the human
  436. 19:12was not reviewing the outputs carefully
  437. 19:14enough. Um
  438. 19:17they did note that the the group that
  439. 19:19was trained was uh the best uh better
  440. 19:22than the group that was not trained on
  441. 19:24prompt engineering which also motivates
  442. 19:26why this lecture matters. Um so that
  443. 19:28you're you're you're within that group
  444. 19:30afterwards. One other insights um were
  445. 19:33the century.
  446. 19:36They noticed that consultants had the
  447. 19:37tendency to work with AI in one of two
  448. 19:39ways and you might yourself find be part
  449. 19:43of one of these groups. The centaurs are
  450. 19:46mythical creatures that are half human,
  451. 19:49half
  452. 19:50I think half
  453. 19:53what? Horses. Yeah, horses. Half horses,
  454. 19:56half something. Um, and those were
  455. 19:59individuals that would divide and
  456. 20:00delegate. They might give a pretty big
  457. 20:02task to the AI. So imagine you're
  458. 20:04working on a PowerPoint, which
  459. 20:06consultants are known to do. um you
  460. 20:08might actually write a very long prompt
  461. 20:10on how you wanted to do your PowerPoint
  462. 20:12and then let it work for some time and
  463. 20:14then come back and it's done when others
  464. 20:16would act as cyborgs. Cyborgs are fully
  465. 20:18blended bionic human robots, a human
  466. 20:22robot and robot augmented with robotic
  467. 20:25parts. Uh and those individuals would
  468. 20:27not delegate fully a task. They would
  469. 20:29actually work super quickly with the
  470. 20:31model and like back and forth. I find
  471. 20:33that a lot of students are actually more
  472. 20:35working like cyborgs than centaurs. But
  473. 20:38while maybe in the enterprise when
  474. 20:40you're trying to automate a workflow,
  475. 20:41you're thinking more like a centaur.
  476. 20:44Yeah, that's just something good to keep
  477. 20:46in mind. Also, a lot of companies would
  478. 20:48tell you, oh, we're hiring prompt
  479. 20:49engineers, etc., it's a curer. I I don't
  480. 20:52buy that. I think it's just a skill that
  481. 20:54everybody should have. uh you're not
  482. 20:55going to make a career out of prompt
  483. 20:56engineering, but you're probably going
  484. 20:58to use it as a very powerful skill in
  485. 21:01your career.
  486. 21:04Um so let's talk about basic prompt
  487. 21:07design principles. Uh I'm giving you a
  488. 21:09very simple prompt here. Summarize this
  489. 21:12document and then the document is
  490. 21:14uploaded alongside it. And the model has
  491. 21:16not um much context around you know what
  492. 21:20should be the summary, well how long
  493. 21:22should be read the summary, what should
  494. 21:23it talk about. etc. You can actually
  495. 21:26improve this prompt um by by doing you
  496. 21:29know something like summarize this
  497. 21:3010page scientific paper on renewable
  498. 21:34energy in five bullet points focusing on
  499. 21:37key findings and implications for policy
  500. 21:39makers. That's already better, right?
  501. 21:41You're sharing the audience and it's
  502. 21:44going to tailor it to the audience.
  503. 21:45You're saying that you want five bullet
  504. 21:48points and you want focus only on key
  505. 21:50findings. You know, that's a better
  506. 21:52prompt, you would argue. Um, how could
  507. 21:55you even make this prompt better? What
  508. 21:57are other techniques that you've heard
  509. 21:59of or tried yourself that could make
  510. 22:01this oneshot prompt better?
  511. 22:08>> Yeah.
  512. 22:13>> Okay. Right. Right. Example. So, say uh
  513. 22:15you mean here is an example of a great
  514. 22:17summary. Yeah, you're right. That's a
  515. 22:19good idea.
  516. 22:21to be like someone act like you are
  517. 22:24>> very popular technique act like
  518. 22:27renewable energy expert giving a
  519. 22:30conference at Davos let's say yeah
  520. 22:32that's great someone
  521. 22:39>> you're the best in the world at this
  522. 22:41explain yeah actually I mean these
  523. 22:43things work it's funny but uh it does
  524. 22:45work to to say act like xyz it's a very
  525. 22:48popular prompt template We we'll see a
  526. 22:50few examples. What else could you do?
  527. 22:56>> Yes.
  528. 22:57>> I personally like to say critique your
  529. 22:59own.
  530. 23:00>> Okay.
  531. 23:01>> Critique your own project. So you're
  532. 23:03using reflection. So you might actually
  533. 23:05do one output and then ask it to
  534. 23:06critique it and then give it back. Yeah,
  535. 23:09we'll see that. That's a great one.
  536. 23:10That's that's the one that probably
  537. 23:11works best within those typically, but
  538. 23:13we we'll see some examples. What else?
  539. 23:16Yeah,
  540. 23:17>> tasks. Okay. Break the task down into
  541. 23:20steps. Do you know how that is called?
  542. 23:23>> No.
  543. 23:23>> Okay. Chain of thoughts. So, um this is
  544. 23:27actually a popular method that's been
  545. 23:28shown in in research that it improves.
  546. 23:31You could actually give a clear
  547. 23:32instruction and also encourage the model
  548. 23:34to think step by step. Approach the task
  549. 23:37step by step and do not skip any step.
  550. 23:39And then you give it some steps such as
  551. 23:41step one, identify the three most
  552. 23:44important findings. Step two, explain
  553. 23:45how key each finding impact renewable
  554. 23:48energy policy. Step three, write the
  555. 23:50five bullet summary with each point
  556. 23:52addressing a finding um etc. So, chain
  557. 23:56of thoughts, I linked the paper from
  558. 23:582023 uh that popularized chain of
  559. 24:01thoughts. Chain of thoughts is very very
  560. 24:03popular right now especially in AI
  561. 24:05startups that are trying to control
  562. 24:06their elements.
  563. 24:10Okay, to go back to your examples about
  564. 24:14act like XYZ,
  565. 24:16I what I like to do, Andrew also talks
  566. 24:18about that uh is to look at other
  567. 24:20people's prompts and in fact in online
  568. 24:23you have a lot of prompt repositories
  569. 24:25for free on GitHub. In fact, I I linked
  570. 24:28the awesome prompt template repo on
  571. 24:31GitHub where you have so many examples
  572. 24:33of great prompt that engineers have
  573. 24:35built. They said it works great for us
  574. 24:37and they published it online. And a lot
  575. 24:39of them starts with act as you know act
  576. 24:43as a Linux terminal, act as an English
  577. 24:46translator, act like um a position
  578. 24:48interviewer, etc.
  579. 24:52The advantage of a prompt template is
  580. 24:54that you can actually put it in your
  581. 24:56code and scale it for many user
  582. 24:59requests. So let me give you an example
  583. 25:01from worker. um you know where Ker
  584. 25:05evaluates skills. Some of you have taken
  585. 25:07the assessments already and um tries to
  586. 25:10personalize it to the user. Um and in
  587. 25:12fact if you actually read in an HR
  588. 25:14system in an enterprise in the HR system
  589. 25:17you might have Jane is a product manager
  590. 25:20level three and she is uh in the US and
  591. 25:24her preferred language is English. And
  592. 25:26actually that metadata can be inserted
  593. 25:29in a prompt template that we personalize
  594. 25:31for Jane. And similarly for Joe whose
  595. 25:34favorite language is preferred language
  596. 25:36is Spanish, it will it will tailor it to
  597. 25:39Joe and that's called a prompt template.
  598. 25:44>> Foundation models they don't use
  599. 25:46something you have to.
  600. 25:50>> So uh the question is do the foundation
  601. 25:54models use a prompt templates or do you
  602. 25:56have to integrate it yourself? So the
  603. 25:59foundation models probably use a system
  604. 26:01prompt that you don't see like when
  605. 26:03actually you type on chat GPT it is
  606. 26:06possible it's not public that open AI
  607. 26:10behind the scene has like act like a
  608. 26:12very helpful assistant for this user and
  609. 26:15by the way here is your memories about
  610. 26:17your the user that we kept in a database
  611. 26:20uh you can actually check your memories
  612. 26:22and then your prompt goes under and then
  613. 26:24the generation starts. So probably
  614. 26:26they're using something like that, but
  615. 26:28it doesn't mean you can't um add one
  616. 26:30yourself. So in fact, if you if you
  617. 26:32think about a prompt template for for
  618. 26:34the work here example I was showing,
  619. 26:36maybe it starts when you call OpenAI by
  620. 26:39act like a helpful assistant and then
  621. 26:41underneath it's like act like a great uh
  622. 26:44AI mentor that helps people in their
  623. 26:46career and open AAI's Chrome template
  624. 26:49also has follow the instruction from the
  625. 26:51creator or something like that. You
  626. 26:52know, it's possible. Yeah.
  627. 26:56Uh questions about prompt templates.
  628. 26:58Again, I would encourage you to go and
  629. 27:00read examples of prompts. Some of them
  630. 27:01are quite thoughtful.
  631. 27:04Um let's talk about zero shot versus few
  632. 27:06shot prompting. It came up earlier.
  633. 27:09Here's an example. Again, going back to
  634. 27:11the categorization of your product
  635. 27:13reviews. Let's say that we're working on
  636. 27:16a task where the prompt is classify the
  637. 27:19tone of this sentence as positive,
  638. 27:21negative, or neutral. and the the and
  639. 27:24then you paste the review which is the
  640. 27:26product is fine but I was expecting
  641. 27:28more.
  642. 27:31If I were to survey the room I would bet
  643. 27:34that some of you would say it's
  644. 27:36negative, some of you would say it's
  645. 27:38neutral because you actually have a
  646. 27:39first part that is relatively
  647. 27:42positive. It's fine and then the second
  648. 27:44part I was expecting more which is
  649. 27:46relatively negative. So where do you
  650. 27:48land? This can be a subjective question
  651. 27:50and maybe in one industry this would be
  652. 27:52considered amazing and another one it
  653. 27:54would be considered really bad because
  654. 27:56people are used to really flourishing
  655. 27:58reviews. And so the way you can actually
  656. 28:01align the model to your task is by
  657. 28:03converting that zeros short prompt. Zero
  658. 28:05shot refers to the fact that it's not
  659. 28:06beginning given any example uh into a
  660. 28:09few short prompts where the model is
  661. 28:12given in the prompt a set of examples to
  662. 28:15align it to what you want it to do. So
  663. 28:17the example here is you you again you
  664. 28:19paste the same prompt as before with the
  665. 28:21user review and then you add here are
  666. 28:23examples of tone classifications. This
  667. 28:26exceeded my expectation completely.
  668. 28:28Positive. It's okay but I wish it had
  669. 28:32more features. Negative. The service was
  670. 28:35adequate. Neither good nor bad. Uh
  671. 28:38neutral. Now classify the tone of this
  672. 28:41sentence. um you know after you've heard
  673. 28:43about these things and the model then
  674. 28:45says negative and the reason it says
  675. 28:48negative of course is likely um because
  676. 28:51of the second example which was it's
  677. 28:54okay but I wish it had more features
  678. 28:56which we told the model that was
  679. 28:58negative because the model saw that it's
  680. 29:00aligned now with your expectations
  681. 29:03fot prompts are very popular
  682. 29:06um and in fact for AI startups that are
  683. 29:08slightly more sophisticated you might
  684. 29:10see them keep a prompt up to date
  685. 29:13whenever the a user says something and
  686. 29:16they might have a human label it and
  687. 29:18then add it as a few shots in the
  688. 29:20relevant prompt in their codebase. um
  689. 29:23you can think of that as almost building
  690. 29:25a data set, but instead of actually
  691. 29:27building a separate data set like we've
  692. 29:28seen with supervised fine-tuning and
  693. 29:30then train fine-tuning the model on it,
  694. 29:33you're just putting it directly in the
  695. 29:34prompts. And turns out it's probably
  696. 29:36faster to do that if you want to
  697. 29:38experiment quickly because you don't
  698. 29:39touch the model parameters. You just
  699. 29:41update your prompts. And you know, if
  700. 29:44it's text examples, you can actually,
  701. 29:46you know, concatenate so many examples
  702. 29:49in a single prompt. At some point it
  703. 29:51will be too long and you will not have
  704. 29:53the necessary context window but it's a
  705. 29:55pretty strong approach that is quick to
  706. 29:57align an element.
  707. 30:12So question was is there any research on
  708. 30:15how long the prompt can be before the
  709. 30:18model essentially loses itself or
  710. 30:20doesn't follow instructions anymore?
  711. 30:22There is uh the problem is that research
  712. 30:25is outdated every few months uh because
  713. 30:28models get better. Um and so I I don't
  714. 30:31know where the state-ofthe-art is. You
  715. 30:32can probably find it online on
  716. 30:34benchmarks on like we see that I I I
  717. 30:36give an example uh on the worker product
  718. 30:40you have um a voice conversation for
  719. 30:43some of you that have tried it where you
  720. 30:44you ask explain what is a prompt and
  721. 30:46then you explain and then there's a
  722. 30:47scoring algorithm in your we know that
  723. 30:49after eight turns the model loses itself
  724. 30:53after eight turns because you always
  725. 30:56paste the previous user response it just
  726. 30:58starts going wild and so the techniques
  727. 31:01we use in the background is we actually
  728. 31:03create chapters of the conversation.
  729. 31:05Maybe one chapter is the first aid
  730. 31:06prompt and then you actually start over
  731. 31:08from another prompt. You you can
  732. 31:10summarize the first part of the
  733. 31:11conversation, insert the summary and
  734. 31:13then keep going. You know, those are
  735. 31:15engineering hacks that engineers might
  736. 31:17have figured out in the background.
  737. 31:18Yeah. Because yeah, eight turns makes a
  738. 31:21prompt quite long actually.
  739. 31:28Uh let's move on to chaining. uh
  740. 31:31chaining is the most popular technique
  741. 31:32out of everything we've seen so far in
  742. 31:35uh uh in prompt engineering. Uh it's not
  743. 31:38chain of thought. So chain of thought
  744. 31:40we've seen is think step by step, step
  745. 31:42one, step two, step three, do not skip
  746. 31:43any step. This is different. This is
  747. 31:46chaining complex prompt to improve
  748. 31:48performance. And this is what it looks
  749. 31:50like. Um, you take a singlestep prompt
  750. 31:54such as read this customer review and
  751. 31:57write a professional response that
  752. 31:59acknowledges their concern, explains the
  753. 32:01issue, offers a resolution, and then you
  754. 32:03paste the customer review, which is, I
  755. 32:05ordered a laptop, it arrived 3 days
  756. 32:08late, the packaging was damaged, very
  757. 32:10disappointing. I need it that urgently
  758. 32:13for work. And then the output is an
  759. 32:15email that is immediately given to you
  760. 32:18by the LLM after it reads the prompt.
  761. 32:22Um, so this might work, but it might be
  762. 32:27hard to control, you know, cuz think
  763. 32:30about it. There's multiple steps that
  764. 32:32you have listed and everything is
  765. 32:34embedded in the same prompt. And if you
  766. 32:37wanted to debug step by step and know
  767. 32:39which step is weaker, you couldn't. You
  768. 32:41would have everything mixed together. So
  769. 32:43one advantage of chaining is you know
  770. 32:46you would you would separate the prompts
  771. 32:48so that you can debug them separately
  772. 32:50and it will also lead to um an easier
  773. 32:53manner to improve your workflow. Um
  774. 32:56let's say a first prompt is extract the
  775. 32:59key issues. Identify the key concerns
  776. 33:01mentioned in this customer review. Paste
  777. 33:03a customer review. Second prompt using
  778. 33:07these issues. So you paste back the
  779. 33:09issues. Draft an outline for a
  780. 33:11professional response that acknowledges
  781. 33:13concerns, explains possible reasons, and
  782. 33:15offer a resolution.
  783. 33:19So this is not, you know, prompt number
  784. 33:23three, write the full response. So using
  785. 33:25the outline, um, write the professional
  786. 33:28response
  787. 33:30and then you get your final output.
  788. 33:33So in theory, you can tell me, oh, the
  789. 33:37second approach is better than the first
  790. 33:38one at first. But what you can notice is
  791. 33:41that we can actually test those three
  792. 33:43prompts separately from each other and
  793. 33:46determine if we will get the most gains
  794. 33:48out of uh engineering the first prompt,
  795. 33:52optimizing it or the second one or the
  796. 33:54third one. We now have three prompts
  797. 33:56that are independent from each other.
  798. 33:58And you know maybe if uh the outline was
  799. 34:01better um the performance of uh the
  800. 34:05email the email how much it will the
  801. 34:07open rate maybe or the user satisfaction
  802. 34:10on the response will actually get higher
  803. 34:12you know and so chaining
  804. 34:15improves performance but most
  805. 34:16importantly helps you control your
  806. 34:19workflow and debug it more seamlessly.
  807. 34:23>> Yes. So if we know that the three pump
  808. 34:28independently work very well if we
  809. 34:31combine them one pump and we highlight
  810. 34:33that stepbystep thinking process do we
  811. 34:37does on average we get the same or we
  812. 34:41still have to do that.
  813. 34:44So let me try to rephrase. You say let's
  814. 34:46say we look at the first prompt which
  815. 34:48has uh all three task built in that
  816. 34:50prompt. Um what what exactly do you
  817. 34:54mean? You mean like if we evaluate the
  818. 34:56output and we measure some user insight
  819. 34:58satisfaction etc. Um why don't we just
  820. 35:02modify that prompt and essentially see
  821. 35:04how it improves user satisfaction?
  822. 35:06>> Yeah.
  823. 35:10>> I see. So why do we need the three
  824. 35:12steps?
  825. 35:12>> Yeah.
  826. 35:13>> Yeah. I mean, think about it. The
  827. 35:14intermediate output is what you want to
  828. 35:17see. Like, if I'm debugging the first um
  829. 35:20approach, uh the way I would do it is I
  830. 35:23would capture user insights. Like,
  831. 35:25here's the email. How good was the
  832. 35:26response? Thumbs up, thumbs down. Um was
  833. 35:29your issue issue resolved? Uh thumbs up,
  834. 35:32thumbs down. Those would tell me how
  835. 35:34good is my prompt. And I can engineer
  836. 35:35that prompt, optimize it, and I would
  837. 35:37probably drive some gains. um but I will
  838. 35:40not be able easily to trace back to what
  839. 35:42the problem was. While in the second
  840. 35:44approach, not only I can use the end to
  841. 35:47end metrics to improve my process, I can
  842. 35:49also use the intermediate steps. For
  843. 35:50example, if I look at prompt two and I
  844. 35:53look at the outline and I see the
  845. 35:54outline is actually meh, it's not great.
  846. 35:57Then I think I can get a lot of gains
  847. 35:59out of the outline. Um or the outline is
  848. 36:02actually really good, but the last
  849. 36:04prompt doesn't do a good job at
  850. 36:05translating it into an email. So the
  851. 36:07outline is exactly what I want the the
  852. 36:09LLM to do, but the the translation in a
  853. 36:12customerf facing email is not good. In
  854. 36:14fact, it doesn't follow our vocabulary
  855. 36:16internally. Then I know the third prompt
  856. 36:18is where I would get the most gains.
  857. 36:21So that that's what it allows me to do.
  858. 36:23Have intermediate steps to review. Yeah.
  859. 36:26>> Are there any latency? If it's
  860. 36:29>> we'll talk about it. Are there any
  861. 36:30latency concerns? Yes. um in certain
  862. 36:33applications you don't want to use a
  863. 36:35chain or you don't want to use a long
  864. 36:37chain because it adds latency.
  865. 36:41We'll talk about that later. Good point.
  866. 36:44So practically this is what uh chaining
  867. 36:47complex prompts look like. You have your
  868. 36:49first prompt with your first task. It
  869. 36:51outputs the output is is pasted in the
  870. 36:54second prompt with the second task being
  871. 36:56defined. The output is then pasted into
  872. 36:58the third prompt with the third task
  873. 37:00being defined and so on. That's what it
  874. 37:02looks like in practice.
  875. 37:08Super. U
  876. 37:11we'll talk more later about testing your
  877. 37:13prompts, but there are methods now to do
  878. 37:16it and we'll we'll see later in this
  879. 37:17lecture with our case study how uh we
  880. 37:20can test our prompts. Uh but here is an
  881. 37:23example of how you you might do it. Um
  882. 37:27you you might have a a summarization
  883. 37:31workflow, you know, prompt um that is
  884. 37:34the baseline. It's a single uh prompt.
  885. 37:37You might have a refined summarization
  886. 37:39which is a modified prompt of this or a
  887. 37:42workflow with a chain, you know. Um and
  888. 37:46then you have your test case which is
  889. 37:48the input that you want to summarize,
  890. 37:51let's say. And then you have the
  891. 37:53generated output and you can have humans
  892. 37:55go and rate these outputs and you would
  893. 37:58notice uh that the baseline is better or
  894. 38:01worse than the refined prompt. Of
  895. 38:03course, this manual approach takes time.
  896. 38:06Um but it's a good way to start and
  897. 38:09usually the advice is get hands-on at
  898. 38:12the beginning because you would quickly
  899. 38:13notice some issues and it will give you
  900. 38:15better intuition on what tweaks can lead
  901. 38:17to better performance. However, if you
  902. 38:19wanted to scale that system across many
  903. 38:21products, many parts of your codebase,
  904. 38:24uh you might want to find a way to do
  905. 38:25that automatically without asking humans
  906. 38:27to review and grade summaries, right? Um
  907. 38:31one approach is to use, you know,
  908. 38:34platforms like at Portera, we our team
  909. 38:37uses a platform called Promptu that
  910. 38:39allows you to actually automate part of
  911. 38:41this testing. Um, in a nutshell, what it
  912. 38:44does is it can allow you to run the same
  913. 38:48prompt with five different LLMs
  914. 38:51immediately, put everything in a table
  915. 38:52that makes it super easy for a human to
  916. 38:54grade, let's say. Or alternatively, it
  917. 38:57might allow you to define define LLM
  918. 39:01judges.
  919. 39:02LLM judges can come in different
  920. 39:04flavors. For example, I can have an LLM
  921. 39:07judge that does a pair-wise comparison.
  922. 39:10So what the LLM is asked to do is here
  923. 39:12are two summaries just tell me which one
  924. 39:14is better than the other one. Um that's
  925. 39:17what the LLM does and that can be used
  926. 39:19as a proxy for how good the
  927. 39:20summarization baseline versus the
  928. 39:22refined version is. Another way to do an
  929. 39:25LLM judge is if you do it for a single
  930. 39:29answer grading. So here's a summary,
  931. 39:31rate it from 1 to five, you know, and
  932. 39:34then you can go even deep deeper and do
  933. 39:37a a reference guided pair wise
  934. 39:39comparison or you add also a rubric. You
  935. 39:41say a five is when a summary is below
  936. 39:45100 characters. I'm just making up below
  937. 39:47100 characters mentions at least three
  938. 39:50key points that are distinct and starts
  939. 39:52with a first sentence that displays the
  940. 39:54overview and then goes into detail.
  941. 39:56That's a great summary. Number five out
  942. 39:57of five. zero is the LLM failed to
  943. 40:01summarize and actually was very verbose
  944. 40:04let's say and so you put a rubric behind
  945. 40:07it and you have an LLM as just finding
  946. 40:09the rubric of course you can now pair
  947. 40:12different techniques you can do a few
  948. 40:13shot for the rubric you can actually
  949. 40:15give examples of a five out of fives
  950. 40:17four out of fours three out of threes
  951. 40:19because now you know multiple techniques
  952. 40:22okay
  953. 40:24does that make sense
  954. 40:26yeah Okay, so that was the second
  955. 40:29section on prompt engineering uh or the
  956. 40:32first line of optimization. Now let's
  957. 40:36say you've exhausted all your chances
  958. 40:38for prompt engineering and you're
  959. 40:40thinking about actually touching the
  960. 40:41model, modifying its weights or
  961. 40:44fine-tuning it. In other words, um I was
  962. 40:47telling you I'm not a a fan of
  963. 40:49fine-tuning. There's a few reasons why.
  964. 40:52Um, one, it requires substantial label
  965. 40:57data typically to fine-tune. Although
  966. 41:00now there are approaches that are
  967. 41:02getting better at fine-tuning that look
  968. 41:04more like few shot prompting actually
  969. 41:06than fine-tuning. It's sort of merging.
  970. 41:09Uh, although one modifies the weight,
  971. 41:11the other doesn't modify the weights.
  972. 41:13Um, fine-tune models may also overfeed
  973. 41:15to specific data. We're going to see a
  974. 41:17funny example actually. um losing uh
  975. 41:20their general purpose utility. So you
  976. 41:22might fine-tune a model and actually
  977. 41:24when someone asks a pretty generic
  978. 41:26question, it doesn't do well anymore.
  979. 41:28You know, it might do well on your task.
  980. 41:29So it might be relevant or not. And then
  981. 41:31it's it's time and cost intensive.
  982. 41:33That's my main problem. And you know, at
  983. 41:36at work here, we don't we don't we steer
  984. 41:39away from fine-tuning as much as
  985. 41:41possible. uh because by the time you're
  986. 41:43done fine-tuning your model, the next
  987. 41:44model is out and it's actually beating
  988. 41:46your fine-tuned version of the previous
  989. 41:48model. So I would steer away from
  990. 41:50fine-tuning as much as you can. The
  991. 41:52advantage of the prompt engineering
  992. 41:53methods we've seen is you can put the
  993. 41:56next best pre-trained model directly in
  994. 41:59your code. It will update everything
  995. 42:01immediately. Fine-tuning doesn't work
  996. 42:02like that.
  997. 42:06There are advantages though where it
  998. 42:07still makes sense. if the task requires
  999. 42:10repeated high precision outputs such as
  1000. 42:12legal scientific explanation and if the
  1001. 42:14general purpose LLM struggles with
  1002. 42:17domain specific language. So uh let's
  1003. 42:20look at a quick example together um
  1004. 42:23which is an example uh from Ross
  1005. 42:27Lazerovit's um I think it was a couple
  1006. 42:29of years ago September 23 where um
  1007. 42:34uh Ross uh tried to do Slack finetuning.
  1008. 42:38So he looked at a lot of Slack messages
  1009. 42:40within his company and he was like, I'm
  1010. 42:43going to fine-tune a model that speaks
  1011. 42:45like us or operates like us because this
  1012. 42:47is how we work, right? This is the data
  1013. 42:49that represents how people work at the
  1014. 42:51company. Um and so it he actually went
  1015. 42:56ahead and fine-tuned the model. Um gave
  1016. 42:58it a prompt like hey write a you know he
  1017. 43:01was delegating to the model write a
  1018. 43:02500word blog post on prompt engineering.
  1019. 43:06And the model responded I shall work on
  1020. 43:08that in the morning. Um
  1021. 43:11and and then he tries to push the model
  1022. 43:14a little further and say it's morning
  1023. 43:16now. Um and the model said I'm writing
  1024. 43:19right now. It's 6:30 a.m. here. Uh,
  1025. 43:22write it now. Okay, please.
  1026. 43:26Okay, I shall write it now. I actually
  1027. 43:28don't know what you would like me to say
  1028. 43:29about prompt engineering. I can only
  1029. 43:31describe the process. The only thing
  1030. 43:33that comes to mind for a headline is how
  1031. 43:34do we build a prompt? You know, it's
  1032. 43:37kind of a funny example for fine-tuning
  1033. 43:40because it's true that it went wrong.
  1034. 43:43like he was supposed to think like I
  1035. 43:45want the model to speak like us at work
  1036. 43:47and it ended up acting like people and
  1037. 43:50not actually following instructions.
  1038. 43:55So one example why I would steer away
  1039. 43:57from fine tuning.
  1040. 44:02Super
  1041. 44:05uh
  1042. 44:07let's talk about rags. Um rags is
  1043. 44:10important. It's important to know out
  1044. 44:12there and at least having the basics.
  1045. 44:14It's a very common interview question by
  1046. 44:15the way. If you go interview for a job,
  1047. 44:18they might ask you to explain in a
  1048. 44:20nutshell to a 5-year-old what is a rag
  1049. 44:22and hopefully after that you'll be able
  1050. 44:23to do it. Um, so, uh, we've seen some of
  1051. 44:27the challenges with standalone LLMs.
  1052. 44:30Those challenges include the context
  1053. 44:32window being small, the fact that it's
  1054. 44:35hard to remember details within a large
  1055. 44:37context window, um, knowledge gaps, you
  1056. 44:41know, uh, cutoff dates. you mentioned
  1057. 44:43earlier the model might be trained up to
  1058. 44:44a date and then it cannot follow the
  1059. 44:46trends or be up to date hallucinations
  1060. 44:50there are some fields think about
  1061. 44:52medical diagnosis where hallucination
  1062. 44:54are very costly you can't afford a
  1063. 44:55hallucination you know even in education
  1064. 44:58imagine deploying a model for the US
  1065. 45:01youth education and it hallucinates and
  1066. 45:03it teaches millions of people something
  1067. 45:05completely wrong it's a problem um and
  1068. 45:08then lack of sources uh a lot of fields
  1069. 45:12love sources. Research fields love
  1070. 45:14sources. Education loves sources. Legal
  1071. 45:18loves sources as well. And so um the
  1072. 45:21pre-trained LLM doesn't do a good job to
  1073. 45:24source. And in fact, if you if you have
  1074. 45:26tried to find sources on a plain LLM, it
  1075. 45:29actually hallucinates a lot. It makes up
  1076. 45:31research papers. It just lists like
  1077. 45:33completely fake stuff. Um so how do we
  1078. 45:36solve that? Um with a rag. Rag
  1079. 45:39integrates with external knowledge
  1080. 45:41sources, databases, documents, APIs. Uh
  1081. 45:46it ensures that answers are more
  1082. 45:49accurate, up-to-date and grounded
  1083. 45:51because you can actually update your
  1084. 45:53document. Your your drive is always up
  1085. 45:55to date. I mean ideally you're always
  1086. 45:57pushing new documents to it. And when
  1087. 45:59you query what is our Q4 performance in
  1088. 46:02sales hopefully there is the last board
  1089. 46:04deck in uh the drive and it can read the
  1090. 46:07last board deck you know um and more
  1091. 46:10developer control. We'll see why um rags
  1092. 46:14allow for targeted customization without
  1093. 46:16actually requiring the retraining of the
  1094. 46:18model. In fact, you don't touch the
  1095. 46:19model with rags. It's really a technique
  1096. 46:22that is put on top of the model. So to
  1097. 46:25see an example of a a rag, this is a
  1098. 46:27question in answering a application
  1099. 46:31where we're in the medical field and a
  1100. 46:34user is asking uh a query. What are the
  1101. 46:38side effects of drug X?
  1102. 46:41This is an important question. You can't
  1103. 46:43hallucinate. You need to source. You
  1104. 46:45need to be up to date. Maybe there is a
  1105. 46:48new
  1106. 46:49update to that drug that is now in the
  1107. 46:51database and you need to read that. So
  1108. 46:53you have to like a rag is a great
  1109. 46:55example of what you would want to use
  1110. 46:56here. The the way it works is you have
  1111. 46:59your knowledge base of a bunch of
  1112. 47:00documents. Um what you do is you use an
  1113. 47:04embedding to embed those documents into
  1114. 47:07lower dimensional representations. Uh so
  1115. 47:10for example if the document is a PDF a
  1116. 47:13long PDF uh you might you know read the
  1117. 47:17PDF understand it and then embed it.
  1118. 47:19We've seen plenty of embedding
  1119. 47:21approaches together. triplet loss etc.
  1120. 47:23remember. So imagine one of them here
  1121. 47:26for LLMs is embedding those documents
  1122. 47:28into lower representation.
  1123. 47:31If the representation is too small, you
  1124. 47:34will lose information. If it's too big,
  1125. 47:36you will add latency, right? It's a
  1126. 47:38trade-off. Um
  1127. 47:41you will store typically those
  1128. 47:43representation into a database um called
  1129. 47:46a vector database. There's a lot of
  1130. 47:48vector database providers as out there.
  1131. 47:51Um, you know, I I I think I've listed a
  1132. 47:55couple that are very common. No, I
  1133. 47:57haven't listed but I can I can share
  1134. 47:59afterwards. Um, the vector database is
  1135. 48:01essentially storing those vector in a
  1136. 48:02very efficient manner allowing the fast
  1137. 48:05retrieval
  1138. 48:06with a certain distance metric. So um
  1139. 48:09what you do is you also embed usually
  1140. 48:11with the same algorithm the user prompts
  1141. 48:15and you run a retrieval process which is
  1142. 48:18essentially saying um based on the
  1143. 48:22embedding from the user query and the
  1144. 48:24vector database find the relevant
  1145. 48:26documents based on the distance between
  1146. 48:29those embeddings.
  1147. 48:31Once you found the relevant documents,
  1148. 48:33you pull them and then you add them to
  1149. 48:35the user query with a system prompt or a
  1150. 48:38prompt template on top. So the prompt
  1151. 48:40template can be um answer user query
  1152. 48:45based on list of documents.
  1153. 48:48If answer not in the document say I
  1154. 48:51don't know. That's your chrome templates
  1155. 48:54where the user query is pasted. the
  1156. 48:56documents are pasted and then your
  1157. 48:58output um should be what you want
  1158. 49:00because it's now grounded in the
  1159. 49:02document. You can also add to this
  1160. 49:04prompt templates tell me the exact page,
  1161. 49:07chapter line of the document that was
  1162. 49:09relevant and in fact link it as well
  1163. 49:12just to be more precise.
  1164. 49:17Any question on rags? There's a simple
  1165. 49:20vanilla rag.
  1166. 49:23Yeah. Yes. Do the embeddings still
  1167. 49:26retain information about what's found on
  1168. 49:28web page and
  1169. 49:31>> question is do the document embeddings
  1170. 49:33still retain the information of the
  1171. 49:36location of the information within that
  1172. 49:37document especially in big documents. Um
  1173. 49:40great question. We'll get to it in a in
  1174. 49:42a second because you're right that the
  1175. 49:44vanilla rag might not do a good job with
  1176. 49:46very large documents. So let's say you
  1177. 49:49know when you open a medication box and
  1178. 49:52you have this gigantic white paper with
  1179. 49:55all the information and it's very long
  1180. 49:59uh maybe a vanilla rag would not cut it.
  1181. 50:01So what people have figured out is a
  1182. 50:03bunch of techniques to improve rags and
  1183. 50:05in fact chunking is a great technique
  1184. 50:07that is very popular. So you might
  1185. 50:09actually store in the vector database
  1186. 50:11the embedding of the full document and
  1187. 50:13on top of that you will also store a
  1188. 50:15chapter level uh vector. Yeah. And when
  1189. 50:18you retrieve you retrieve the document,
  1190. 50:20you retrieve the chapter and that allows
  1191. 50:22you to be more precise with the
  1192. 50:24sourcing. It's one example.
  1193. 50:26Um another technique that's popular is
  1194. 50:29hide um hypothetical document embeddings
  1195. 50:34where a group of researchers
  1196. 50:37um published a paper showing that when
  1197. 50:40you get your user query one of the main
  1198. 50:43problem is the user query actually does
  1199. 50:45not look like your documents. For
  1200. 50:48example, the user query might be what
  1201. 50:50are the side effect of drug X when
  1202. 50:52actually in the document in the vector
  1203. 50:55database the vectors represents very
  1204. 50:57long documents. So how do you guarantee
  1205. 50:59that the vector embedding is going to be
  1206. 51:01close to the document embedding? What
  1207. 51:03they do is they use the user query to
  1208. 51:06generate a fake hallucinated document.
  1209. 51:09They embed that document and then they
  1210. 51:11compare to the vector in the vector
  1211. 51:15database. Does that make sense? So for
  1212. 51:18example, the user says, "What is the
  1213. 51:20side effect of drug X?" There's a prompt
  1214. 51:23that this is given to another prompt
  1215. 51:24that says based on this user query,
  1216. 51:26generates a five-page report answering
  1217. 51:30the user query. It generates potentially
  1218. 51:32a completely fake um u answer. You embed
  1219. 51:37that and it will be closer to the
  1220. 51:39document that you're looking for likely.
  1221. 51:42Yeah,
  1222. 51:44it's one example of a of a rag approach.
  1223. 51:47Again, the purpose of this lecture is
  1224. 51:49not to go through all these three and
  1225. 51:51explain you every single methods that
  1226. 51:52has been discovered for rags, but I just
  1227. 51:55wanted to show you how much research has
  1228. 51:56been done between 2020 and 2025 in rags
  1229. 52:00and how many branches of research you
  1230. 52:02you now have that you can uh learn from.
  1231. 52:06The survey paper is linked in the
  1232. 52:07slides, by the way, and I'll share them
  1233. 52:09after the lecture.
  1234. 52:17Super.
  1235. 52:21So, uh, we've made some progress.
  1236. 52:24Hopefully now you feel like if you were
  1237. 52:26to start an LLM application, you know
  1238. 52:29how to do better prompts, you know how
  1239. 52:30to do chains, you know how to do
  1240. 52:31fine-tuning. Uh you also know how to do
  1241. 52:33retrieval and you have the baggage of
  1242. 52:36techniques that you can go and read and
  1243. 52:37find the code base, pull the code, v
  1244. 52:40code it, but you have the breath. Now
  1245. 52:45u the next set of uh topics we're going
  1246. 52:49to see is around uh the question of how
  1247. 52:52could we extend the capabilities of LLM
  1248. 52:54from performing single tasks and hence
  1249. 52:56with external knowledge to handling
  1250. 52:59multi-step autonomous workflows. Yeah.
  1251. 53:02And this is where we get into proper
  1252. 53:04agent TKI.
  1253. 53:08So let's talk about agent TKI workflows
  1254. 53:12towards autonomous and specialized
  1255. 53:14systems.
  1256. 53:15Then we'll talk about evals. Then we'll
  1257. 53:17see multi- aent systems. And we'll end
  1258. 53:20with a with a little
  1259. 53:23thoughts on what's next in AI.
  1260. 53:27So um Andrew actually um coined the term
  1261. 53:33agentic AI workflows. Um and his reason
  1262. 53:37was that a lot of companies use uh let's
  1263. 53:40say agents, agents, agents everywhere,
  1264. 53:42agents everywhere.
  1265. 53:44If you go and work at these companies,
  1266. 53:46you would notice that they mean very
  1267. 53:48different things by agent. Some people
  1268. 53:49actually have a prompt and they call it
  1269. 53:51an agent. You know, other people, they
  1270. 53:53have a very complex multi- aent system.
  1271. 53:57They call it an agent. And so calling
  1272. 53:59everything an agent doesn't do it
  1273. 54:00justice. So Andrew says, let's call it
  1274. 54:03agentic workflows because in practice
  1275. 54:06it's a bunch of prompts with tools with
  1276. 54:10additional resources, API calls that
  1277. 54:13ultimately are put in a workflow and you
  1278. 54:15can call that workflow agentic. So it's
  1279. 54:18all about the multi-step process
  1280. 54:22um to complete the task.
  1281. 54:26Also calling it agentic workflow allows
  1282. 54:29us to not mix it up with what I called
  1283. 54:31agent the in the last lecture with
  1284. 54:33reinforcement learning because in RL
  1285. 54:35agent has a very specific definition
  1286. 54:38interacts with an environment passes
  1287. 54:39from one state to the other has a reward
  1288. 54:41and an observation. You remember that
  1289. 54:42chart right?
  1290. 54:47So um here's an example of how we move
  1291. 54:50from a one-step prompt to a a multi-step
  1292. 54:53agentic workflow. Let's say a user
  1293. 54:56queries a a a
  1294. 55:00product, what is your refund policy on a
  1295. 55:02chatbot? Um, and the response using a
  1296. 55:05rag says refunds are available within 30
  1297. 55:08days of purchase. And maybe the rag can
  1298. 55:10even look linked to the policy document.
  1299. 55:13That's what we learned so far. Um,
  1300. 55:15instead an agent workflow can function
  1301. 55:18like this. The user says, "Can I get a
  1302. 55:21refund for my order?" And the response
  1303. 55:24via the agentic workflow is the agent
  1304. 55:27retrieves the refund policy using a rag.
  1305. 55:29The agent then follows up with the users
  1306. 55:32and says can you provide your order
  1307. 55:34number? Um then the agent queries an API
  1308. 55:37to check the order details and finally
  1309. 55:39comes back to the user and confirms your
  1310. 55:41order qualifies for a refund. The amount
  1311. 55:44will be processed in 3 to five business
  1312. 55:46days. This is much more thoughtful than
  1313. 55:48the first version which is sort of
  1314. 55:50vanilla, right?
  1315. 55:53So that's what we're going to talk about
  1316. 55:54in the next couple of slides is how do
  1317. 55:57we get from the first one to the second
  1318. 55:58one.
  1319. 56:01Uh there are plenty of specialized
  1320. 56:04agentic workflows online. You know,
  1321. 56:05you've heard and if you hang out in SF,
  1322. 56:07you probably see a bunch of billboards,
  1323. 56:10uh, you know, AI software engineer, AI
  1324. 56:12skills mentor you've interacted with in
  1325. 56:14the class to work, AIS SDR, AI lawyers,
  1326. 56:17AI,
  1327. 56:19you know, specialized cloud engineer,
  1328. 56:23you know, it would be a stretch to say
  1329. 56:25that everything works, but there's work
  1330. 56:26being done towards that. You know,
  1331. 56:33I'm not personally a fan of putting a
  1332. 56:35face behind those things. I think it's
  1333. 56:37gimmicky and I think in a few years from
  1334. 56:38now, actually, very few products will
  1335. 56:41have a human face behind it. Uh, but
  1336. 56:44might be a marketing tactic from some
  1337. 56:45startups.
  1338. 56:47It's more scary than it is engaging,
  1339. 56:49frankly. Um, okay. I want to talk about
  1340. 56:52the parading shift. Uh, and that's
  1341. 56:54especially useful. Let's say you're a
  1342. 56:56software engineer or you're planning to
  1343. 56:58be a software engineer because software
  1344. 57:00engineering as a discipline is sort of
  1345. 57:02shifting or at least the best engineers
  1346. 57:04I've worked with are able to move from a
  1347. 57:07deterministic mindset to a fuzzy mindset
  1348. 57:11and balance between the two whenever
  1349. 57:13they need to get something done. So
  1350. 57:14here's the paradigm shift between
  1351. 57:16traditional software and agent TKI
  1352. 57:19software. The first one is uh the way
  1353. 57:22you handle data. Traditional software
  1354. 57:24deals with structured data. You have
  1355. 57:26JSONs, you have databases. They're
  1356. 57:28pasted in a very structured manner in a
  1357. 57:31data engineering pipeline and then
  1358. 57:33they're used to be displayed on a
  1359. 57:35certain interface. The user might for
  1360. 57:37feel a form that is then retrieved and
  1361. 57:40pasted in the database. All of that
  1362. 57:41historically has been structured data.
  1363. 57:44Now more and more companies are handling
  1364. 57:46uh free form text, images um and and all
  1365. 57:50of that requires dynamic interpretation
  1366. 57:53um uh to transform an input into an
  1367. 57:55output. Um the software itself used to
  1368. 58:00be deterministic. Now you have a lot of
  1369. 58:01software that is fuzzy and fuzzy
  1370. 58:04software creates so many issues. I mean,
  1371. 58:07imagine if you let your user ask
  1372. 58:09anything on your website.
  1373. 58:11The chances that it breaks is
  1374. 58:13tremendous. The chances that you're
  1375. 58:15attacked is tremendous. The chances,
  1376. 58:17it's really, really complicated. It's
  1377. 58:18more complicated than people make it
  1378. 58:20seem uh on Twitter. Um, fuzzy
  1379. 58:23engineering is truly hard. You know, you
  1380. 58:26might get hate as a company because one
  1381. 58:28user did something that you authorized
  1382. 58:30them to do that ended up breaking the
  1383. 58:32database and ended up, you know, we've
  1384. 58:34seen that with many companies in the
  1385. 58:35last couple of years. So, it takes a
  1386. 58:37very specialized engineering mindset to
  1387. 58:39do fuzzy engineering, but also know when
  1388. 58:41you need to be deterministic.
  1389. 58:44Um, the other thing I call is with
  1390. 58:47agentic AI software, um, you you you
  1391. 58:50sort of want to think about your
  1392. 58:52software as like you're a manager. So
  1393. 58:55you're familiar with the monolith or or
  1394. 58:58you know microservices approaches in
  1395. 59:00software you know where you structure
  1396. 59:02your software in different you know
  1397. 59:05boxes that can talk to each other and it
  1398. 59:07allows teams to debug one section at a
  1399. 59:10time you know now the equivalent with
  1400. 59:12agentic AI is you think as a manager. So
  1401. 59:14you think okay if I was to delegate my
  1402. 59:17product to be done by a group of humans
  1403. 59:20what would be those roles? Would I have
  1404. 59:22a graphic designer that then you know
  1405. 59:24puts together a chart and then sends it
  1406. 59:26to a marketing manager that converts it
  1407. 59:28into a nice blog post that then gives it
  1408. 59:30to the performance marketing expert that
  1409. 59:32then publishes the work the blog post
  1410. 59:34and then optimizes and AB tests then to
  1411. 59:36a data scientist that analyzes the data
  1412. 59:39and then puts hypothesis and validates
  1413. 59:41them or invalidates them. That's how you
  1414. 59:43would typically think if you're building
  1415. 59:45an agency AI software, you know, when
  1416. 59:48actually the equivalent of that in
  1417. 59:50traditional software might be completely
  1418. 59:52different. It might be we have a data
  1419. 59:54engineer box right here that handles all
  1420. 59:56our data engineering. And then here we
  1421. 59:59have the UIUX stuff. Everything UIUX
  1422. 1:00:02related goes here. And you know,
  1423. 1:00:04companies might structure it in very
  1424. 1:00:06different ways. And here's the business
  1425. 1:00:07logic that we want to care about. And
  1426. 1:00:09there's five engineers working on the
  1427. 1:00:10business logic, let's say.
  1428. 1:00:14Okay.
  1429. 1:00:16Uh testing and debugging is also very
  1430. 1:00:18different. Um and we'll talk about it in
  1431. 1:00:21the next section.
  1432. 1:00:24Uh the other thing that uh I feel
  1433. 1:00:28matters is with AI in engineering the
  1434. 1:00:31cost of experimentation is going down
  1435. 1:00:33drastically and so people I feel should
  1436. 1:00:36be more comfortable throwing away code.
  1437. 1:00:39You know, it's it's like in traditional
  1438. 1:00:41software engineering, you probably don't
  1439. 1:00:43throw away code a ton. You you build a
  1440. 1:00:45code and it's solid and it's bulletproof
  1441. 1:00:47and then you you update it over time.
  1442. 1:00:50When we've seen AI companies be more
  1443. 1:00:53comfortable throwing away codes, you
  1444. 1:00:55know, which has advantages in terms of
  1445. 1:00:58the speed at which you move, but also
  1446. 1:01:00disadvantages in terms of the quality of
  1447. 1:01:03your software that it can break more.
  1448. 1:01:05No.
  1449. 1:01:08Okay. So anyway, just wanted to do an
  1450. 1:01:10apartate on the the paring shift from
  1451. 1:01:12deterministic to fuzzy engineering.
  1452. 1:01:17Um,
  1453. 1:01:20oh, and actually I can give you an
  1454. 1:01:21example from uh from workera that we
  1455. 1:01:24learned uh probably over the last 12
  1456. 1:01:26months is like if you if you've used
  1457. 1:01:28work era you you might have seen that
  1458. 1:01:30the interface has um asks you sometimes
  1459. 1:01:33multiple choice questions and sometimes
  1460. 1:01:35it asks you multiple select and
  1461. 1:01:37sometimes it asks you drag and drop
  1462. 1:01:38ordering matching whatever right those
  1463. 1:01:41are example of deterministic item types
  1464. 1:01:44meaning you answer the question on a
  1465. 1:01:46multiple choice there's one correct
  1466. 1:01:47answer. It's fully deterministic. On the
  1467. 1:01:50other hand, you sometimes have voice
  1468. 1:01:52questions uh where you go through a role
  1469. 1:01:55play or you have voice plus coding
  1470. 1:01:57questions where your code is being read
  1471. 1:01:59by the interface or whatever. Those are
  1472. 1:02:02fuzzy meaning the scoring algorithm
  1473. 1:02:05might actually make mistakes and those
  1474. 1:02:07mistakes might be costly. And so
  1475. 1:02:09companies have to figure out a human in
  1476. 1:02:12the loop system um which you might have
  1477. 1:02:14seen with the appeal feature at the end.
  1478. 1:02:16So at the end of the assessment you have
  1479. 1:02:17an appeal feature where it allows you to
  1480. 1:02:19say I want to appeal the agent because I
  1481. 1:02:22want to challenge what the agent said on
  1482. 1:02:24my answer because I thought I was better
  1483. 1:02:26than what the agent thought. And then
  1484. 1:02:28you bring a human in the loop that then
  1485. 1:02:30can fix the agent can tell the agent
  1486. 1:02:31actually you were too harsh on the
  1487. 1:02:33answer of this person. Um, and you know
  1488. 1:02:36that's an example of a fuzzy engineered
  1489. 1:02:39system that then adds a human in the
  1490. 1:02:41loop to make it more aligned. And so if
  1491. 1:02:44you're building a company, I would
  1492. 1:02:45encourage you to think about what can I
  1493. 1:02:48get done with determinism and let's get
  1494. 1:02:50that done. And then the fuzzy stuff, I
  1495. 1:02:52want to do fuzzy because it allows more
  1496. 1:02:54interaction. It allows more back and
  1497. 1:02:56forth, but I need to put guard rails
  1498. 1:02:58around it. And how am I going to design
  1499. 1:03:00those guard rails pretty much?
  1500. 1:03:04Here's another example uh from
  1501. 1:03:06enterprise workflows um which are likely
  1502. 1:03:10to change due to agent TKI. Um this is a
  1503. 1:03:13paper from McKenzie uh I believe from
  1504. 1:03:16last year where they looked at a
  1505. 1:03:18financial institution and they said that
  1506. 1:03:20you know we observed that they often
  1507. 1:03:22spend one to four weeks to create a
  1508. 1:03:24credit risk memo and here's the process.
  1509. 1:03:27A relationship manager
  1510. 1:03:29um gathers data from 15 and more than 15
  1511. 1:03:33sources on the borrower, loan type,
  1512. 1:03:36other factors. Then the relationship
  1513. 1:03:39manager and the credit analyst
  1514. 1:03:41collaboratively analyze that data from
  1515. 1:03:43these sources. Then the credit analyst
  1516. 1:03:45typically spends you know 20 more 20
  1517. 1:03:48hours or more writing a memo and then
  1518. 1:03:51goes back to the relationship manager.
  1519. 1:03:53They give feedback and then they go
  1520. 1:03:55through this loop again and again and uh
  1521. 1:03:58it takes a long time to get a credit
  1522. 1:03:59memo out.
  1523. 1:04:01They then run a a research study where
  1524. 1:04:05they changed the process. They said
  1525. 1:04:07Genai agents could actually cut time um
  1526. 1:04:09by 20 to 60% on credit risk memos and
  1527. 1:04:14the process has changed to the
  1528. 1:04:15relationship manager directly work with
  1529. 1:04:17the genai agent system provides relevant
  1530. 1:04:20materials that needs to produce the
  1531. 1:04:22memo. The agent subsizes the project
  1532. 1:04:24into tasks that are assigned to
  1533. 1:04:26specialist agents gathers and analyzes
  1534. 1:04:29the data from multiple sources. Drafts a
  1535. 1:04:31memo. Then the relationship manager and
  1536. 1:04:34the credit analyst sit down together,
  1537. 1:04:35review the memo, give feedback to the
  1538. 1:04:37agent, and within, you know, 20 to 60%
  1539. 1:04:40less time are done. And so this is an
  1540. 1:04:43example where you're actually not
  1541. 1:04:45changing the human stakeholders, you're
  1542. 1:04:47just changing the process and adding
  1543. 1:04:49genai to uh reduce the time it takes to
  1544. 1:04:52get a credit memo out. It turns out that
  1545. 1:04:55um imagine you're an enterprise and you
  1546. 1:04:58have you know 100,000 employees and
  1547. 1:05:01there's a lot of enterprises with
  1548. 1:05:03100,000 employees out there. Uh you are
  1549. 1:05:06currently under crisis in terms of
  1550. 1:05:08redesigning your workflows. You you are
  1551. 1:05:11you know it turns out that if you
  1552. 1:05:12actually pull the job descriptions from
  1553. 1:05:14the HR system and you interpret them you
  1554. 1:05:17also pull the business process workflows
  1555. 1:05:20that you have encoded in your drive. you
  1556. 1:05:23actually can find gains in multiple
  1557. 1:05:25places and uh in the next few years
  1558. 1:05:28you're probably going to see workflows
  1559. 1:05:29being more optimized to add genai. Um
  1560. 1:05:32even if that happens the hardest part is
  1561. 1:05:34changing people. What we know this is
  1562. 1:05:36this is great in theory but now let's
  1563. 1:05:39try to fit that second workflow for
  1564. 1:05:4210,000 credit risk analysts and
  1565. 1:05:45relationship managers. My guess is it
  1566. 1:05:48will take years. It will take 10 20
  1567. 1:05:50years to get to this being actually done
  1568. 1:05:53at scale within an organization because
  1569. 1:05:56change is so hard you know so hard to
  1570. 1:05:58rewire business workflows job
  1571. 1:06:02descriptions incentivize people to do
  1572. 1:06:04different and be different and train
  1573. 1:06:06them and so so you know this is what the
  1574. 1:06:09world is going towards but it's going to
  1575. 1:06:11take a long time I think
  1576. 1:06:14um okay then I want to talk about how
  1577. 1:06:17the agent actually works and What are
  1578. 1:06:18the core core components of an agent? U
  1579. 1:06:22imagine um a travel booking AI agent.
  1580. 1:06:25That's an easy example you've all
  1581. 1:06:27thought about. I still haven't been able
  1582. 1:06:29to get an agent to book a trip for me or
  1583. 1:06:31or I was scared because it was going to
  1584. 1:06:33book a very expensive or long trip. But
  1585. 1:06:36in theory um uh you you can you can have
  1586. 1:06:39a travel booking agent that has prompts.
  1587. 1:06:42So the prompts we've seen we know the
  1588. 1:06:44methods to optimize those prompts. That
  1589. 1:06:46travel agent also has a content
  1590. 1:06:48management context management system
  1591. 1:06:50which is essentially the memory of what
  1592. 1:06:52it knows about the user. That context
  1593. 1:06:54management system might include a core
  1594. 1:06:57memory or working memory and an archival
  1595. 1:07:00memory. Okay. What the difference is um
  1596. 1:07:04within memory um is not every memory
  1597. 1:07:08needs to be fast to access. Like think
  1598. 1:07:11about it. you you're morted on a product
  1599. 1:07:13and the first question is hi what's your
  1600. 1:07:15name and I say my name is Keon that's
  1601. 1:07:19probably going to sit in the working
  1602. 1:07:20memory because the agents every time
  1603. 1:07:22he's going to talk to me is going to
  1604. 1:07:23want to use my name right but then maybe
  1605. 1:07:25the second question is Keon what's your
  1606. 1:07:27birthday and I give it my birthday does
  1607. 1:07:29it need my birthday every day probably
  1608. 1:07:31not so it's probably going to park it on
  1609. 1:07:33the long-term memory or the archival
  1610. 1:07:35memory and those memories are slower to
  1611. 1:07:39access farther down the stack and you
  1612. 1:07:42know that structure allows agent to
  1613. 1:07:44determine what's the working memory and
  1614. 1:07:46what's the long-term memory you know and
  1615. 1:07:49that makes it easier for the agent to
  1616. 1:07:50retrieve super fast cuz think about it
  1617. 1:07:52when you interact with GPT you feel that
  1618. 1:07:55it's very personal at times right you
  1619. 1:07:57feel like it understands you um imagine
  1620. 1:08:00every time you call it it has to read
  1621. 1:08:02the memories right and that can be
  1622. 1:08:03costly it's like a very it's a very
  1623. 1:08:06burdensome cost because it happens every
  1624. 1:08:08time you you talk to it So you want to
  1625. 1:08:11be highly optimized with the working
  1626. 1:08:13memory. You know, if it takes three
  1627. 1:08:15seconds to look in the memory, every
  1628. 1:08:17time you're going to talk to your LLM,
  1629. 1:08:18it's going to take three seconds, which
  1630. 1:08:19you don't want. So anyway, and then you
  1631. 1:08:23have the tools. The tools can include uh
  1632. 1:08:25APIs like a flight search API, hotel
  1633. 1:08:28booking API, car rental API, weather
  1634. 1:08:30API, and then the payment processing
  1635. 1:08:33API. And typically, you would want to
  1636. 1:08:36tell your agent how that API works. It
  1637. 1:08:39turns out that agents or LLMs I should
  1638. 1:08:42say are very good at reading API
  1639. 1:08:44documentation. So you give it the API
  1640. 1:08:46documentation and it reads the JSON and
  1641. 1:08:48it reads what does a get request look
  1642. 1:08:50like and this is the format that I need
  1643. 1:08:53to push and then it pushes it in that
  1644. 1:08:55format let's say and then it retrieves
  1645. 1:08:58something.
  1646. 1:09:00Does that make sense? Those different
  1647. 1:09:02components
  1648. 1:09:04u you know entropic also talks about
  1649. 1:09:06resources. resources is data that is
  1650. 1:09:09sitting somewhere that you might let
  1651. 1:09:11your agent read. For example, if you're
  1652. 1:09:14building your startups, you have a CRM.
  1653. 1:09:16A CRM has data in it and you want to do
  1654. 1:09:19lookups in that data. You will probably
  1655. 1:09:21give a lookup tool and you will give
  1656. 1:09:24access to the resource and it will do
  1657. 1:09:26lookups whenever you want super fast.
  1658. 1:09:31uh this type of architecture can be
  1659. 1:09:34built with different degrees of autonomy
  1660. 1:09:36from the least autonomous to the most
  1661. 1:09:38autonomous and I'll give you a few
  1662. 1:09:40examples
  1663. 1:09:41less autonomous would be you've
  1664. 1:09:44hardcoded the steps so let's say I tell
  1665. 1:09:46the travel agent
  1666. 1:09:48first identify the intent then um look
  1667. 1:09:52up in the database the history of this
  1668. 1:09:55customer with us and their preferences
  1669. 1:09:58then go to the flight API blah blah blah
  1670. 1:10:00then go to the I would hardcode the
  1671. 1:10:02steps. Okay, that's the least
  1672. 1:10:04autonomous. The semi autonomous is I
  1673. 1:10:08might hardcode the tools but I'm not
  1674. 1:10:11going to hardcode the steps. So, I'm
  1675. 1:10:13going to tell the agent um you're act
  1676. 1:10:16like a travel agent and
  1677. 1:10:19um and uh you your task is to help the
  1678. 1:10:24person book a travel and these are the
  1679. 1:10:26tools that you have accessible to
  1680. 1:10:28yourself. And so I'm not hard coding the
  1681. 1:10:30steps. I'm just hard coding the tools
  1682. 1:10:31that you have access to for yourself. Uh
  1683. 1:10:34the more autonomous is the agent decides
  1684. 1:10:37the steps and can create the tools. So
  1685. 1:10:40that's where you might give actually
  1686. 1:10:42access to a code editor to the agent.
  1687. 1:10:44And the agent might actually be able to
  1688. 1:10:47ping any API in the web, perform some
  1689. 1:10:49web search. It might even be able to
  1690. 1:10:51create some code to display data to the
  1691. 1:10:54user. It might even be able to perform
  1692. 1:10:56some calculations like, oh, I'm going to
  1693. 1:10:58calculate the fastest route to get from
  1694. 1:11:01San Francisco to New York. Um, and which
  1695. 1:11:04one might be the most appropriate for
  1696. 1:11:06what the user is looking for. And then I
  1697. 1:11:08want to calculate the distance between
  1698. 1:11:10the airport and that hotel versus that
  1699. 1:11:12hotel. And I'm going to write code to do
  1700. 1:11:13that. So it's actually fully autonomous
  1701. 1:11:16from that perspective.
  1702. 1:11:20Okay.
  1703. 1:11:22So yeah, remember those keywords,
  1704. 1:11:24memory, uh, prompts, tools, etc.
  1705. 1:11:30Now I presented the flight API, but it
  1706. 1:11:33does not have to be an API. You probably
  1707. 1:11:35have heard the term MCP or model context
  1708. 1:11:38protocol that was coined by entropic. I
  1709. 1:11:40I pasted the seminal article on MCP at
  1710. 1:11:44the bottom of this slide. But let me
  1711. 1:11:45explain in a nutshell why those things
  1712. 1:11:47would differ. Um uh in in the API case
  1713. 1:11:52um you would actually teach your LLM to
  1714. 1:11:57ping an API. So you would say this is
  1715. 1:11:59how you ping this API and this is the
  1716. 1:12:01data that it will send you back and um
  1717. 1:12:04you would have to do that in a one-off
  1718. 1:12:06manner. So you would have to build or
  1719. 1:12:08sort of give the API documentation of
  1720. 1:12:10your flight API, your booking hotel API,
  1721. 1:12:14your car rental API and then you would
  1722. 1:12:17give tools for your model to communicate
  1723. 1:12:19with those APIs. Um
  1724. 1:12:22it doesn't scale very well um you know
  1725. 1:12:25versus MCP. MCP um it's really about you
  1726. 1:12:31know putting a system in the middle sort
  1727. 1:12:34of that would make it simpler for your
  1728. 1:12:36LLM to communicate with that endpoint.
  1729. 1:12:39So for instance um you might you know
  1730. 1:12:42have an MCP server and MCP client where
  1731. 1:12:44you're trying to communicate with that
  1732. 1:12:46travel uh database or the flight API or
  1733. 1:12:50MCP and uh your agent might actually
  1734. 1:12:53just communicate with it and say hey
  1735. 1:12:55what do you need in order to give me
  1736. 1:12:57more flight information and that that
  1737. 1:12:59agent will respond by I would like you
  1738. 1:13:02to tell me where is the origin flight
  1739. 1:13:04where is the destination and what you're
  1740. 1:13:05looking for at a high level this is my
  1741. 1:13:07requirement okay let me get back you
  1742. 1:13:09with my requirement. Oh, you forgot to
  1743. 1:13:12tell me your budget, whatever. Oh, let
  1744. 1:13:14me give you my budget, etc. Um, and uh
  1745. 1:13:17it's it's it's agent to agent
  1746. 1:13:19communication which allows more
  1747. 1:13:21scalability. You don't need to hardcode
  1748. 1:13:24everything. Companies have displayed
  1749. 1:13:26their MCPS out there and you can your
  1750. 1:13:28agent can communicate with them and
  1751. 1:13:30figure out how to get the data it needs.
  1752. 1:13:32Does that make sense?
  1753. 1:13:34>> Yeah.
  1754. 1:13:42API
  1755. 1:13:44that
  1756. 1:13:47>> yes is not a strict issue.
  1757. 1:13:51Uh I think it it is ultimately the
  1758. 1:13:55question is is it achiefing issue
  1759. 1:13:56because anyway if an API has to be
  1760. 1:13:58updated the MCP has to be updated is
  1761. 1:14:00what you say right? Yes that that's
  1762. 1:14:02correct. But at least um it allows the
  1763. 1:14:04agent to sort of go back and forth and
  1764. 1:14:06figure out what the requirements are.
  1765. 1:14:08But at the end of the day ideally if
  1766. 1:14:11you're a startup you have some
  1767. 1:14:12documentation and automatically have an
  1768. 1:14:14agent or an LLM workflow that reads that
  1769. 1:14:17documentation and updates the code
  1770. 1:14:18accordingly. Yeah, but I agree it's not
  1771. 1:14:21it's not something that is fully
  1772. 1:14:22autonomous. Yeah. Yeah.
  1773. 1:14:28>> Why is that?
  1774. 1:14:30>> Which security specifically?
  1775. 1:14:34>> Yeah. So, are there security issues with
  1776. 1:14:37MCPS?
  1777. 1:14:38So, think about it this way. Uh MCPS
  1778. 1:14:42depending on the data that you get
  1779. 1:14:44access to might have different
  1780. 1:14:45requirements. Lower stake or higher
  1781. 1:14:47stake. I'm not an expert at, you know,
  1782. 1:14:48the full range, but it wouldn't surprise
  1783. 1:14:51me that um, you know, when you when you
  1784. 1:14:54when you expose an MCP to an I think you
  1785. 1:14:58would a lot of MCPs have authentication.
  1786. 1:15:01So, you know, you might actually need a
  1787. 1:15:03code to actually talk to it just like
  1788. 1:15:04you would with an API or a key. Um,
  1789. 1:15:07yeah, but that's a good question. I'm,
  1790. 1:15:09you know, I'm not an expert at the
  1791. 1:15:11security of these systems, but you know,
  1792. 1:15:13we can look into it.
  1793. 1:15:17U any other questions on what we've seen
  1794. 1:15:20with the agentic workflows, APIs, tools,
  1795. 1:15:24MCPS, memory, all of that is under
  1796. 1:15:27progress. So even memory is not a solved
  1797. 1:15:28problem by any mean. It's pretty hard
  1798. 1:15:30actually to get done. Yes,
  1799. 1:15:33>> you do need an MCP,
  1800. 1:15:37right? You can have the FCP just makes
  1801. 1:15:40it easier to access the API, but
  1802. 1:15:42technically
  1803. 1:15:44looking uh your way to achieving from
  1804. 1:15:48the API,
  1805. 1:15:50right?
  1806. 1:15:56>> Exactly. Exactly. Yeah. Is is MCP about
  1807. 1:15:59efficiency or accessing more data? It's
  1808. 1:16:01about efficiency. It's like, you know,
  1809. 1:16:03let's say you have a coding agent um and
  1810. 1:16:07you know, it has an MCP client and
  1811. 1:16:09there's multiple MCP servers that are
  1812. 1:16:11exposed out there. Um that agent can
  1813. 1:16:14communicate very efficiently with them
  1814. 1:16:16and find what it needs. um uh and it's
  1815. 1:16:19it's a more efficient process and
  1816. 1:16:21actually uh displaying APIs and the APIs
  1817. 1:16:24on that side and how to ping them and
  1818. 1:16:26what the protocol you know but it's not
  1819. 1:16:28about the data that is being exposed
  1820. 1:16:30because ultimately you control the data
  1821. 1:16:31that is being exposed
  1822. 1:16:34you probably you know depending on how
  1823. 1:16:36the MCP is built my guess is you
  1824. 1:16:38probably expose yourself to other risks
  1825. 1:16:40because your your MCP server can can see
  1826. 1:16:45any input pretty much from another LLM
  1827. 1:16:48and so it has to be robust. Um
  1828. 1:16:51but yeah, super. Uh so let let's look at
  1829. 1:16:55an example of a step-by-step uh uh uh
  1830. 1:16:59workflow for the travel agent. So let's
  1831. 1:17:01say the user says I want to plan a plan
  1832. 1:17:04a trip um to Paris from December 15th to
  1833. 1:17:0820th um with flights hotels near the FL
  1834. 1:17:13tower and then an itinary of must visit
  1835. 1:17:16places that's the task to the travel
  1836. 1:17:18agent step two the agent plans the steps
  1837. 1:17:22so it says I'm going to find flights use
  1838. 1:17:24the flight search API uh to get option
  1839. 1:17:27for December 15th search hotels generate
  1840. 1:17:29recommendation for places to visit,
  1841. 1:17:31validate preferences,
  1842. 1:17:33um, budget, etc. Book the trip with the
  1843. 1:17:36payment processing API. Step three,
  1844. 1:17:39that's just the planning, by the way.
  1845. 1:17:41Step three, execute the plan. Use your
  1846. 1:17:43tools, combine the results, and then
  1847. 1:17:46proactive user interaction and booking.
  1848. 1:17:48It might make a first proposal to the
  1849. 1:17:51user and ask the user to validate or
  1850. 1:17:53invalidate and then may repeat uh that
  1851. 1:17:55planning and execution process. And then
  1852. 1:17:58finally, it might actually update the
  1853. 1:18:01memory. It might say, "Oh, I just
  1854. 1:18:03learned through this interaction that
  1855. 1:18:04the user only likes direct flights. Next
  1856. 1:18:07time I I'll only give direct flights."
  1857. 1:18:11Or, "I noticed users are fine with three
  1858. 1:18:14star hotels or fourstar hotels and in
  1859. 1:18:17fact they're they don't want to go above
  1860. 1:18:19budget or something like that."
  1861. 1:18:22Um, so that hopefully makes sense by now
  1862. 1:18:25on, you know, how you might do that. My
  1863. 1:18:27question for you is uh how would you
  1864. 1:18:30know if this works and if you had such a
  1865. 1:18:33system running in production, how would
  1866. 1:18:35you improve it?
  1867. 1:18:44>> Yeah.
  1868. 1:18:47>> So that's an example. So let users rate
  1869. 1:18:50their experience at the end. Uh that
  1870. 1:18:53would be an end to end test, right?
  1871. 1:18:55You're looking at the user experience
  1872. 1:18:57through the steps and say how good was
  1873. 1:18:59it from 1 to five, let's say. Yeah, it's
  1874. 1:19:02a good way. And then if you learn that a
  1875. 1:19:04user says one, what how do you improve
  1876. 1:19:07the the workflow?
  1877. 1:19:15>> Okay, so you would go down a tree and
  1878. 1:19:17say, okay, you said one uh what what was
  1879. 1:19:21your issue? and then the user says uh
  1880. 1:19:23the prices were too high, let's say, and
  1881. 1:19:26then you would go back and fix that
  1882. 1:19:27specific uh uh tool or prompt or Yeah.
  1883. 1:19:30Okay. Any other ideas?
  1884. 1:19:44Yeah, good. So, that's a good insight.
  1885. 1:19:46Separate the LLM related stuff from the
  1886. 1:19:48non LLM related stuff, the deterministic
  1887. 1:19:50stuff. The deterministic stuff, you
  1888. 1:19:52might be able to fix it, you know, more
  1889. 1:19:55objectively essentially. Yeah, there was
  1890. 1:19:59what else?
  1891. 1:20:12So give me an example of an objective
  1892. 1:20:16issue that you can notice and how you
  1893. 1:20:18would fix it versus a subjective issue.
  1894. 1:20:21>> Yeah.
  1895. 1:20:31>> Okay. So let's say you say there's the
  1896. 1:20:34same flight but one is cheaper than the
  1897. 1:20:35other. Let's say it's objectively worst.
  1898. 1:20:38And so you can capture that almost
  1899. 1:20:40automatically. Yeah. So you could
  1900. 1:20:42actually build evals that are objective
  1901. 1:20:45that are tracked across your users. And
  1902. 1:20:48you might actually run an analysis after
  1903. 1:20:50and see that for the objective stuff. We
  1904. 1:20:52notice that our LLM AI agentic AI
  1905. 1:20:56workflow is bad with pricing. It just
  1906. 1:20:59doesn't read price as well because it
  1907. 1:21:01always gives a more expensive option.
  1908. 1:21:03Yeah, you're perfectly right. How about
  1909. 1:21:05the subjective stuff?
  1910. 1:21:13Yeah.
  1911. 1:21:14>> Like do you choose a direct or indirect
  1912. 1:21:17flight if the indirect is a little bit
  1913. 1:21:19cheaper?
  1914. 1:21:20>> Yeah, good one. Do you do you choose a
  1915. 1:21:22direct flight or an indirect flight if
  1916. 1:21:24the indirect is cheaper but the direct
  1917. 1:21:26is more comfortable? Um yeah, that's a
  1918. 1:21:29good one actually. Um so how would you
  1919. 1:21:33capture that information? Let's say this
  1920. 1:21:34is used by thousands of users.
  1921. 1:21:39Um, could you feed something in about
  1922. 1:21:41us?
  1923. 1:21:43>> Uh, could you feed something in? Yeah, I
  1924. 1:21:46mean you could you could u could you
  1925. 1:21:48feed something in uh about the user
  1926. 1:21:50preferences? Well, you could you could
  1927. 1:21:52build a data set that has some of that
  1928. 1:21:55information. So, you build 10 prompts
  1929. 1:21:58where the user is asking specifically
  1930. 1:22:00for direct is saying that I prefer
  1931. 1:22:03direct flights because I care about my
  1932. 1:22:05time. let's say and then you look at the
  1933. 1:22:07output and you actually give a good the
  1934. 1:22:09example of a good output and you
  1935. 1:22:12probably are able to capture the
  1936. 1:22:14performance of your agentic workflow on
  1937. 1:22:17this specific eval whether does it
  1938. 1:22:20prioritize does it understand price
  1939. 1:22:22conscious is it price conscious
  1940. 1:22:23essentially and comfort conscious yeah
  1941. 1:22:28what about the tone let's say let's say
  1942. 1:22:31the LLM right now is not very friendly
  1943. 1:22:34how would
  1944. 1:22:35notice that and how would you fix it?
  1945. 1:22:41>> Yeah.
  1946. 1:22:42>> Test user and like run prompts and see
  1947. 1:22:46if there's something wrong with that.
  1948. 1:22:48>> Okay. Have a test user run the prompt
  1949. 1:22:50and see if there's something wrong with
  1950. 1:22:52that. Tell me about the last step. How
  1951. 1:22:53would you notice that something is
  1952. 1:22:55wrong?
  1953. 1:22:56So test
  1954. 1:23:05and see if it's satisfy.
  1955. 1:23:07>> Yeah, I agree with your approach. Have
  1956. 1:23:09LLM judges that evaluate the response
  1957. 1:23:11against a certain rubric of what
  1958. 1:23:12politeness look like. So here in this
  1959. 1:23:15case you could actually start uh with
  1960. 1:23:17error analysis. So you start you you
  1961. 1:23:19have a thousand users and you know you
  1962. 1:23:21can pull up 20 user interaction and read
  1963. 1:23:23through it and you might notice at first
  1964. 1:23:26sight the LLM seems to be very rude. You
  1965. 1:23:30know it's just super super short in its
  1966. 1:23:33answers and it's not very helpful. Um
  1967. 1:23:36you notice that with your air analysis
  1968. 1:23:38manually. Then you go to the next stage.
  1969. 1:23:40You actually put evals behind it. you
  1970. 1:23:42say, "I'm going to create a set of um a
  1971. 1:23:46set of LM judges that are going to look
  1972. 1:23:49at the user interaction and are going to
  1973. 1:23:52rate how polite it is and I'm going to
  1974. 1:23:54give it a rubric. Then what I'm going to
  1975. 1:23:57do is I'm going to flip my LLM. Instead
  1976. 1:23:58of using GPT4, I'm going to use Grock
  1977. 1:24:01and instead of using Grock, I'm going to
  1978. 1:24:02use Lama. And then I'm going to run
  1979. 1:24:04those three LLM side by side, give it to
  1980. 1:24:07my LLM judges, and then get my
  1981. 1:24:10subjective score at the end to say, "Oh,
  1982. 1:24:14X model was more polite on average."
  1983. 1:24:18Yeah, perfectly right. That's an example
  1984. 1:24:19of an eval that is very specific and
  1985. 1:24:21allows you to choose between LM. You
  1986. 1:24:23could actually do the same eval across
  1987. 1:24:25LM, but fix the LLM, change the prompt.
  1988. 1:24:28You actually instead of saying act like
  1989. 1:24:30a travel agent, you say act like a
  1990. 1:24:32helpful travel agent and then you see
  1991. 1:24:34the influence of that word on your eval
  1992. 1:24:36with the LLM as judges. Does that make
  1993. 1:24:38sense? Okay.
  1994. 1:24:41Uh super. So let's let's move forward
  1995. 1:24:43and do a case study with eval and then
  1996. 1:24:45we're we're almost done um for today. Uh
  1997. 1:24:49let's say your product managers uh
  1998. 1:24:51manager asks you to build an AI agent
  1999. 1:24:54for customer support.
  2000. 1:24:56Okay, where do you start? And here is an
  2001. 1:24:59example of the user prompt. I need to
  2002. 1:25:01change my shipping address for order
  2003. 1:25:03blah blah blah. I move to a new address.
  2004. 1:25:07So what what do you start if I'm giving
  2005. 1:25:09you that project, you know?
  2006. 1:25:20>> Yes.
  2007. 1:25:21research online,
  2008. 1:25:31>> right? So, do some research, see
  2009. 1:25:33benchmarks and how different models
  2010. 1:25:36perform at customer support and then
  2011. 1:25:37pick a model. That's what you mean?
  2012. 1:25:39Yeah, you it's true. You could do that.
  2013. 1:25:41What What else could you do? Yeah.
  2014. 1:25:50>> Okay. Yeah, I like that. Try to
  2015. 1:25:53decompose the different tasks that it
  2016. 1:25:54will need and try to guess which ones
  2017. 1:25:57will be more of a struggle, which ones
  2018. 1:25:58should be fuzzy, which one should be
  2019. 1:25:59deterministic. Yeah, you're right.
  2020. 1:26:11>> Yeah, similar to what you said. That's
  2021. 1:26:14what I would recommend as well. You say
  2022. 1:26:16I would sit down with a customer support
  2023. 1:26:18agent for a day or two and I would
  2024. 1:26:19decompose the task they're going
  2025. 1:26:21through. I will ask them where do they
  2026. 1:26:22struggle, how much time it takes. Yes,
  2027. 1:26:24that's usually where you want to start
  2028. 1:26:26with task decomposition. So let's say
  2029. 1:26:28we've done that work and we have this
  2030. 1:26:31list. I'm simplifying but the customer
  2031. 1:26:33support agent human typically would
  2032. 1:26:36extract key info then look up in the
  2033. 1:26:38database to retrieve the customer record
  2034. 1:26:41then check the policy you know are we
  2035. 1:26:44allowed to update the address or is it a
  2036. 1:26:46fixed data point um and then draft the
  2037. 1:26:48response email and sends the email okay
  2038. 1:26:51so we've decomposed that task
  2039. 1:26:54u once you've decomposed that task uh
  2040. 1:26:58how do you design your agentic workflow
  2041. 1:27:19Yes.
  2042. 1:27:23>> Which one?
  2043. 1:27:25Which one? We're going to use a method
  2044. 1:27:28or whatever method in each task what are
  2045. 1:27:30you going to use for
  2046. 1:27:33>> exactly so to repeat I'm going to you're
  2047. 1:27:35going to look at the de composition of
  2048. 1:27:37tasks
  2049. 1:27:38get an instinct of what's fuzzy what's
  2050. 1:27:40deterministic and then determine which
  2051. 1:27:44line is going to be an LLM one shot
  2052. 1:27:47which one will require maybe a rag which
  2053. 1:27:50one will require a tool which one will
  2054. 1:27:52require memory which one so you will
  2055. 1:27:54start designing that map completely
  2056. 1:27:57right that's also what I would recommend
  2057. 1:27:58you you might actually uh draft it and
  2058. 1:28:01say okay I take the user prompt um and
  2059. 1:28:04the first step of my task de composition
  2060. 1:28:08was extract information that seems to be
  2061. 1:28:11a vanilla LLM you you you can guess that
  2062. 1:28:14the vanilla lm would probably be good
  2063. 1:28:16enough at extracting the user wants to
  2064. 1:28:19change their address and this is the
  2065. 1:28:20order number and this is the new address
  2066. 1:28:22you probably don't need too much
  2067. 1:28:24technology there other than the LLM. Um,
  2068. 1:28:27the next step, it feels like you need a
  2069. 1:28:29tool because you're actually going to
  2070. 1:28:31have to look up in the database and also
  2071. 1:28:34update the address.
  2072. 1:28:37So, that might be a tool and you might
  2073. 1:28:38have to build a custom tool for the LLM
  2074. 1:28:40to say, let me connect you to that
  2075. 1:28:42database or let me give you access to
  2076. 1:28:44that resource with an MCP. You know,
  2077. 1:28:48after that, you probably need an LLM
  2078. 1:28:50again to draft the email, but you would
  2079. 1:28:52probably paste confirmation. you paste a
  2080. 1:28:54confirmation that your address has been
  2081. 1:28:56updated from X to Y and then the LLM
  2082. 1:28:59will draft an answer. And of course,
  2083. 1:29:00just to not forget, you might need a
  2084. 1:29:02tool to send the email. You know, you
  2085. 1:29:05might actually need to um you know, post
  2086. 1:29:09something to for the email to to go out
  2087. 1:29:12and then you'll get the output. Does
  2088. 1:29:14that make sense? So, exactly what you
  2089. 1:29:16described.
  2090. 1:29:17Okay. Now, moving to the next step. Once
  2091. 1:29:19we have de composed our tasks, then we
  2092. 1:29:22have designed an agentic workflow around
  2093. 1:29:24it. It took us five minutes. In
  2094. 1:29:26practice, it would take you more if
  2095. 1:29:27you're building your startup on that.
  2096. 1:29:29You want to make sure your task
  2097. 1:29:30composition is accurate. Your thing is
  2098. 1:29:31accurate here. And then you can have a
  2099. 1:29:34lot of work done on every tool and
  2100. 1:29:36optimize it and latency and cost. But
  2101. 1:29:38let's say now we want to know how um uh
  2102. 1:29:41if it works, you know, and I'm going to
  2103. 1:29:44assume that you have LLM traces. LLM
  2104. 1:29:47traces are very important. Actually, if
  2105. 1:29:49you're interviewing with an AI startup,
  2106. 1:29:51I would recommend you in the interview
  2107. 1:29:53process to ask them, do you have LLM
  2108. 1:29:56traces? Because if they don't have LM
  2109. 1:29:57traces, it is pretty hard to debug an
  2110. 1:30:00LLM system. you know, because you don't
  2111. 1:30:02have visibility on the chain of complex
  2112. 1:30:04prompts that were called and where the
  2113. 1:30:06bug is and you know, so it's a basic
  2114. 1:30:10sort of part of an AI startup stack to
  2115. 1:30:13have LLM traces.
  2116. 1:30:16So let's assume you have traces. How
  2117. 1:30:18would you know if your system work? You
  2118. 1:30:20know, we I I you know, I I'm going to
  2119. 1:30:23summarize some of the things I heard
  2120. 1:30:25earlier. Um you gave us an example of an
  2121. 1:30:29end to end metric. You look at the user
  2122. 1:30:32satisfaction at the end. Um you can also
  2123. 1:30:34do a component-based approach where you
  2124. 1:30:37actually will look at the tool the
  2125. 1:30:39database updates and you will manually
  2126. 1:30:42do an error analysis and see oh the tool
  2127. 1:30:45actually always forgets to update the
  2128. 1:30:47email. It just fails at writing you know
  2129. 1:30:49and I'm going to fix that. This is
  2130. 1:30:50deterministic pretty much. um or um you
  2131. 1:30:54know when it tries to send the email and
  2132. 1:30:57ping the system that is supposed to send
  2133. 1:30:59the email, it doesn't send it in the
  2134. 1:31:01right format and so it bugs at that
  2135. 1:31:04point. Again, you could fix that. Um
  2136. 1:31:07draft of the email, the LM doesn't do a
  2137. 1:31:09great job. It's not very polite at
  2138. 1:31:10drafting the email, you know. So, you
  2139. 1:31:12could look at component by component and
  2140. 1:31:15it's actually easier to debug than to
  2141. 1:31:16look at it end to end. You would
  2142. 1:31:18probably do a mix of both.
  2143. 1:31:20Um, another way to look at it is what is
  2144. 1:31:23objective versus what is subjective. So,
  2145. 1:31:26for example, an objective example would
  2146. 1:31:28be the LLM um extracted the wrong order
  2147. 1:31:33ID. You know, the the user said my order
  2148. 1:31:35ID is X and the LLM when it's actually
  2149. 1:31:38pasted looped up in the database, it
  2150. 1:31:40used the wrong order ID. This is
  2151. 1:31:42objectively wrong. you can actually
  2152. 1:31:44write a Python code that checks that
  2153. 1:31:46checks just the alignment between what
  2154. 1:31:47the user mentioned and and what was
  2155. 1:31:49actually pasted in the database or for
  2156. 1:31:52the lookup. You also have subjective
  2157. 1:31:54stuff which we talked about where you
  2158. 1:31:56probably want to do either human rating
  2159. 1:31:58or LLM as judges. It's very relevant for
  2160. 1:32:01subjective evals.
  2161. 1:32:04Um and finally you will find yourself
  2162. 1:32:07having quantitative evals and more
  2163. 1:32:10qualitative evals. So quantitative would
  2164. 1:32:12be percentage of successful address
  2165. 1:32:15updates. The latency you could actually
  2166. 1:32:17track the latency component based and
  2167. 1:32:19see which one is the slowest. Let's say
  2168. 1:32:21sending the email is 5 second you know
  2169. 1:32:24it's too long let's say you would notice
  2170. 1:32:26component based or the full workflow and
  2171. 1:32:28then you will decide where am I
  2172. 1:32:30optimizing my latency and how am I going
  2173. 1:32:32to do that and then finally qualitative
  2174. 1:32:35uh you might actually do some error
  2175. 1:32:37analysis and look at you know where are
  2176. 1:32:41the hallucinations
  2177. 1:32:43um where are the tone mismatches
  2178. 1:32:46uh you know are the user confused and by
  2179. 1:32:49what they're confused you know that
  2180. 1:32:50would more qualitative and typically it
  2181. 1:32:52would take more um you know white glove
  2182. 1:32:56approaches to do that. Okay, so here's
  2183. 1:32:59what it could look like. I gave you some
  2184. 1:33:01examples but you would build evals to
  2185. 1:33:05determine objectively, subjectively,
  2186. 1:33:07component based, end to end based and
  2187. 1:33:09then quantitatively and qualitatively
  2188. 1:33:11where is your LLM failing and where it's
  2189. 1:33:14doing well.
  2190. 1:33:18Does that give you a sense of the type
  2191. 1:33:19of stuff you could do to fix improve
  2192. 1:33:22that agentic workflow?
  2193. 1:33:25Super. Well, that was our case study on
  2194. 1:33:27EVA. We're not going to del deeper into
  2195. 1:33:29it, but hopefully it gave you a sense of
  2196. 1:33:31the type of stuff you can do with LM
  2197. 1:33:33judges with, you know,
  2198. 1:33:36objective, subjective, component based,
  2199. 1:33:38end to end, etc. U last section on
  2200. 1:33:42multi- aent workflows.
  2201. 1:33:44So you might you might ask uh hey uh why
  2202. 1:33:48do we need a multi- aent workflow when
  2203. 1:33:50we when the workflow already has
  2204. 1:33:53multiple steps already calls the LLM
  2205. 1:33:55multiple times already gives them tools
  2206. 1:33:58why do we need multiple agents and so
  2207. 1:34:01many people are talking about multi-
  2208. 1:34:02aent system online it's not even a new
  2209. 1:34:04thing frankly I mean multi- aent system
  2210. 1:34:05have been around for a long time the the
  2211. 1:34:08main advantage of a multi- aent system
  2212. 1:34:10is going to be parallelism it's like is
  2213. 1:34:13there something that I wish I would run
  2214. 1:34:17in parallel
  2215. 1:34:19sort of independently but maybe there
  2216. 1:34:21are some syncs in the middle but that's
  2217. 1:34:23where you want to put a multi- aent
  2218. 1:34:25system it's when it's parallel the other
  2219. 1:34:28advantage that some companies um have
  2220. 1:34:30with multi- aent system is an agent can
  2221. 1:34:33be reused. So let's say in a company you
  2222. 1:34:36have an agent that's been built for
  2223. 1:34:37design. That agent can be used in the
  2224. 1:34:40marketing team and it can be used in the
  2225. 1:34:42product team, you know, and so now
  2226. 1:34:44you're optimizing an agent which has
  2227. 1:34:46multiple stakeholders that can
  2228. 1:34:48communicate with it and benefit from its
  2229. 1:34:50uh performance.
  2230. 1:34:53Um actually I'm going to ask you a
  2231. 1:34:55question and take a few uh maybe a
  2232. 1:34:57minute to think about it. Let's say you
  2233. 1:34:59were uh building smart home automation
  2234. 1:35:02for your apartment or your home. What
  2235. 1:35:06agents would you want to build? Yeah,
  2236. 1:35:09write it down. And then I'm going to ask
  2237. 1:35:11you in in a minute to share some of the
  2238. 1:35:14agents that you will build. Also, think
  2239. 1:35:17about how you would put a hierarchy
  2240. 1:35:19between these agents or how you would
  2241. 1:35:20organize them or who should communicate
  2242. 1:35:22with who. Okay. Okay. Take a minute for
  2243. 1:35:25that.
  2244. 1:35:27be creative also because I'm gonna ask
  2245. 1:35:29all of your agents and maybe you have an
  2246. 1:35:31agent that nobody has thought of.
  2247. 1:35:37Okay, let's get started. Who wants to
  2248. 1:35:40give me a a set of agents that you would
  2249. 1:35:43want for your home smart home? Yes.
  2250. 1:35:47So u the first is like set of agents
  2251. 1:35:50that drive my movements in the house and
  2252. 1:35:52drop information about my house.
  2253. 1:35:55Another agent receive that information
  2254. 1:35:58and adjust the room temperature
  2255. 1:36:03and another
  2256. 1:36:16Okay. So, let me repeat. You have four
  2257. 1:36:18agents. I think roughly one that tracks
  2258. 1:36:22biometric like you're where are you in
  2259. 1:36:24the home, where are you moving, how
  2260. 1:36:26you're moving, things like that. That
  2261. 1:36:28sort of knows your location. The second
  2262. 1:36:31one um determines the temperature
  2263. 1:36:35of the rooms and has the ability to
  2264. 1:36:37change it. The third one tracks energy
  2265. 1:36:41efficiency and might be feedback on
  2266. 1:36:44energy and energy usage and might be I
  2267. 1:36:47don't know maybe it has the control over
  2268. 1:36:49the temperature as well I don't know
  2269. 1:36:51actually or the gas uh or the water
  2270. 1:36:55might cut your water at some point the
  2271. 1:36:58and then you have an orchestrator agent
  2272. 1:37:00what is exactly the orchestrator doing
  2273. 1:37:08Okay. Passes instructions. So is that
  2274. 1:37:11the agent that communicates mainly with
  2275. 1:37:14the user?
  2276. 1:37:15>> Y.
  2277. 1:37:15>> Okay. So if I have I'm coming back home
  2278. 1:37:17and I'm saying I want the oven to be
  2279. 1:37:20preheated. I communicate with the
  2280. 1:37:22orchestrator and then it would funnel to
  2281. 1:37:24another agent. Okay. Sounds good. Yeah.
  2282. 1:37:27So that's an example of a I want to say
  2283. 1:37:29a hierarchical um agency multi- aent
  2284. 1:37:32system.
  2285. 1:37:35Um what else? Any other ideas? What what
  2286. 1:37:38would you add to that?
  2287. 1:37:39>> Yeah,
  2288. 1:37:40>> like kind of identification where it's
  2289. 1:37:43not true and what is it's true.
  2290. 1:37:45>> You can have like depending on who it
  2291. 1:37:47is, you have like access to the like
  2292. 1:37:50minimal action that you can do. Imagine
  2293. 1:37:52entering a room or just entering a
  2294. 1:37:54computer or just opening the minimum
  2295. 1:37:57kind of action like lot of agent per
  2296. 1:38:07who is it.
  2297. 1:38:10>> Oh, I like that. That's a really good
  2298. 1:38:12one. So, let me summarize. You have a
  2299. 1:38:15security agent that determines if you
  2300. 1:38:17can enter or not. And when you enter, it
  2301. 1:38:20understands who you are. And then it
  2302. 1:38:22gives you certain sets of permissions
  2303. 1:38:25that might be different depending of if
  2304. 1:38:27you're a parent or a kid or you know you
  2305. 1:38:30might have access to certain cars and
  2306. 1:38:32not others or the kid cannot open the
  2307. 1:38:34fridge or I don't know like something
  2308. 1:38:36like that. Yeah. Or okay. I like that.
  2309. 1:38:38That's a good one. Yeah. And it does
  2310. 1:38:40feel like it's a complex enough workflow
  2311. 1:38:43where you want a specific workflow tied
  2312. 1:38:47to that. I I agree.
  2313. 1:38:50What What else?
  2314. 1:38:55Yes. Continuing on
  2315. 1:39:00energy savings
  2316. 1:39:08open house.
  2317. 1:39:16Um, another as well.
  2318. 1:39:28>> Well, that's really good actually. So,
  2319. 1:39:30you mentioned two of them. One is maybe
  2320. 1:39:33an agent that has access to external
  2321. 1:39:35APIs that can understand the weather out
  2322. 1:39:37there, the wind, the sun, and then has
  2323. 1:39:40control over certain devices at home,
  2324. 1:39:44temperature, blinds, things like that,
  2325. 1:39:46and also understand your preferences for
  2326. 1:39:48it. That does feel like it's a good use
  2327. 1:39:50case because you could give that to the
  2328. 1:39:52orchestrator, but it might lose itself
  2329. 1:39:54because it's doing too much. So you
  2330. 1:39:56probably and also these problems are
  2331. 1:39:57tied together like temperature outdoor
  2332. 1:40:00with the weather API might influence the
  2333. 1:40:02temperature inside how you want it etc.
  2334. 1:40:05And then the second one which I also
  2335. 1:40:08like is you might have an agent that
  2336. 1:40:10looks at your fridge and what's inside
  2337. 1:40:12and it might actually have access to the
  2338. 1:40:14camera in the fridge for example um and
  2339. 1:40:17know your preferences and also has
  2340. 1:40:19access to the e-commerce API to order
  2341. 1:40:22Amazon groceries ahead of time. Um, I
  2342. 1:40:25agree. And maybe the orchestrator would
  2343. 1:40:27be the communication line with the user,
  2344. 1:40:29but it might communicate with that agent
  2345. 1:40:31um in order to get it done. Uh, yeah, I
  2346. 1:40:34like those. So, those are all really
  2347. 1:40:36good examples here. Here is the list I
  2348. 1:40:38had um up there. So, climate control,
  2349. 1:40:42lighting, security, energy management,
  2350. 1:40:45entertainment, notification agent,
  2351. 1:40:47alerts about the system updates, energy
  2352. 1:40:49saving and orchestrator. So, all of them
  2353. 1:40:51you mentioned actually. Um, and then we
  2354. 1:40:54didn't talk about the different
  2355. 1:40:56interaction patterns, but you do have
  2356. 1:40:59different ways to organize a multi- aent
  2357. 1:41:01system, flat, hierarchical. It sounds
  2358. 1:41:04like this would be hierarchical. I
  2359. 1:41:07agree. And the reason is UIUX is I would
  2360. 1:41:10rather have to only talk to the
  2361. 1:41:12orchestrator rather than have to go to a
  2362. 1:41:14specialized application to do something
  2363. 1:41:17like it feels like the orchestrator
  2364. 1:41:18could be responsible for that. And so I
  2365. 1:41:20agree. I would probably go for a
  2366. 1:41:21hierarchical setup here, but maybe you
  2367. 1:41:24might act also add some connections
  2368. 1:41:27between other agents like in the flat
  2369. 1:41:29system where it's all to all for example
  2370. 1:41:32uh with climate control and energy. If
  2371. 1:41:33you want to connect those two, you might
  2372. 1:41:35actually allow them to speak with each
  2373. 1:41:36other. When you allow agents to speak
  2374. 1:41:39with each other, it is basically an MCB
  2375. 1:41:41protocol by the way. So you treat the
  2376. 1:41:43agent like a tool, exactly like a tool.
  2377. 1:41:46Here is how you interact with this
  2378. 1:41:47agent. Here is what it can tell you.
  2379. 1:41:49here is what it needs from you
  2380. 1:41:51essentially.
  2381. 1:41:53Okay, super. And then without going into
  2382. 1:41:55the details, there are advantages to
  2383. 1:41:58multi- aent workflows versus, you know,
  2384. 1:42:00single agent such as debugging. It's
  2385. 1:42:03easier to special debug a specialized
  2386. 1:42:05agent and to be debug an entire system.
  2387. 1:42:08Parallelization as well. It's easier to
  2388. 1:42:10have things run in parallel um and you
  2389. 1:42:13can earn time. um you know there are
  2390. 1:42:15some advantages to doing that and I
  2391. 1:42:17leave you with this slide if you want to
  2392. 1:42:18go deeper. Super. So we've learned so
  2393. 1:42:22many techniques to optimize LLMs uh from
  2394. 1:42:24prompts to chains to finetuning
  2395. 1:42:26retrieval um and to multi- aent system
  2396. 1:42:29as well. And then just to end on um a
  2397. 1:42:32couple of trends I want you to watch. Uh
  2398. 1:42:35I think next week is Thanksgiving. Is
  2399. 1:42:36that it? Is Thanksgiving break? No, the
  2400. 1:42:38week after. Okay. Well, ahead of the
  2401. 1:42:40Thanksgiving break. So if you're
  2402. 1:42:42traveling you can think about these
  2403. 1:42:43things. Um what's next is in AI. I
  2404. 1:42:47wanted to call out a couple of trends.
  2405. 1:42:49So Elas discover one of the OGs of uh
  2406. 1:42:53you know LLM's um and you know OpenAI
  2407. 1:42:56co-founder um raised that question about
  2408. 1:42:59are we plateauing or not? You know the
  2409. 1:43:01question of are we going to see in the
  2410. 1:43:04coming years LLM sort of not improve as
  2411. 1:43:07fast as we've seen in the past. It's
  2412. 1:43:10been the feeling in the community
  2413. 1:43:12probably that you know the last version
  2414. 1:43:14of GPT um did not bring the level of
  2415. 1:43:18performance that people were expecting
  2416. 1:43:21although it did make it so much easier
  2417. 1:43:23to use for consumers because you don't
  2418. 1:43:25need to interact with different models.
  2419. 1:43:26It's all under the same hood. So it
  2420. 1:43:28seems that it's progressing. Um but the
  2421. 1:43:31plateau is is unclear. The the way I
  2422. 1:43:33would think about it is um
  2423. 1:43:36the LLM scaling laws tell us that if we
  2424. 1:43:39continue to improve compute and energy
  2425. 1:43:42then LLM should continue to improve but
  2426. 1:43:44at some point it's going to plateau. So
  2427. 1:43:45what's going to take us to the next
  2428. 1:43:46step? Um it's probably architecture
  2429. 1:43:49search. Still a lot of LLM even if we
  2430. 1:43:52don't understand what's under the hood
  2431. 1:43:53are probably transformerbased today. But
  2432. 1:43:56we know that the human brain does not
  2433. 1:43:58operate the same way. There's just
  2434. 1:43:59certain things that we do that are much
  2435. 1:44:01more efficient, much faster. We don't
  2436. 1:44:03need as much data. So theoretically, we
  2437. 1:44:06have so much to learn in terms of
  2438. 1:44:07architecture search that we haven't
  2439. 1:44:09figured out. It's not a surprise that
  2440. 1:44:11you see those labs hire so many
  2441. 1:44:13engineers. Because it is possible that
  2442. 1:44:16in the next few years, you're going to
  2443. 1:44:17have thousands of engineers trying to
  2444. 1:44:19figure out the different engineering
  2445. 1:44:20hacks and tactics and architectural
  2446. 1:44:22searches that are going to lead to
  2447. 1:44:24better uh models. and one of them
  2448. 1:44:26suddenly will find the next transformer
  2449. 1:44:28and it will reduce by 10x the need for
  2450. 1:44:31compute and the need for energy. Um you
  2451. 1:44:34know it's sort of if you've read Isak
  2452. 1:44:36Azimoff's
  2453. 1:44:38foundation series individuals can have
  2454. 1:44:41an amazing impact on the future because
  2455. 1:44:43of their decisions. You know whoever
  2456. 1:44:46discovered transformers had a tremendous
  2457. 1:44:48impact on the direction of AI. I think
  2458. 1:44:51we're going to see more of that in the
  2459. 1:44:52coming years where some group of
  2460. 1:44:54researcher that is iterating fast might
  2461. 1:44:56discover certain things that would
  2462. 1:44:58suddenly unlock that plateau and take us
  2463. 1:45:00to the next step and it's going to
  2464. 1:45:01continue to improve like that and so it
  2465. 1:45:03doesn't surprise me that there's so many
  2466. 1:45:05companies hiring engineers right now to
  2467. 1:45:07figure out those hacks and those those
  2468. 1:45:09techniques. Um the other set of gains
  2469. 1:45:13that we might see is from multimodality.
  2470. 1:45:15So the way to think about it is we've
  2471. 1:45:17we've we've had LLMs first textbased and
  2472. 1:45:20then we've added imaging and today you
  2473. 1:45:23know models are very good at images
  2474. 1:45:25they're very good at text turns out that
  2475. 1:45:27being good at images and being good at
  2476. 1:45:29text makes the whole model better. So
  2477. 1:45:31the fact that you're good at
  2478. 1:45:32understanding a cat image makes you
  2479. 1:45:34better at text as well for a cat. Now
  2480. 1:45:37you add another modality like audio or
  2481. 1:45:39video the whole system gets better. So
  2482. 1:45:41you're better at writing about a cat if
  2483. 1:45:44you know what a cat sounds like. If you
  2484. 1:45:46can look at a cat on an image as well.
  2485. 1:45:47Does that make sense? So we see gains
  2486. 1:45:49that are translated from one modality to
  2487. 1:45:51another. And that might lead in the
  2488. 1:45:53pinnacle of robotics where all these
  2489. 1:45:54modalities come together and suddenly
  2490. 1:45:56the robot is better at running away from
  2491. 1:45:58a cat because it understands what a cat
  2492. 1:46:01is, how it sounds like, what it looks
  2493. 1:46:02like, etc. That make sense? Um the other
  2494. 1:46:05one is the multiple methods working in
  2495. 1:46:07harmony. In the Tuesday lectures, we've
  2496. 1:46:10seen supervised learning, unsupervised
  2497. 1:46:13learning, self-supervised learning,
  2498. 1:46:14reinforcement learning, proxy
  2499. 1:46:16engineering, rags, etc. If you look at
  2500. 1:46:18um how babies learn, um it is probably a
  2501. 1:46:23mix of those different approaches like a
  2502. 1:46:25baby um might have some metalarning,
  2503. 1:46:28meaning you know it has some survival
  2504. 1:46:30instinct that is in the encoded in the
  2505. 1:46:33DNA most likely. Um, and that's like the
  2506. 1:46:36baby's pre-training, if you will. On top
  2507. 1:46:38of that, uh, the mom or the dad, um, is
  2508. 1:46:42pointing at stuff and saying, "Bad,
  2509. 1:46:44good, bad, good, supervised learning."
  2510. 1:46:46On top of that, the baby's falling on
  2511. 1:46:48the ground and getting hurt and that's a
  2512. 1:46:50reward signal for reinforcement
  2513. 1:46:51learning. On top of that, the baby's
  2514. 1:46:54observing other people doing stuff or
  2515. 1:46:55other babies, you know, doing stuff,
  2516. 1:46:58unsupervised learning. You see what I
  2517. 1:46:59mean? It's we're probably a mix of all
  2518. 1:47:01these methods. And um and I think that's
  2519. 1:47:04where the trend is going is where those
  2520. 1:47:06methods that you've seen in CS230 come
  2521. 1:47:08together in order to build an AI system
  2522. 1:47:11that learns fast, is low latency, is
  2523. 1:47:14cheap, energy efficient, and makes the
  2524. 1:47:16most out of all of these methods. Um
  2525. 1:47:19finally, and this is especially true at
  2526. 1:47:20Stanford, um you have research going on
  2527. 1:47:24that you would consider human centric
  2528. 1:47:26and some research that is nonhuman
  2529. 1:47:28ccentric. by humanentric, I should say
  2530. 1:47:30human approaches that are modeled after
  2531. 1:47:33the brain and approaches that are not
  2532. 1:47:35modeled after humans because it turns
  2533. 1:47:36out that the human body is very
  2534. 1:47:39limiting. And so if you actually only do
  2535. 1:47:42research on what the human brain looks
  2536. 1:47:43like, you're probably missing out on
  2537. 1:47:45compute and energy and stuff like that
  2538. 1:47:46that you can optimize even beyond
  2539. 1:47:48neuronal connections in the brain. But
  2540. 1:47:50you still can learn a lot from the human
  2541. 1:47:52brain. And that's why there are
  2542. 1:47:54professors that are running labs right
  2543. 1:47:56now that try to understand how does back
  2544. 1:47:58propagation work for humans. And in
  2545. 1:48:01fact, it's probably that we don't have
  2546. 1:48:03back propagation. We don't use back
  2547. 1:48:04propagation. We only do forward
  2548. 1:48:06propagation, let's say. So this type of
  2549. 1:48:08stuff is interesting research that I
  2550. 1:48:10would encourage you to read if you're
  2551. 1:48:11curious about the direction of of AI.
  2552. 1:48:14Um, and then finally, um, one thing
  2553. 1:48:17that's going to be pretty clear, I call
  2554. 1:48:18it all the time, but it's the velocity
  2555. 1:48:21at which things are moving. You're
  2556. 1:48:22noticing part of the reason we're we're
  2557. 1:48:24giving you a breath in CS230 is because
  2558. 1:48:26these methods are changing so fast. So,
  2559. 1:48:28I don't want to bother going and
  2560. 1:48:30teaching you the number 17 methods on
  2561. 1:48:33rag that optimizes the rag because in
  2562. 1:48:34two years, you're not going to need it,
  2563. 1:48:36you know. So I would rather you think
  2564. 1:48:38about what is the breadth of things you
  2565. 1:48:40want to understand and when you need it
  2566. 1:48:42you are sprinting and learning the exact
  2567. 1:48:43thing you need faster because the
  2568. 1:48:45half-life of ski is so low you know you
  2569. 1:48:47want to come out of the class with a
  2570. 1:48:49good breath and then have the ability to
  2571. 1:48:51go deep whenever you need after the
  2572. 1:48:53class and so that's sort of how that
  2573. 1:48:54class is designed as well. Um yeah
  2574. 1:48:57that's it for today.

About this transcript

This page contains the full transcript of Claude AI Full Course: Build & Automate Anything in 2 Hour I 99% Of People Are Missing Claude AI by frugle 1.7M, generated from the public captions YouTube serves with the video. The transcript has 17,587 words across 2,574 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.