YouTube2Text

AI Engineering Full Course | 8+ Hours | LLMs, RAG, LangChain, LangGraph, AI Agents — Transcript

by neonlearn · 72,203 words · 7,637 segments · language en · Watch on YouTube

Full transcript

  1. 0:00Hi everyone, welcome to the NeonLearn YouTube channel.
  2. 0:03This is a comprehensive AI engineering course.
  3. 0:06My goal is simple, I want to teach you the most
  4. 0:09important AI engineering concepts in just
  5. 0:11one course. And this course is going to be fully project oriented.
  6. 0:16We will build 10 real world projects together and cover a lot of concepts and tools.
  7. 0:22We will work with LLMs, vector databases, embedding models and libraries like
  8. 0:28LangChain, LangGraph, Transformers and Sentence Transformers.
  9. 0:32And the only prerequisite is Python.
  10. 0:35So we will learn everything about AI engineering from scratch.
  11. 0:40Ladies and gentlemen, welcome aboard Neon Learn.
  12. 0:44Before we take off, your captain would like to walk you through the journey ahead.
  13. 0:4910 worlds, 10 missions, 1 destination.
  14. 0:52Please give your full attention to the captain.
  15. 0:55As I said, this course includes 10 projects.
  16. 0:59In our first project, we will build a chatbot from scratch.
  17. 1:02We will add memory to our chatbot, stream responses token by token,
  18. 1:06and summarize a conversation every 10 messages to save tokens and reduce costs.
  19. 1:12By the end of this project, you will know how to build
  20. 1:16an AI assistant for your own website or business.
  21. 1:19In project 2, we will build an AI system that can classify text.
  22. 1:23For example, it can detect if a customer is happy or unhappy with a product.
  23. 1:28This is called sentiment analysis.
  24. 1:30It can also classify articles into topics like health, economics, or technology.
  25. 1:36And we will do this without using an LLM.
  26. 1:39Instead, we'll use Natural Language Inference, or NLI.
  27. 1:44In project 3, we learn about embeddings.
  28. 1:48Embeddings are one of the most important building blocks of AI engineering
  29. 1:52and we will use them to build a semantic search system.
  30. 1:56In project 4 we will build a RAG system.
  31. 1:59RAG stands for Retrieval Augmented Generation.
  32. 2:02It is one of most valuable skills in AI engineering
  33. 2:06and I'm sure you have heard about it a lot.
  34. 2:09In project 5, we will build an AI agent.
  35. 2:12We will give our agent a set of tools and it can use them whenever it needs to.
  36. 2:17In project 6, we'll learn more about LangChain.
  37. 2:21First, we explore Prompt Template, which can help us manage our prompts.
  38. 2:25Then, we learn about Structured Outhood, which
  39. 2:28help us control the output of a model.
  40. 2:31We could give it a Wikipedia article about a person.
  41. 2:35Then, the model extracts details like
  42. 2:37their name, age, wealth, and other information.
  43. 2:41It returns everything in the exact format our application needs.
  44. 2:45In project 7, we will take memory management to the next level.
  45. 2:49We will use CheckPointer to save conversations in a database.
  46. 2:53And this means each user can start new conversations with a chatbot
  47. 2:59and return to old ones whenever they want.
  48. 3:02In project 8, you will learn about middleware.
  49. 3:05Have you ever used an AI assistant that asks for
  50. 3:07your permission before taking certain actions?
  51. 3:10In this project, you will learn how to build this kind of feature using middleware.
  52. 3:15It gives you more control over how your AI applications work.
  53. 3:21In project 9, we will explore Lang Graph.
  54. 3:24I will show you how to build an AI agent that improves its own work using a graph.
  55. 3:28Here's how it works. First, an LLM writes an article about a topic we provide.
  56. 3:34Then another LLm reviews article and gives it a score from 1 to 10.
  57. 3:40If the score is below 7, the article goes back to the first LL.
  58. 3:45And this process continues until the article gets a high score.
  59. 3:50Finally, in project 10, we will explore multi-agent systems.
  60. 3:54Instead of using just one agent, we'll use several agents.
  61. 3:58Each agent will specialize in a specific task.
  62. 4:01Together they can work on more complex problems.
  63. 4:0810 worlds of AI engineering lie ahead. Your journey begins at the launch base.
  64. 4:13Fasten your seat belt. Next stop, play a chatbot.
  65. 4:16Here you'll build your own AI chatbot from scratch
  66. 4:19with memory, real-time streaming, and smart summarization.
  67. 4:24Your first world awaits. Enjoy the journey.
  68. 4:32In this session, we're going to set up the Python
  69. 4:34environment that we need for this project.
  70. 4:37If you already know how to do this, I mean, how to have a virtual Python
  71. 4:41environment, write Python code and install third party packages.
  72. 4:45You can skip the session or at least fast forward this session. Okay.
  73. 4:50For writing Python code in this project, I'm
  74. 4:53going to use VS Code, Visual Studio Code.
  75. 4:55And if you don't have this code editor, you need to go to code.visualstudio.com
  76. 5:01and install the version that is compatible with your machine.
  77. 5:05And this is really straightforward.
  78. 5:08We don't need to follow this step together because it would be a waste of your time.
  79. 5:13And in addition, you'll need the Python interpreter as well.
  80. 5:17You need to get Python interpreter installed on your machine
  81. 5:20For doing that, head over to python.org
  82. 5:23Here again you can find a version that is suitable for your machine.
  83. 5:27Following this step to make sure that everything is right,
  84. 5:31I open up a terminal and I type Python 3.
  85. 5:35And if you are on Windows, just type Python. And I press Enter.
  86. 5:39And this way I can make sure that Python is correctly installed on my machine.
  87. 5:44The next step is to create our exciting project.
  88. 5:48And to do that, there are quite different ways.
  89. 5:52My favorite one is to create a folder first.
  90. 5:55I give it a name like AI engineering for the sake of this project.
  91. 6:01Then I open a terminal in this folder and I type code space period and this way.
  92. 6:08As you can see, this folder is successfully opened in my
  93. 6:12VS Code, which is really simple, really straightforward.
  94. 6:16But you could also first open your Visual Studio
  95. 6:20Code and go to File tab and open folder.
  96. 6:23And you could do that as well, but it's my preference.
  97. 6:26You could choose whatever you want.
  98. 6:28Then here I add a new file called chatbot.py.
  99. 6:34And this file is going to contain the code for
  100. 6:36our first exciting project, which is designing
  101. 6:39a chatbot.
  102. 6:40The next step is to install some extensions in this extension panel.
  103. 6:45And these extensions will turn our VS Code, which
  104. 6:49is a code editor, into a very powerful IDE.
  105. 6:52The first and the most important extension that we need to install is Python.
  106. 6:59which is created by Microsoft itself. So we need to install this extension.
  107. 7:05As you can see, I've already installed.
  108. 7:06And the point is that this extension uses PyLint as its linter.
  109. 7:11So you need to have PyLind as well. And to do that, we need install it in terminal.
  110. 7:17And it's easier for me to open up the VS Code
  111. 7:21internal terminal by pressing Ctrl and back tick.
  112. 7:26I need to execute this command, python3
  113. 7:30or if you are on Windows, again say python and then "-m pip
  114. 7:36install dash u pilot okay
  115. 7:38I've already
  116. 7:39installed pilot so I don't need to execute
  117. 7:43this comment but you have to if if you haven't for the next step we need to
  118. 7:48install a formatter which I use AutoPep8 and this formatter will format your code
  119. 7:54according to pep8 standard which is a popular preference in many projects and I
  120. 8:00After installing this formatter, whenever you want to format your Python code,
  121. 8:06open up your control panel by pressing Command-Shift-NP and type Format Document.
  122. 8:12It's tedious when you want it and it happens a lot when we want to repeat this step.
  123. 8:18So in order to avoid this step, I go to CodeTab,
  124. 8:22Preferences, Settings, and here I type Formats
  125. 8:26and Save.
  126. 8:28Okay, I ticked.
  127. 8:29this item and by doing that whenever I press ctrl and s the document is
  128. 8:35formatted for me automatically which is really simple and the next extension
  129. 8:42that we need is code runner which makes it
  130. 8:45easier for us when we want to run our code
  131. 8:47and after installing this extension we need to apply an important setting
  132. 8:53because in our project we want get an input from the user
  133. 8:58but the point is that Coderunner runs your code in Output tab
  134. 9:02which you cannot enter any input
  135. 9:05so to change this behavior again I go to settings
  136. 9:11and I search Coderaunner terminal you need to tick this item
  137. 9:17by doing that your program will be executed in terminal instead of Output Tab
  138. 9:23Now as a final step we need to create a virtual environment.
  139. 9:28It gives us an isolated environment
  140. 9:30to install third-party packages.
  141. 9:32In this way our projects can have their own dependencies
  142. 9:36which is exactly what we want in many cases
  143. 9:39and there are different ways to do that to
  144. 9:41create a Virtual environment but in this tutorial I will use pipenv. I use pipenv to
  145. 9:48create a virtual environment. This tool gives you an experience similar to NPM.
  146. 9:53NPM in JavaScript, which I really like it.
  147. 9:55This tool prevents you from memorizing a lot of
  148. 9:59other commands, which is really convenient.
  149. 10:02Okay, to use pipenv, first we need to install pipenv itself.
  150. 10:06And to do that, again, I open internal terminal
  151. 10:09in VS code by pressing control and backtick.
  152. 10:12And I type pip3, or if you're on Windows, just pip and pip.
  153. 10:17three install pipenv okay i've already done this step i don't need to do that
  154. 10:24in order to create a virtual environment with this tool
  155. 10:27you just need to say pipenv install it doesn't install anything it just create a
  156. 10:33virtual environment on your machine and as you can see
  157. 10:37two new files are created for us one of them
  158. 10:40is for tracking dependencies only and another one tracks the dependencies
  159. 10:47with the exact version of them okay as i told you pipenv
  160. 10:52create a virtual environment for you.
  161. 10:54This virtual environment has an address and
  162. 10:58we need, we do need this address because we
  163. 11:01need it in two places for two important settings.
  164. 11:05In order to get this address, you simply type pipenv dash dash venv.
  165. 11:12This gives you the address of that environment that is created for us.
  166. 11:17and the first place that we need this address is for code runner
  167. 11:21code runner needs to know where is this location
  168. 11:25this is the location of virtual environment
  169. 11:27otherwise it will use a wrong interpreter and to change this i open settings
  170. 11:34go to preferences settings
  171. 11:36okay i go to work space because i want to apply this setting
  172. 11:41only for this project not my whole system i switch to json mode
  173. 11:46and I type executor map
  174. 11:50And as you can see, many settings, many rows are
  175. 11:53added for different programming languages.
  176. 11:56We are working with Python. So we need to find Python.
  177. 11:59I replace this Python here with this address and I add slash bin slash Python 3.
  178. 12:07And if you're on Windows, again, Python, which is the Python interpreter itself.
  179. 12:12Now we are giving code runner the exact address of the Python.
  180. 12:16Python interpreter that we need for this project in this virtual environment.
  181. 12:21Okay, another place that we need to use its address is for VSGOD itself.
  182. 12:27VSGOd currently is using the wrong interpreter. I say the wrong
  183. 12:32interpreter because now we have a virtual environment and we need to
  184. 12:36use the interpreter that you just saw.
  185. 12:38And in order to change that, I open control panel by pressing Command-Shift-P and
  186. 12:46I type Select Interpreter
  187. 12:48Then I say, enter interpreter path, and I paste the address that I copied.
  188. 12:56and I press enter and now everything is ready for developing our exciting projects.
  189. 13:04Ladies and gentlemen, before our next flight
  190. 13:07begins, a quick request from your cabin crew.
  191. 13:10Please switch your phone to airplane mode or better yet silent.
  192. 13:14Then stow it away out of sight in your bag or in another room.
  193. 13:18Close any other tabs and let every distraction stay on the ground.
  194. 13:23If this journey needs your full attention,
  195. 13:25we'll now dim the cabin lights so you can focus. Please
  196. 13:28fasten your seatbelt.
  197. 13:30We're ready for
  198. 13:32takeoff.
  199. 13:42Okay, in this session we will start writing our first
  200. 13:45exciting project which is developing a chatbot.
  201. 13:48First, I create my main function, a function called main.
  202. 13:53And since this function doesn't have any
  203. 13:56implementation yet, I just put pass keyboard
  204. 13:59inside.
  205. 13:59Because in Python, when a block doesn't happen implementation,
  206. 14:04Python will start complaining about this matter.
  207. 14:07So for this reason, I put pass inside here. Then here I put an if statement.
  208. 14:13Let's write its code and I will explain it to you. Don't worry about it.
  209. 14:18First I say double underscore.
  210. 14:21name double underscore
  211. 14:24is equal to
  212. 14:25inside codes double underscore
  213. 14:29main double underscore in that case I want the main function to be executed
  214. 14:35okay what does that mean?
  215. 14:36let me explain it by saying that in Python there is a special variable called name
  216. 14:43and in the case that
  217. 14:46this file I mean chatbot.py is executed directly
  218. 14:51the value of this variable would be double underscore main double underscore
  219. 14:56only in that case and in that case I want the main function to be executed
  220. 15:02otherwise I mean in the case that this file is executed by another file I
  221. 15:09mean not directly the value
  222. 15:12of this variable will not be main this is a pattern
  223. 15:15that you will see a lot in different projects when
  224. 15:19you when we want to treat this file as a script
  225. 15:23okay now let's implement our main function
  226. 15:27At the first step, I want to get a piece of text from the user.
  227. 15:30So I use the input function, which is a built-in function in Python.
  228. 15:37And as an argument, I pass the label that I want the user to see.
  229. 15:42For example, ask something. Okay, that's
  230. 15:46great.
  231. 15:47And I get the value that user enters and I
  232. 15:50store it inside a variable called userInput.
  233. 15:55after that I'm gonna send this value for a function for a function called bot
  234. 16:00I define this function def bot
  235. 16:04this function is gonna contain the logic of our bot
  236. 16:07and this function will connect to a real LLM shortly
  237. 16:11I want this function to have a parameter called message
  238. 16:15which is the question that this bot is gonna answer in other word the query
  239. 16:20the query that this function supposed to answer but
  240. 16:24at this stage I'm not going to use this parameter because at this
  241. 16:27stage I want to keep things really simple and at this stage I
  242. 16:31want this function to send a simple piece of text for every single query
  243. 16:36this piece of texts is going to be this but hello how can I help you
  244. 16:43Okay, great. And I store it in a variable called answer.
  245. 16:47And I simply return this answer.
  246. 16:49Now I'm gonna call this function, call this
  247. 16:52function inside my main function and
  248. 16:55I send what the user has entered as its argument.
  249. 16:59So here I say user input.
  250. 17:02okay and I store it in a variable called bot answer
  251. 17:06then I print it print bot answer okay as simple as that now let's run our program
  252. 17:15OK, when you run your program for the first time,
  253. 17:18you see this virtual environment is activated.
  254. 17:22The next time you see it says ask something and it's
  255. 17:25better to put a colon here alongside the space.
  256. 17:28Okay, I say hello and it answers hello, how can I help you?
  257. 17:33As you can see, it's not smart. It sends this response for every single query.
  258. 17:37And in the following sessions, we're gonna make this
  259. 17:40function smarter by connecting it to a real LLM.
  260. 17:44Ladies and gentlemen, a quick note about your luggage.
  261. 17:48The full source code for every project in this course is waiting for you on GitHub.
  262. 17:54You'll find the link in the description of this video.
  263. 17:58If it helps you on your journey, please give the repository a star.
  264. 18:02and follow the captain on GitHub so you never miss a new flight.
  265. 18:07Thank you and enjoy the flight.
  266. 18:16Alright, now we are getting to a point where instead of using our dummy chatbot,
  267. 18:21we're gonna do the exciting part. I mean connecting our chatbot to an LLM.
  268. 18:25What does that mean exactly?
  269. 18:27Up to this point, you are using something like
  270. 18:30ChatGPT or Cloud or whatever chatbot you are
  271. 18:33using these days. What do you do?
  272. 18:35You're asking a question, you're getting an answer, and that's the process.
  273. 18:40Now, instead of that, instead using a chatbot directly,
  274. 18:44you're gonna write a program, you're going to send your question to this program
  275. 18:48and this program will send your questions to
  276. 18:51an LLM, get the response and return it to you
  277. 18:54that's the whole point of this project that we are doing
  278. 18:57now at this point to understand things better
  279. 19:00we need to cover a little bit of theory
  280. 19:02I hate theory too, but don't worry.
  281. 19:05We will cover, we will see, we'll see everything I explain in theory, in practice.
  282. 19:11On the other hand, it's necessary because in this course, there is no point.
  283. 19:16There is no in just writing a bunch of code
  284. 19:18without understanding what it actually means.
  285. 19:21So let's start by saying that if we take a step back and look
  286. 19:26at the bigger picture, what are we actually going to do?
  287. 19:31to see, to do.
  288. 19:32We are going to become an AI engineer.
  289. 19:35We are gonna add an AI feature to our application.
  290. 19:39We are goint to use an AI model and
  291. 19:42add it to our application.
  292. 19:44And this whole field is called AI engineering. Okay.
  293. 19:49In other words, if you follow this path all the
  294. 19:52way through and specialize in this field,
  295. 19:54you will become an A.I. engineer. which is actually a very hot and in demand field
  296. 20:01and you will see tons of job postings for it
  297. 20:05Another field you might come across is ML
  298. 20:07engineering or machine learning engineering.
  299. 20:10So
  300. 20:10what is the difference between an ML engineer and an AI engineer?
  301. 20:17An ML engineer typically focuses on building, building
  302. 20:21models, building and training machine learning models.
  303. 20:25And they do it from scratch. And it tends to be more math-heavy, research focused.
  304. 20:32And yes, they are working on things like designing models, training them, improving
  305. 20:38their performance and things like that.
  306. 20:41On the other hand, an AI engineer usually works with pre-trained models
  307. 20:47models that are already ready and they use
  308. 20:50these models to build smarter applications
  309. 20:54so instead of building the models from scratch
  310. 20:58they focus on using these models and integrating them into their products
  311. 21:04their real world applications and yes they focus
  312. 21:11these AI-powered features to their application.
  313. 21:14So, to summarize it, to put it in a very simple way,
  314. 21:18I can say ML engineers build and train models.
  315. 21:22On the other hand, AI engineers use these models.
  316. 21:26Another thing that I should explain it for
  317. 21:30you is NLP or Natural Language Processing.
  318. 21:34The goal of NLP is to enable computers to understand and generate human language.
  319. 21:42And we achieved this through algorithms, through model architectures.
  320. 21:47There is no magic behind and to achieve these
  321. 21:51goals we need algorithms and these architectures
  322. 21:54For example, in the past we had and still today we had an
  323. 22:00architecture called RNN or Recurrent Neural Network.
  324. 22:05And later transformers came along and I can say
  325. 22:09they completely revolutionized this field.
  326. 22:12In fact, much of the power behind these AI
  327. 22:16systems, these chatbots you see these days can
  328. 22:20be traced back to these architecture transformers and attention mechanisms.
  329. 22:27And the next word you hear is LM
  330. 22:33And the next one is LLM. Let's first focus on LM.
  331. 22:37An LM or language model is an AI model that
  332. 22:41can understand and generate human language.
  333. 22:45Okay, so language model.
  334. 22:48Now, what about an LLM?
  335. 22:50It stands for large language model, a language model that is large.
  336. 22:56It's a language that has been trained on a massive amount of data.
  337. 23:01Most of which comes from the internet.
  338. 23:04In other words, you can think of an L LL as a person,
  339. 23:07as a a person who has read and memorized an enormous amount of information.
  340. 23:13from the internet.
  341. 23:15Now what about LMM?
  342. 23:18LMM stands for Large Multi-Model Model
  343. 23:23The difference is that in addition to text, it can work with other types of data,
  344. 23:28or more accurately we say modalities, such as images, audio, and sometimes
  345. 23:34video.
  346. 23:35and after knowing these keywords we should answer
  347. 23:41what does an LLM actually do under the hood at a very basic level
  348. 23:47it just predicts the next word
  349. 23:50Pay attention to this, just one word at a time.
  350. 23:54And here, technically saying word, it's not quite accurate.
  351. 24:00The more precise term is token.
  352. 24:02But for now, don't worry, don´t worry about tokens.
  353. 24:05Just think of it as a word that we have in our language.
  354. 24:10For example, imagine we give your model something like this. Today I am very.
  355. 24:16the model tries to predict what comes next, the next word.
  356. 24:21For example, it might predict tired. Today I'm very tired.
  357. 24:25Okay, but the
  358. 24:26model, of course, doesn't stop here.
  359. 24:29It takes the text that you gave it, it adds the word that it just predicted,
  360. 24:35which becomes today I am very tired, and it tries to predice the next world.
  361. 24:40For example, today I am very tired because
  362. 24:44okay now we have this whole sentence so the
  363. 24:48mechanism is like this it predicts one word
  364. 24:52it adds it to the text and it keeps repeating these loops, this step.
  365. 24:58So this is the basic loop behind LLMs, behind text generation.
  366. 25:04So to summarize it, when you send a query to a language model,
  367. 25:09you give the model some text and the model predicts the next token
  368. 25:14then predicts next one, the next one the next and so on
  369. 25:17So if you ask a model a question and it gives you a 100 word answer
  370. 25:24It didn't generate those 100 words at once
  371. 25:28It generated them one token at a time in a loop Okay?
  372. 25:32Until it decide that the response is complete
  373. 25:36There is a special token called completion token
  374. 25:39And when this token appears, it's done and it returns the response So...
  375. 25:45That's a basic idea behind how an LLM generates text.
  376. 25:50and there are lots of different options
  377. 25:54lots of language models that you hear their names a lot
  378. 25:58for example ChatGPT, you use it, it is created by OpenAI
  379. 26:03Cloud is from Anthropic, Gemini, Llama, Mistral, DeepSeek, Qwen
  380. 26:09and instead of just listing a long list of different models
  381. 26:16it's better to see, to show you It's mine.
  382. 26:21Ladies and gentlemen, as we orbit planet chatbot,
  383. 26:24we're now approaching one of its moons.
  384. 26:27Its name is Hugging Face.
  385. 26:29In the world of AI engineering, this moon is a true landmark.
  386. 26:34A vast open harbor of free AI models, data
  387. 26:37sets, and tools shared by a global community.
  388. 26:41Almost every AI engineer docs here sooner or later,
  389. 26:45and on this journey, so will you. Please
  390. 26:47remain seated as we begin our approach.
  391. 26:57In the world of AI engineering, HuggingFace is
  392. 26:59a website whose name you're gonna hear a lot.
  393. 27:02And not only are you going to hear about it, you're
  394. 27:04actually going to be working with it quite a lot
  395. 27:07At first, this appearance interface might look a little
  396. 27:10bit intimidating, but don't worry about it at all
  397. 27:13In fact, it's really, really simple
  398. 27:16and they've done a really great job organizing everything
  399. 27:20And honestly, I'm really glad that something like this exists. It's a hub for AI.
  400. 27:25So, what exactly is HuggingFace? You can think of it as something similar to GitHub.
  401. 27:30You should definitely be familiar with GitHub.
  402. 27:33It allows people to publish their projects, collaborate on each other's code,
  403. 27:37work together to improve, to build something in general open source projects.
  404. 27:42And Hugging Face has the same goal.
  405. 27:44And these hubs, hubs like Huggingface and GitHub,
  406. 27:49has given us a huge amount of power.
  407. 27:51and has contributed a lot to the progress of
  408. 27:54software development and AI development these days.
  409. 27:56So I really appreciate their work and on HuggingFace, anyone can
  410. 28:01take a model, a model they have trained and publish it here.
  411. 28:05Other people can come and find these models, download them and even improve them.
  412. 28:10improve and publish them again so that's the
  413. 28:14purpose of hugging face and it's not it's
  414. 28:17not limited to AI models it you can find data
  415. 28:20sets as well the spaces what about the spaces
  416. 28:23the spaces allow you to to take something we have
  417. 28:25built publish it and basically say say to others
  418. 28:28here I have this try it and you don't even have to have hardware you can
  419. 28:34experiment with it without having hardware.
  420. 28:38And yeah, that's HuggingFace and let's take a quick look around.
  421. 28:42The first and the most important section is here, models, model section.
  422. 28:48Here you can find different kinds of models that have been published on Huggingface.
  423. 28:52You can search for a specific model by its name
  424. 28:56and you can apply different filters here.
  425. 29:00there are many many many different filters here and just keep in mind that
  426. 29:04hugging face changes a lot I mean this appearance filters might change for example
  427. 29:11you might see a particular task here today but it might not be here tomorrow or
  428. 29:17you might a task here that wasn't here yesterday anyway
  429. 29:22here there are different tasks that you can filter models based on this
  430. 29:27tasks.
  431. 29:27You also have parameters and parameters basically tell you how large a model is and
  432. 29:35they have a significant impact on both the size and the capacity
  433. 29:39the capacity of a model
  434. 29:42so generally speaking a model with fewer parameters
  435. 29:46is smaller and requires fewer resources
  436. 29:49but it also has a lower capacity and here we have libraries
  437. 29:53you can see different libraries here like TensorFlow, PyTorch, Transformers
  438. 29:58which is HuggingFace's own library and we have different providers here
  439. 30:03Honestly, I'm gonna keep this part, keep this session very brief
  440. 30:08because at this stage you don't need to worry about these details
  441. 30:11and we will cover them later, but I just don't want you to be intimidated.
  442. 30:16So that's about models and as I told you, you have datasets here.
  443. 30:21Here again, you can search through different
  444. 30:24datasets and you can apply different filters.
  445. 30:27Yeah, that's enough for now and I highly recommend that you
  446. 30:31you spent some time exploring Hagenface, click
  447. 30:35around, search for different models and
  448. 30:38just get familiar with the platform.
  449. 30:40Now let's take a quick look at one of these models, for example, Llama.
  450. 30:46which is released by Meta or Facebook
  451. 30:49you can see these different models and hugging face
  452. 30:52and I can say you for almost every popular model you find here
  453. 30:57there are different variations there are difference versions
  454. 31:00and in most cases the underlying model is the same but that company or maybe others
  455. 31:06may have changed that model for example a change in the number of parameters
  456. 31:12and they publish it again
  457. 31:13That's the reason you can see different versions for Llama here
  458. 31:17For example, one of them has 8 billion parameters Another one, 70 billion parameters
  459. 31:24And let's take a look at this one One important section is this part
  460. 31:28You can find the model's identifier here
  461. 31:31You're gonna need this name when you want to use this model
  462. 31:35When you want download it from HuggingFace
  463. 31:37And with this icon you can copy this identifier
  464. 31:41And here what you are looking at right now is
  465. 31:44called model card and you can see that here.
  466. 31:48You can think of the model card as a documentation for this model.
  467. 31:52It contains all the information that you need, that
  468. 31:56the publisher has decided to share with others.
  469. 31:59And you can read about what the model is, who created
  470. 32:03it, how it was trained, what data sets were used and
  471. 32:07benchmarks it was evaluated on and many many other details that you might need.
  472. 32:13Basically, if you want to understand the model in
  473. 32:17more depth and gather some information about it,
  474. 32:21decide whether it is a good model for you or not, this
  475. 32:24is where you can find the necessary information.
  476. 32:28and for some models you can even test the model here and you
  477. 32:33can see you can enter a prompt and
  478. 32:36and see the model's response right here without downloading
  479. 32:41this model on your system or on your server so
  480. 32:45there is a lot more information on these pages and i really recommend that you
  481. 32:51spend some time
  482. 32:53don't be afraid don't afraid to explore and
  483. 32:56Just click around, read things and just see where it takes you. Okay.
  484. 33:01In addition, before ending the session, there is a cool thing here.
  485. 33:08There is an icon here and it says use this model, different
  486. 33:12libraries and different providers and we have a notebook.
  487. 33:17For example, if I click on Hugging Face, you
  488. 33:21can see it guides you and that's really cool.
  489. 33:25Even if you are completely beginner and new to
  490. 33:28Hugging face, it tells you how to use this model.
  491. 33:32So here it has suggested that you can use a pipeline or you can
  492. 33:37auto classes and these auto classes gives you more control.
  493. 33:42That's really great and we will cover this, don't be worried.
  494. 33:45I'm just getting you familiar with these stuff.
  495. 33:49Yes, the whole point of this session is to know that Hugging Face exists,
  496. 33:54and it's basically one of the most important hubs,
  497. 33:58and you're going to work with it a lot. So just explore it.
  498. 34:06Now let's talk about how to choose and use a model.
  499. 34:09If the model you want to use is commercial, for
  500. 34:13example, a model like Sonnet by Anthropic,
  501. 34:16then you basically need to go through its provider and you need to purchase a plan.
  502. 34:21Yes, that totally depends on the provider because that model
  503. 34:25is commercial and is only provided by that company itself.
  504. 34:29But what if the model is open source?
  505. 34:31What if we have found a model on Hugging Face and
  506. 34:35we're going to use it in that case you have two options
  507. 34:38the first option is to run the model on your own hardware
  508. 34:43or use it directly this way
  509. 34:47This is the first option, the other option, the second option is to use a provider
  510. 34:50that has already deployed that model on their own hardware
  511. 34:54and they give you the access through an API key.
  512. 34:59So, and what is our approach in this course?
  513. 35:03In this course, since I know many of you don't have enough hardware
  514. 35:09that is enough for running these models locally, I'm going to leave
  515. 35:15the on-premise option aside and we are going
  516. 35:17to use a provider and the question is which
  517. 35:20provider? The answer is Groq.
  518. 35:23Personally, when I'm experimenting with different
  519. 35:26models, I use Groq, but the most important part
  520. 35:28is that it gives you a free tier so you don't
  521. 35:32have to spend money to follow up this course,
  522. 35:36which was really important for me.
  523. 35:42Ladies and gentlemen, a brief disclosure before we continue.
  524. 35:46Along this journey, we'll make stops at several third-party
  525. 35:50services like Groq, Tavily, Hugging Face, and Lang Chain.
  526. 35:54Neon Learn is not affiliated with, sponsored
  527. 35:57by, or endorsed by any of these companies.
  528. 36:00We chose each tool purely because it's great for learning.
  529. 36:04All product names and trademarks belong to their respective owners.
  530. 36:09Thank you for flying with NeonLearn.
  531. 36:19Okay, as I mentioned for this course we're gonna use Groq and to avoid any confusion
  532. 36:24this Groq is different from that Grok, that API chatbot from Elon Musk
  533. 36:29there are different products, the names may sound similar but they're not the same
  534. 36:33Okay, the first question, why have I chosen this product, this provider?
  535. 36:37The answer is pretty simple, it's very generous, it
  536. 36:40is very generous when it comes to its free tier
  537. 36:42Personally, I use it for a lot of my experiments
  538. 36:46Honestly, it happens rarely when I hit limits.
  539. 36:50It's also not a bad option for production.
  540. 36:52I mean, if you move to the paid plan and if you
  541. 36:56want to use Groq in your real-world projects,
  542. 36:59the pricing is still very reasonable.
  543. 37:02And the good point is that it offers a good
  544. 37:04range of different models which are suitable
  545. 37:06for different kinds of tasks.
  546. 37:08And it has models that are capable enough for your tasks.
  547. 37:12Another benefit is pricing.
  548. 37:14In terms of pricing is really good in comparison with many other popular providers.
  549. 37:20So for production is also good.
  550. 37:23And about pricing later on in this course, I
  551. 37:26will show you how to calculate pricing, how
  552. 37:30to calculate the cost of using these models,
  553. 37:32which is really important in the world of AI
  554. 37:35engineering.
  555. 37:36But for now, the most important thing that I
  556. 37:38want to say is that we're going to use Groq
  557. 37:41and it gives us a free tier and it's enough. It's enough for us.
  558. 37:48So in the first step you need to create an account and it's really simple.
  559. 37:53I've already done this step.
  560. 37:55So if I click on start building, it takes me to my dashboard.
  561. 38:00Here we have something very important API key.
  562. 38:03You are going to need that.
  563. 38:04You're going to create and API key and please, please do not try to copy my API key,
  564. 38:10I'm intentionally not hiding mine because I don't want to waste my time to edit.
  565. 38:16these videos hiding my api keys and it's also good
  566. 38:20for you because you can uh clearly follow me along
  567. 38:23this course and understand things better so
  568. 38:27i'm telling you again don't copy my ap i key
  569. 38:30and i i will delete i will the api key that i use
  570. 38:34during this course after uh after recording
  571. 38:38and here uh let's take a quick look at our playground playground is where
  572. 38:43we can directly interact with the models that Groq offers.
  573. 38:47And here we can see a list of different models that are currently available.
  574. 38:53In the future, maybe other models will be provided,
  575. 38:57and maybe some of these models are not provided anymore.
  576. 39:01And in that case, Groq will inform you if you are currently using these models.
  577. 39:07and for example look at this model Qwen this model
  578. 39:12is by Alibaba and you can choose this model
  579. 39:15send some prompts to this model and get the response
  580. 39:18currently I'm saying that you can experiment
  581. 39:21with this model here and later on we will connect
  582. 39:25these models to your own application so we can
  583. 39:28send queries through our own application which is really interesting and
  584. 39:33Here another point is that another interesting thing,
  585. 39:36another thing that you might be wondering is that
  586. 39:39earlier I told you that if you want to use OpenAI's model,
  587. 39:44you normally need to go to OpenAI and purchase some credit.
  588. 39:49And now you might say, why do I see OpenAI models here in Groq?
  589. 39:53Well, the important distinction is that the famous
  590. 39:58OpenAI model, yes, they are enterprises are paid.
  591. 40:02to go to OpenAI itself. But the point is that OpenAI has also released some
  592. 40:08open-weight models and Groq supports those models exactly.
  593. 40:14So these models are open-weighted.
  594. 40:17You can go to Hugging Face, download these
  595. 40:19models and open when you're harder yourself.
  596. 40:21So anyone can do that.
  597. 40:23And the final thing, if you look at these models, if
  598. 40:27hover them, you can see the limits on your free tier.
  599. 40:31For example, how many liquids you can make per minute, per day.
  600. 40:36and how many tokens you can process during the day.
  601. 40:40So at this step, the only thing that I want you to do is to create a Groq account
  602. 40:47and create your own API key. We need that API key in the following sessions.
  603. 40:55Now, the really interesting thing about Groq is that
  604. 40:59they have built specialized hardware and infrastructure
  605. 41:03that is extremely well optimized for running these kinds of models.
  606. 41:08And because of that, the responses are incredibly fast.
  607. 41:12In fact, even if you download and run these models on your
  608. 41:16hardware, you cannot get the same result only if you provide
  609. 41:21that kind of hardware which would be really pricey for you.
  610. 41:25So for these purposes, using a provider really makes sense.
  611. 41:29And I always say that only focus on your product, your AI
  612. 41:33features, and use a provider because they know how to do that.
  613. 41:37They use the proper type of hardware, which is really important.
  614. 41:49Okay, here is my API key.
  615. 41:52Hopefully by the time of watching this video, you
  616. 41:55have already created your account and created
  617. 41:58your API key because as I told you, we are
  618. 42:00going to need it in the following sessions.
  619. 42:02But now I'm going to talk about the connection between you and Groq.
  620. 42:07How exactly do we establish a connection between
  621. 42:10our application and the provider, which in
  622. 42:12this case is Groq, okay? Okay.
  623. 42:15Imagine we have a query. A query, for example, what is the capital of France?
  624. 42:19The question is that how do we send this query
  625. 42:22to the provider and get a response back?
  626. 42:25We do it through an API. If you are already familiar with web development,
  627. 42:30this concept should be familiar to you. If not, I can say at the simplest level,
  628. 42:36you can think of it as a URL that you are seeing at your browser.
  629. 42:40You are watching this video on YouTube and there is a URL.
  630. 42:45When you are working with API, we have a similar thing,
  631. 42:48but instead of opening your browser and entering that URL,
  632. 42:52we are using that API in our application. So that is what API means.
  633. 42:59Now there is something important that you should know about
  634. 43:04you should about it when it comes to AI engineering and working with APIs
  635. 43:08In practice, you are really going to interact with these APIs yourself
  636. 43:12For example, you're not going to use Python's request package
  637. 43:17constructing an HTTP request, adding your API
  638. 43:20key to it, sending it and parsing the response
  639. 43:24No, no, you don't do that
  640. 43:26Technically you can do, but in AI engineering, we usually
  641. 43:30have a better approach, more standard approach.
  642. 43:32Instead, we use something called a wrapper. So why exactly is a wrapper?
  643. 43:37Think about the word itself, okay?
  644. 43:39When you wrap something, you put something around
  645. 43:42it, like a sandwich, okay, like the shower one.
  646. 43:45So a wrapper is basically a piece of software.
  647. 43:50that sits around another piece of software and
  648. 43:53makes it easier for you to interact with.
  649. 43:56and you can see an example of it here here it suggests that you use Groq
  650. 44:02the wrapper of Groq itself it says instead of interacting with my api directly
  651. 44:07just use my python package
  652. 44:09the point is that they have already developed this package
  653. 44:12and this package handles a lot of complexity for you
  654. 44:16and you don't have to create the request yourself
  655. 44:19you just import this package and create an instance and client
  656. 44:22and you start using it
  657. 44:24and that's really simple and it's just a wrapper around that API
  658. 44:29and Groq isn't the only provider that does that
  659. 44:33if you work with OpenAI models, you have the same concept there
  660. 44:37and even if you use other libraries and other providers, you'll have the thing there
  661. 44:44for example, Langchain is an AI engineering library, they use it a lot
  662. 44:49To elaborate on that, to explain why don't we
  663. 44:53just communicate with the API directly again.
  664. 44:56One simple, as I told you, is simplicity.
  665. 44:59And if we did everything yourself, you would have to handle a lot of edge cases.
  666. 45:04For example, you had to deal with authentication, constructing requests
  667. 45:09correctly, handle errors, parse responses, and many, many other things.
  668. 45:14It's clear that.
  669. 45:16The more code you write yourself, the more opportunities there are for mistakes.
  670. 45:20So when there is a better approach, that doesn't make any sense.
  671. 45:24The second advantage is standardization.
  672. 45:28It gives you a standard way to interact with the model.
  673. 45:31We can communicate with our provider in a more consistent way.
  674. 45:36And this becomes especially important when you join a team.
  675. 45:40Imagine you're joining a team and you have to ask many questions.
  676. 45:43For example, what parameter do I use for API keys?
  677. 45:47And how do I tell the model how many tokens it should generate?
  678. 45:51and what do I call this parameter, that parameter, the
  679. 45:55parameter for maximum number of tokens and many other things.
  680. 45:59With these wrappers, things are much easier and we have a
  681. 46:04standardized interface and we all agree that standard.
  682. 46:10and many many details are already taken care of.
  683. 46:14So this way instead of everyone inventing their own way of communicating,
  684. 46:20we have a standard and consistent way and that's important
  685. 46:25when you are collaborating on a project.
  686. 46:28Ladies and gentlemen, a special message from the captain.
  687. 46:32This journey was originally planned as a premium first-class flight.
  688. 46:37But the captain made a decision to open the doors
  689. 46:40and bring as many passengers on board as possible.
  690. 46:44So this entire course is now completely free right here on YouTube.
  691. 46:49If it's helping you, please share it with friends who'd love this journey.
  692. 46:53Every passenger you bring makes this ship stronger.
  693. 46:57The captain is truly grateful to every one of you.
  694. 47:00Thank you for flying with Neon Learn.
  695. 47:04Okay, in the previous session we talked about wrappers.
  696. 47:07I told you that instead of using an API, we're going to use a wrapper.
  697. 47:11And as an example, we saw this Groq wrapper, which is published by Groq itself.
  698. 47:16It suggests us to use this wrapper.
  699. 47:19However, in this tutorial, we are not going to use Groq's wrapper.
  700. 47:22We're going use OpenAI wrapper.
  701. 47:23The reason is that honestly, I think OpenAI Wrapper is more widely used.
  702. 47:27So it's more useful to you.
  703. 47:30In addition, Langchain is also extremely popular and you
  704. 47:33will see these too often in job hostings as a requirement.
  705. 47:36And you should be familiar with these two,
  706. 47:38OpenAI and Langchain, if you want to get a job.
  707. 47:41You should understand how to work with these
  708. 47:44wrappers, what parameters they have and what
  709. 47:46is the format of output.
  710. 47:48That's really important and one of my goals here
  711. 47:51is to get you closer to the actual job market.
  712. 47:54So in this tutorial, I've decided to use OpenAI Wrapper.
  713. 47:57something that is really important and you will see a lot.
  714. 48:00But you might be wondering, you might say,
  715. 48:03wait a moment, earlier we decided to use Groq as our provider.
  716. 48:07So what exactly is happening here?
  717. 48:09Because obviously the OpenAI wrapper is associated with OpenAI.
  718. 48:14while Groq is a completely different provider and what's happening here?
  719. 48:18How can we use OpenAI Wrapper with something else like Groq?
  720. 48:21Well, the interesting thing is that even if we don't want to use
  721. 48:26Open AI's models, there are still cases we can use OpenAi Wrappr.
  722. 48:31In other words, you can use the OpenAI wrapper, but
  723. 48:35instead of sending your request to OpenAI servers,
  724. 48:38you can configure the wrapper and send the request to another provider such as Groq
  725. 48:44of course this only works if the provider supports the OpenAI compatible API format
  726. 48:51so that's important and fortunately Groq supports this
  727. 48:55so that is exactly what we are going to use we are gonna use OpenAI Wrapper
  728. 49:00but instead of sending request to OpenAI we are goint to send our request to Groq
  729. 49:06and honestly I'm getting a little bit tired of theory myself.
  730. 49:10So let's stop talking about theory and let's get into the code.
  731. 49:16Ladies and gentlemen, we are now beginning our descent onto planet Chatbot.
  732. 49:22Please fasten your seat belts and prepare for landing.
  733. 49:25Touchdown, welcome to the surface.
  734. 49:28At the heart of this planet lives a large language model,
  735. 49:31and it's time for our first contact.
  736. 49:34We'll send our very first message and wait for its reply.
  737. 49:48After having many theory-based sessions, now
  738. 49:51we want to come back to our code and continue
  739. 49:53developing our project, which is a chatbot.
  740. 49:57So the first step is to install the wrapper I told you, OpenAI.
  741. 50:01And this is the first time, I guess, we want to install a third-party package.
  742. 50:04I open up internal terminal by pressing Ctrl and back tick, and I type pipenv.
  743. 50:13install.
  744. 50:15whenever you want to use this tool, I mean pipenv to install a package
  745. 50:18you simply say pipenv install and then you list the name of the packages you want
  746. 50:23to install now we have two packages that we want to install
  747. 50:27the first one is OpenAI, as I told you, the wrapper itself
  748. 50:30and I also want to instal.env at the same time
  749. 50:35I will explain what this package is for just wait a moment
  750. 50:40I just press enter to install these two packages We need to wait a bit.
  751. 50:46Okay, installation is done and everything is ready.
  752. 50:49I mean, these two packages are installed and we can start using them.
  753. 50:53As I told you, when
  754. 50:55you want to use Groq, you need an API key.
  755. 50:58And this is my API key and now you have your own API key
  756. 51:02and the point is that there is absolutely nothing wrong
  757. 51:06with putting your API key directly into your code.
  758. 51:09Just for learning and experimenting purposes.
  759. 51:12but this is not okay at all for production. Why?
  760. 51:17Because eventually you are going to push this code to a repository
  761. 51:21and you definitely don't want your API key sitting directly inside your source code
  762. 51:27where other people can see and access it.
  763. 51:30So in situations like this, when in situations we have a secret key, an API key,
  764. 51:37the best practice is to use environment variables.
  765. 51:41Here in this project, I create a file called.inv
  766. 51:49Okay.
  767. 51:50Then I take this API key.
  768. 51:54And I go here
  769. 51:57I paste it, I cut and paste it here. Now this API key is inside my.n file.
  770. 52:04Now I'm going to give it a name.
  771. 52:06I call it Groq API key. Okay.
  772. 52:12And as a convention, we use uppercase letters. Okay. That's it. Everything is okay.
  773. 52:20with our.n file. So what happens after this?
  774. 52:24Now this API key becomes an environment variable and wherever I need this API
  775. 52:31key, I inject it into my application.
  776. 52:35instead of hard coding this piece of information here
  777. 52:39I use it as an environment variable I store it as environment variable
  778. 52:44and whenever I need it, I load it
  779. 52:46and this is a very common pattern you will see in real world applications
  780. 52:51now the question is that how can we actually access it in Python?
  781. 52:57in Python there are different ways we can handle environment variables
  782. 53:01but one of the most common and popular approaches is
  783. 53:04is using the python.env package that we installed earlier.
  784. 53:09So the first step for using this package is importing it.
  785. 53:15I say import.envs.
  786. 53:18and there is also one more thing that we need which is OS module so I also import OS
  787. 53:27Now, how
  788. 53:28can we use it?
  789. 53:29We simply call load.env, the helper function that this.env provides.
  790. 53:35So I say.envs.load.envs as simple as that.
  791. 53:40So this loads the variables from.enV into the environment.
  792. 53:46And whenever we need it somewhere in our application, we can access it.
  793. 53:51And how can be accessed? Using OS.
  794. 53:56Here I say os.getEnv
  795. 54:00and then you pass the name of the variable you have
  796. 54:04which is Groq API key for us in this case.
  797. 54:08and that's it and to see it in terminal to see everything is working correctly
  798. 54:14I printed.
  799. 54:16Okay, now if I run my program,
  800. 54:19Again, after activating environment variable, as
  801. 54:22you can see, the API key is printed here for us.
  802. 54:26Now, everything is ready to use OpenAI Wrapper. I'm going to import it first.
  803. 54:32I say from...
  804. 54:35OpenAI Import
  805. 54:38OpenAI is a class, it's a class that we can create an instance of it. So down here,
  806. 54:46I say
  807. 54:48OpenAI
  808. 54:49and this way by putting parentheses I can create an instance of it
  809. 54:54after that we need to pass some arguments
  810. 54:57and the first argument is definitely API key
  811. 55:01so here I say API key and you can see that it is suggesting that to us
  812. 55:06so I put the API key that I have here
  813. 55:11and I don't need this anymore I remove it and this is my API key
  814. 55:17and there is also one more important thing this is OpenAI Wrapper
  815. 55:23but we want to connect it to Groq so we need to change something
  816. 55:28and we apply this change using an argument and that argument is baseline
  817. 55:35so base URL I'm sorry so here I say base URL
  818. 55:41and this is a URL that is provided by Groq and when you set it
  819. 55:45your requests are going to be sent to Groq instead of OpenAI
  820. 55:51and the API is api.groq.com OpenAI v1 so here I say HTTPS
  821. 55:59API.
  822. 56:01groq.com
  823. 56:03and open AI
  824. 56:06v1 and finally i'm going to store the client
  825. 56:09object it is a client object inside a variable
  826. 56:12and i call it llm okay
  827. 56:15Usually you will see this variable named something like client, but here I
  828. 56:20prefer to call it LLM because this object is going to be our inference for
  829. 56:25communicating with the language model and I want you to feel it closer.
  830. 56:31now we have our LLM object and we have configured it correctly
  831. 56:36now the next step is to use it to send a request to it
  832. 56:41for this purpose here inside my bot function instead of having this fixed text
  833. 56:47I say I use this LL I say
  834. 56:49LLm chat.completions.create
  835. 56:55okay don't worry I will explain each part of it one by one for now
  836. 57:00Let's just write our code, provide the arguments that we need,
  837. 57:04and we will come back to it shortly.
  838. 57:07So the first thing that we needed is model name.
  839. 57:10and the question is that what model are we
  840. 57:13going to use and where do we choose it from
  841. 57:15where do you find it and the answer is here in your
  842. 57:19dashboard you can see the list of models that are provided
  843. 57:24you can choose the model that you want you can copy
  844. 57:28its name pay attention when you want to use Groq
  845. 57:31you should copy the name of the model here not on hugging face hugging face
  846. 57:37is for when you want use hugging face to use models but here we are dealing with
  847. 57:43Groq so we we need this identifier okay i'm going to use this model and open a model
  848. 57:51and it is gpt-oss 20 billion parameter and i copy its
  849. 57:57name and here i say model and i paste it here inside
  850. 58:05okay so far so good and let's move on to the next argument which is messages okay
  851. 58:14This messages array is a more accurate list in Python we call it list.
  852. 58:19It's going to be a list of messages and each message has a special format.
  853. 58:25It needs two pieces of information, role and content.
  854. 58:30So to put a message here, I say role.
  855. 58:35It should be set and
  856. 58:37Alongside, you should specify content.
  857. 58:41Don't worry about it for now, I will explain to you. For now,
  858. 58:43I just set the role to user.
  859. 58:47and for content I'm going to put the message that this function receives here
  860. 58:54okay that's it basically we have three roles user, system and assistant
  861. 59:00for now just understand that by using user role
  862. 59:04you're telling the LLM that this message is coming from the user
  863. 59:09in other words a human is asking a query
  864. 59:13and you're supposed to enter this query
  865. 59:16Yeah, that's enough for now. We will talk about that in detail later.
  866. 59:21So here I'm going to send this request and whatever comes back,
  867. 59:25I'm gonna store it in a variable called response.
  868. 59:30Now, response itself is an object, it is nested and it contains several fields.
  869. 59:38And when we want to extract the LLM's response, the
  870. 59:41first item that we need to access is choices.
  871. 59:45So I say response.choices
  872. 59:48and I need the first
  873. 59:51item. Again, I will explain it. Please be patient.
  874. 59:55And then inside I have message.
  875. 1:00:00Okay, and finally we have
  876. 1:00:03content.
  877. 1:00:06And finally it gives me the answer that the LLM
  878. 1:00:09has generated and I just return it instead of
  879. 1:00:14the previous thing. I say return and that's it.
  880. 1:00:19I agree.
  881. 1:00:19The syntax is a little bit weird at first, but I can make it more clear to you.
  882. 1:00:26So let's start here when we are creating our request.
  883. 1:00:30What does that mean? What does this chat refer to?
  884. 1:00:33It means that we want to interact with the model through a chat interface.
  885. 1:00:39there are also other types of interactions
  886. 1:00:41like image generation, embeddings and so on
  887. 1:00:44but in this case we are building a chatbot
  888. 1:00:46so we want to communicate with this model using the chat interface
  889. 1:00:51okay and here what does this completions mean?
  890. 1:00:55It refers to the idea of text completion that I talked about.
  891. 1:00:59You mean, I told you that an LLM generates text one token at a time.
  892. 1:01:04And that's essentially the meaning behind these completions here.
  893. 1:01:08And create is just the trigger.
  894. 1:01:11When we call create, we are essentially saying create a completion for me.
  895. 1:01:17And yeah, you are sending your request. And this is a point where
  896. 1:01:23where you are building your request, you're sending your request.
  897. 1:01:27And you send your request and get a response back. So
  898. 1:01:30what do we have here? Response.choices.
  899. 1:01:33Firstindex.Message.Content. What does that mean? Why do we
  900. 1:01:37have a list here? That's because
  901. 1:01:40in some
  902. 1:01:41cases you might have more than one response. But
  903. 1:01:45in 99% of the cases, you are gonna have just one item. And
  904. 1:01:50This array will contain just one item. So we need this single item
  905. 1:01:55and because of that, I'm extracting it. Here we have a list that contains one item
  906. 1:02:01and I'm taking out that one item,
  907. 1:02:05And after this we have message.content.
  908. 1:02:08Be aware that the format of the response is also based on a message.
  909. 1:02:13Remember this message that we constructed here.
  910. 1:02:16We had a role and we had content and we have the same here.
  911. 1:02:21We have a role, and we a content.
  912. 1:02:23Although here we have role assistant instead of role user.
  913. 1:02:28And content is the answer that the LLM has generated.
  914. 1:02:34So again, the user sends a message with the role user
  915. 1:02:38and the response that comes from LLM has the role assistant.
  916. 1:02:43I know this might feel a little confusing right now, but don't worry.
  917. 1:02:48I'm going to repeat these concepts several times
  918. 1:02:51in different contexts and at different speeds.
  919. 1:02:54So we will have plenty of examples, explanations and you don't need to be worried.
  920. 1:03:01So up to this point, we have managed to get input from the user, send it to
  921. 1:03:08a real LLM, and now we have the response.
  922. 1:03:12And the only
  923. 1:03:13thing left is to test our program and make sure everything is working.
  924. 1:03:19correctly.
  925. 1:03:20One more time, okay, it says ask something and I say hello.
  926. 1:03:27Okay, again, I run my program one more time and this time I say
  927. 1:03:34What is the capital?
  928. 1:03:37of
  929. 1:03:38France
  930. 1:03:40And as you can see, it responds and here we go.
  931. 1:03:44You can see that we are successfully communicating with the LLM
  932. 1:03:48and getting a real response back.
  933. 1:03:51So we have officially connected our application to a real language model.
  934. 1:03:57Ladies and gentlemen, please direct your attention
  935. 1:04:00to the screen for a brief safety demonstration.
  936. 1:04:03In the unlikely event that you enjoy this lesson,
  937. 1:04:06a like button will drop down in front of you.
  938. 1:04:09Simply pull it toward you and press.
  939. 1:04:12Please take a moment to locate your nearest subscribe button.
  940. 1:04:16It may be located just below this video.
  941. 1:04:19Comments can be found under your seat.
  942. 1:04:22Please use them freely at any time during the flight.
  943. 1:04:26This concludes our safety demonstration. Enjoy the flight.
  944. 1:04:37In the previous session you saw that whenever
  945. 1:04:39you want to send a request to the provider
  946. 1:04:42you need to specify a number of parameters and in this session
  947. 1:04:46I want to explain what those parameters actually mean and are
  948. 1:04:50The first one is Model and you saw that.
  949. 1:04:52This is very straightforward and we need to
  950. 1:04:55tell the provider which model we want to use.
  951. 1:04:58The provider provides us with a number of different models and we
  952. 1:05:03need to specify exactly which one we want work with.
  953. 1:05:07And the next one is Messages.
  954. 1:05:09As you saw, Messages is a list of messages and
  955. 1:05:12you might be wondering why is this a list.
  956. 1:05:15why isn't just a single message we are only sending one request
  957. 1:05:19for now let me leave it aside I'll make that clear for you in just a little bit
  958. 1:05:26okay so for now, let's ignore it and instead let's focus on this one role
  959. 1:05:34each message should have a role and we have three different roles okay
  960. 1:05:41And the point is that these roles are not specific to OpenAI.
  961. 1:05:47They are a general concept in AI engineering and you will see them in
  962. 1:05:52different places, maybe with different names, but the concept is the same.
  963. 1:05:57So first, what are these roles? These are System, User and Assistant.
  964. 1:06:04So far, we have already seen the User role.
  965. 1:06:06whenever we want to send a message as a user
  966. 1:06:10we specify its role as the user role
  967. 1:06:14the user-role represents what the user is asking okay
  968. 1:06:17but what about the assistant role? I've already said that
  969. 1:06:21that is the response that is generated by the LLM
  970. 1:06:26But I think it is better if I show you
  971. 1:06:30if I show you this in practice so here instead of printing
  972. 1:06:34the content the content of response I just print
  973. 1:06:38the role of the response and
  974. 1:06:40here if I
  975. 1:06:42run my program and
  976. 1:06:45say for example hello as you can see it says this message
  977. 1:06:48that we are printing has this role assistant role
  978. 1:06:52okay we have another role which is system role what
  979. 1:06:55is it used for what is the system role used for
  980. 1:06:59system messages are like instructions are
  981. 1:07:02like instructions that we give to the model
  982. 1:07:05for example when you are talking to chat gpt
  983. 1:07:09You might tell it, I mean during the day when
  984. 1:07:12you have conversation with AI chat, but you
  985. 1:07:15might tell, answer this question in a very natural way, in a
  986. 1:07:21very concise way, explain it using the simplest
  987. 1:07:25language that you can keep the answer
  988. 1:07:29really short. Summarize this for me.
  989. 1:07:31And these are examples of instructions that you specify for the model.
  990. 1:07:36And here in our program when we... want to specify these kinds of instructions,
  991. 1:07:41we need to use this kind of role,
  992. 1:07:45system role.
  993. 1:07:47Now let's move on to another parameter which is temperature
  994. 1:07:52What does temperature mean?
  995. 1:07:54Temperature is related to the model's creativity and randomness. In
  996. 1:07:59other words,
  997. 1:08:00how deterministic do we want the model to be?
  998. 1:08:03there are two well known parameters that can affect this behavior, I mean randomness
  999. 1:08:09one of them is temperature, another one is top P
  1000. 1:08:14They influence how likely the model is to
  1001. 1:08:17produce different responses to the same input.
  1002. 1:08:22In other words, they give us some control over how predictable or creative the model
  1003. 1:08:28is.
  1004. 1:08:29and as a general intuition most of the time you
  1005. 1:08:32will see developers working with temperature
  1006. 1:08:35you will use temperature most of time and we rarely change top peak
  1007. 1:08:40they behave differently under the hood but just as a generally intuition
  1008. 1:08:45and about temperature I should say it is
  1009. 1:08:49typically specified as a value between 0 and
  1010. 1:08:532
  1011. 1:08:53If you bring the temperature closer to zero, the model becomes more deterministic
  1012. 1:08:59and that means if you give it the same prompt multiple times
  1013. 1:09:04you are more likely to get very similar responses
  1014. 1:09:07for example, imagine we say tell me a story
  1015. 1:09:11With a very low temperature, the model will
  1016. 1:09:14tend to produce a more predictable response
  1017. 1:09:17But as you increase the temperature, the model becomes more creative
  1018. 1:09:21So you're more likely to get different responses each time to the same input
  1019. 1:09:26So you can think about it like this
  1020. 1:09:28Lower temperature, more
  1021. 1:09:30deterministic and predictable Higher temperature,
  1022. 1:09:33more varied and creative
  1023. 1:09:35The question is that, when do we want to have a deterministic model?
  1024. 1:09:39when we are dealing with factual tasks like coding tasks
  1025. 1:09:43in these cases we don't need creativity we don' need much creativity
  1026. 1:09:48we care about getting a reliable response a consistent response
  1027. 1:09:52so in these case we can lower the temperature
  1028. 1:09:55on the other hand if you are asking the model to write
  1029. 1:09:58something creative for us, brainstorm ideas to come up with different possibilities
  1030. 1:10:03in that case we can increase the temperature and in most cases
  1031. 1:10:07if we want something in between a value that you can keep in mind is 0.7
  1032. 1:10:14and especially for something like this we are developing a chatbot
  1033. 1:10:18and we want something in between not too creative, not too factual
  1034. 1:10:23okay now another parameter that you should know about is max tokens
  1035. 1:10:27so what does max tokens mean?
  1036. 1:10:29it sets a limit on how many tokens the model is allowed to generate
  1037. 1:10:33as I told you until now I've been using the word word
  1038. 1:10:37to make things easier to understand but in the world of LLMs
  1039. 1:10:42precise term is token and I will definitely talk about what tokens mean
  1040. 1:10:48what tokens are later for now let's just imagine that tokens are same as
  1041. 1:10:54words in our language anyway what does
  1042. 1:10:57this parameter actually mean and do
  1043. 1:11:00what it does is that it puts a ceiling on the amount of text that the model is
  1044. 1:11:07allowed to generate for example Imagine you ask the model to write a story.
  1045. 1:11:12The model might start generating a thousand word story,
  1046. 1:11:15but maybe you only want 100 words.
  1047. 1:11:19and one reason can be cost generated tokens have a
  1048. 1:11:22cost so you might want to control this process to
  1049. 1:11:25control how much the model is allowed to to generate
  1050. 1:11:28tokens for example you might set max tokens
  1051. 1:11:31to 100 but uh there is one important thing that
  1052. 1:11:34you need to know about this this parameter it's
  1053. 1:11:38not a graceful way to stop the model because
  1054. 1:11:41it's more like putting a hard limit let me give
  1055. 1:11:43you an example imagine the model it's gonna generate a story
  1056. 1:11:47this starts with something like this
  1057. 1:11:50once upon a time there was a bird that lived in the forest
  1058. 1:11:54every day would go out to find food and in order to
  1059. 1:11:59and it stops.
  1060. 1:12:01And here the model suddenly hits that limit that we gave it.
  1061. 1:12:07So the model gets stopped
  1062. 1:12:09here.
  1063. 1:12:10It was actually going to continue, but we forced it to stop.
  1064. 1:12:15So when you are setting this
  1065. 1:12:17token, keep in mind that it is a hard limit.
  1066. 1:12:19It doesn't guarantee that the ending is meaningful.
  1067. 1:12:21It may not naturally finish its thought within that limit. And another problem
  1068. 1:12:27parameter is streaming. What about the streaming?
  1069. 1:12:30Normally, when we send a request to the provider
  1070. 1:12:33like what we did in the last session,
  1071. 1:12:35the model generates a response, a complete response, and it is returned to us.
  1072. 1:12:41So we have to wait, we have wait, but this is not similar to this
  1073. 1:12:45experience that we have with ChatGVT, Groq and these kinds of chatbots.
  1074. 1:12:50Instead, the response is generated piece by piece.
  1075. 1:12:53Why? Because users are used to it.
  1076. 1:12:55Users hate staring at a screen, waiting several seconds without saying anything.
  1077. 1:13:02So instead, it feels much better, more natural if
  1078. 1:13:05the response starts appearing piece by peace.
  1079. 1:13:09OK, and we see what is generated immediately by now.
  1080. 1:13:13This way the model generates some text and we
  1081. 1:13:17display it and it generates more text and
  1082. 1:13:20we continue displaying it and that is what streaming is about and allows us to do.
  1083. 1:13:26We will cover that, we will cover streaming in our chatbot project.
  1084. 1:13:30I will show you how to do that.
  1085. 1:13:32And these were the important parameters that you need to know.
  1086. 1:13:37you need to be familiar with. And in the next session, we will add the conversation.
  1087. 1:13:51Now let's make our application more interesting.
  1088. 1:13:54Right now our program works like this.
  1089. 1:13:56Every time I run the program, I type something,
  1090. 1:13:59I get a response and the program terminates.
  1091. 1:14:01But that's not the experience similar to what you have in everyday life.
  1092. 1:14:08You open a chatbot, you say something and then you
  1093. 1:14:11again have an opportunity to ask something else.
  1094. 1:14:15And here we
  1095. 1:14:15need to create a loop, we need add a loop.
  1096. 1:14:18to have the similar experience
  1097. 1:14:22and that is exactly the kind of scenario in programming that we need a loop
  1098. 1:14:29and the question is what kind of loop do we need a while loop here because
  1099. 1:14:33we want this process to continue for as long as the user wants
  1100. 1:14:39the user want to terminate and we terminate the program so here in my
  1101. 1:14:46main function, I'm going to add a while loop
  1102. 1:14:50and as its condition I put true because I want this while loop to run indefinitely
  1103. 1:14:57and
  1104. 1:14:57whenever the user decides they don't want to continue
  1105. 1:15:01we will use the break statement to jump out of the loop
  1106. 1:15:05now here I'm going to move this line of code that gets an input from the user
  1107. 1:15:12and I just want to change its label a little
  1108. 1:15:16bit I say type
  1109. 1:15:19Q
  1110. 1:15:21to
  1111. 1:15:22terminate.
  1112. 1:15:24Okay, here I add a colon and now everything is perfect for the label.
  1113. 1:15:28Okay, the next step I want to clean up.
  1114. 1:15:30I want clean up what the user enters by using two methods.
  1115. 1:15:37The first is strip
  1116. 1:15:40and the other one is lower what is a strip for a strip
  1117. 1:15:44for removing unnecessary spaces from the
  1118. 1:15:47beginning or from the end the user might enter
  1119. 1:15:51something like this space space space hello
  1120. 1:15:55space space and by adding this method these spaces
  1121. 1:15:59are removed okay so this one for a street and the another one a lower it converts
  1122. 1:16:06all letters to lowercase for example if user instead
  1123. 1:16:10of q types q uppercase q it is converted to
  1124. 1:16:14lowercase q and this means that our program doesn't
  1125. 1:16:18have to care about capitalization anymore
  1126. 1:16:21and these were these two methods next we need an if statement because
  1127. 1:16:26if we don't put any if statement here our y loop is going to
  1128. 1:16:31continue forever but we need an if statement and check if what the user has
  1129. 1:16:36entered this queue we want to terminate the program so I say if
  1130. 1:16:41user input
  1131. 1:16:43is equal to
  1132. 1:16:44Q, in that case, print.
  1133. 1:16:46Good luck.
  1134. 1:16:51and then jump out of this loop I use break statement for this
  1135. 1:16:56okay and then I want to keep the rest of my code
  1136. 1:17:00but I move it inside this while loop okay now we have a conversation loop
  1137. 1:17:06so what happens
  1138. 1:17:08now on every iteration
  1139. 1:17:11we get an input and then we check we check if it's Q
  1140. 1:17:18we print good luck and
  1141. 1:17:19exit the loop and finally exit the program and
  1142. 1:17:24if it's not Q we send that input for our LLM.
  1143. 1:17:29And
  1144. 1:17:30and a response arrives we display it on the screen on the
  1145. 1:17:33screen and we give the user another opportunity
  1146. 1:17:36another opportunity to to enter something else and
  1147. 1:17:40this way we have created a continuous continuous
  1148. 1:17:43loop and now let's run the program let's test it
  1149. 1:17:47so i run the programme
  1150. 1:17:50it says something for the first
  1151. 1:17:52scenario i type
  1152. 1:17:54cue as you can see it works correctly
  1153. 1:17:58And now I type hello.
  1154. 1:18:01okay it says hello how can I assist you today and
  1155. 1:18:05you can see I still can continue asking something else okay for example
  1156. 1:18:13What is a
  1157. 1:18:15capital of
  1158. 1:18:17France
  1159. 1:18:19and as you can see it answers and we successfully
  1160. 1:18:23implemented what we wanted to implement
  1161. 1:18:27you
  1162. 1:18:33okay in this session first i'm going to do a
  1163. 1:18:35little bit of refactoring and the reason is that
  1164. 1:18:39if we keep putting all new features in our main
  1165. 1:18:42function it's going to get messy and that is not
  1166. 1:18:45a good approach it it makes our application less
  1167. 1:18:49maintainable and less extensible so let's do that
  1168. 1:18:54first i take this while loop here i cut it
  1169. 1:18:57and i move it into its own its own function
  1170. 1:19:00So I define a new function called chat
  1171. 1:19:04and I paste this piece of code here
  1172. 1:19:07okay, now I go to the main function
  1173. 1:19:10and I call this chat function that I just defined
  1174. 1:19:14so let's run our program to make sure it is working like before
  1175. 1:19:18okay, my visual environment is activated one more time and now I type hello
  1176. 1:19:23as you can see my application is working correctly
  1177. 1:19:27now I'm going to talk about something very very important
  1178. 1:19:32and that is conversation history. Let me show you an example. Here I ask what is
  1179. 1:19:39the capital of France
  1180. 1:19:41and it answers the capital
  1181. 1:19:44of France is Paris now I ask what is
  1182. 1:19:48It's population.
  1183. 1:19:52As you can see, it says I'm not sure what you're talking about
  1184. 1:19:57and it doesn't have any information about the subject but why is that?
  1185. 1:20:03Here I'm asking what is the capital of France
  1186. 1:20:05so it's obvious that this it that you see here is referring to France
  1187. 1:20:10but the model doesn't know what's happening here.
  1188. 1:20:14Ladies and gentlemen, we've just experienced a small communication glitch.
  1189. 1:20:19You may be feeling a little anxious right now. That's completely understandable.
  1190. 1:20:25As you've just seen, large language models have no memory.
  1191. 1:20:28Every message reaches them as if it were the very first.
  1192. 1:20:32To keep a conversation going, we have to remember it for them
  1193. 1:20:36and send the history along with every new message.
  1194. 1:20:39But don't worry, your captain will show you exactly how it's done.
  1195. 1:20:45At this point you should know that LLMs
  1196. 1:20:48don't automatically have memory.
  1197. 1:20:50And the way an LLM works is basically like
  1198. 1:20:54this. A query comes in, the model generates a response,
  1199. 1:20:58and it's over. Okay? It doesn't save any information.
  1200. 1:21:03and when the next query comes in, it's totally new.
  1201. 1:21:07The model doesn't have any access to the previous
  1202. 1:21:10messages, previous conversation, and it remembers
  1203. 1:21:13nothing. So, if I ask, like this scenario, what is the capital of France, and
  1204. 1:21:19it answers Paris, and then I ask a new question, what
  1205. 1:21:23is its population, the model doesn t know, doesn t inherently know what that
  1206. 1:21:30it refers to that's a whole point of what i'm
  1207. 1:21:33going to talk about and you should know that
  1208. 1:21:36conversation history doesn't happen at model level
  1209. 1:21:39it happens at application level if you want
  1210. 1:21:42history you need to use your software knowledge
  1211. 1:21:46to add this feature to your application and here
  1212. 1:21:49you're looking at an actual AI engineering task that means
  1213. 1:21:53we are going to use our software engineering knowledge
  1214. 1:21:58we are going to use the software engineering concepts
  1215. 1:22:01to solve a problem that is related to AI models and what do we do?
  1216. 1:22:05Our LLM doesn't have any memory and we are gonna solve that how can we do that?
  1217. 1:22:12Remember
  1218. 1:22:13The LLM receives an input and produces an output.
  1219. 1:22:20what we can do is to store the conversation history and send it with our query
  1220. 1:22:26and when we do that we have history okay so what
  1221. 1:22:30happened we restore conversation history somewhere
  1222. 1:22:34and now whenever we want to send a new query we send
  1223. 1:22:37that history along with our along with that query
  1224. 1:22:42and now you can understand you can understand it better
  1225. 1:22:45why we have this list here instead of just one message here instead
  1226. 1:22:50of just passing one query we can we can pass previous messages
  1227. 1:22:55previous questions and answers and when we do that
  1228. 1:22:59the model will have enough information about that topic
  1229. 1:23:03you
  1230. 1:23:09Okay, in this session we are going to see how
  1231. 1:23:13we can add conversation history to our code
  1232. 1:23:16and honestly this is very, very easy to implement and logic is actually quite simple
  1233. 1:23:22and the process is at first we need to create
  1234. 1:23:25an empty list and on each iteration we need
  1235. 1:23:29to add the user's query to it and after receiving the response we need
  1236. 1:23:34to add response as well.
  1237. 1:23:36and whenever we want to send a new query we
  1238. 1:23:39need to send that new query plus the history
  1239. 1:23:42so let's do it first up here i mean
  1240. 1:23:46below this llm global variable i'm going to
  1241. 1:23:50define a new
  1242. 1:23:53a variable called messages and I set it to
  1243. 1:23:56and empty list now
  1244. 1:23:58here we should add what the user has asked so here i say messages that append
  1245. 1:24:06It's going to be
  1246. 1:24:08a new message
  1247. 1:24:10Let's copy it from here.
  1248. 1:24:12paste it here okay that is exactly what we need but
  1249. 1:24:17here instead of passing this list i'm going to pass
  1250. 1:24:22this list that I defined up here
  1251. 1:24:25okay now after
  1252. 1:24:26receiving response before returning it to the main function
  1253. 1:24:31do the
  1254. 1:24:32chat function more accurately
  1255. 1:24:36I'm going to store it in a variable called
  1256. 1:24:39response.
  1257. 1:24:41Text.
  1258. 1:24:44Okay, and
  1259. 1:24:46Finally, I'm going to return it.
  1260. 1:24:49Return response text.
  1261. 1:24:51and before that I'm going to add it to this
  1262. 1:24:55list that we have here
  1263. 1:24:57so I say messages.pend
  1264. 1:25:01and here I'm going to have a message
  1265. 1:25:05we need to follow this structure. So
  1266. 1:25:08let me copy it from here and paste it here.
  1267. 1:25:11However, I need to change the role to assistant.
  1268. 1:25:16And instead of message, I should say response text.
  1269. 1:25:21This is what is happening here. First, I have an empty array, an empty list.
  1270. 1:25:28This is my history.
  1271. 1:25:30and on every iteration first i add the query to it
  1272. 1:25:33here i receive the response i extract its text
  1273. 1:25:37and then i'm going to add it to to that history
  1274. 1:25:41and every time that i'm sending the query i'm
  1275. 1:25:45going to pass the whole history okay that is a
  1276. 1:25:49point and that's it let's test our program okay
  1277. 1:25:53Virtual environment is activated. One more time, I ask hello.
  1278. 1:25:57And now I ask what is the capital of France.
  1279. 1:26:03And finally, I ask, what is its population?
  1280. 1:26:09okay we are getting the correct answer and now it's talking about Paris so
  1281. 1:26:15we have successfully implemented a conversation history for our program
  1282. 1:26:21you
  1283. 1:26:27okay in this session i'm going to talk about the
  1284. 1:26:29system role so far we have talked about the user
  1285. 1:26:32role the assistant role but we haven't talked about
  1286. 1:26:35this system role yet the system rule is used
  1287. 1:26:38to define a set of instructions for the model for
  1288. 1:26:41example before we send the rest of messages we can
  1289. 1:26:44tell the model something like this we can say
  1290. 1:26:47you are an assistant respond in this way
  1291. 1:26:50use this format follow these instructions and this
  1292. 1:26:54role is basically for giving instructions and
  1293. 1:26:56usually we put this type of messages at the
  1294. 1:27:00beginning of this list uh your sync here before
  1295. 1:27:03before the rest of the conversation so
  1296. 1:27:06to add a system message to this conversation
  1297. 1:27:10here instead of having an empty list i can add a
  1298. 1:27:14system message so i add a new message the format should be the same
  1299. 1:27:19we have a role and we have
  1300. 1:27:21content so
  1301. 1:27:23Here I say system
  1302. 1:27:25role.
  1303. 1:27:27and my content can be something like you are a helpful assistant you are
  1304. 1:27:34helpful.
  1305. 1:27:36Assistant
  1306. 1:27:38Your name is
  1307. 1:27:40Neonbot?
  1308. 1:27:43And I say
  1309. 1:27:46introduce
  1310. 1:27:48yourself in the first.
  1311. 1:27:51message and in addition I say give
  1312. 1:27:57short and helpful answers.
  1313. 1:28:02Okay, great. And here I'm going to use triple quotes.
  1314. 1:28:08to have it in multiple lines okay great and let's test our application
  1315. 1:28:16I say let's terminate it first I run it one more time
  1316. 1:28:22Okay, again.
  1317. 1:28:24I say hello
  1318. 1:28:26you can see it is introducing itself okay then I'm going to terminate our program
  1319. 1:28:32and here I'm gonna introduce a new topic and that is really important
  1320. 1:28:37as you can here by adding just a simple system message
  1321. 1:28:41we influenced the model's behavior a lot then that is extremely useful
  1322. 1:28:48and here i need to introduce you a new keyword that is foundation models
  1323. 1:28:55What are foundation models? These
  1324. 1:28:57models, these models that we are working with,
  1325. 1:29:00the models that you use in AI chat pods and these AI chat pod that we have now.
  1326. 1:29:06And why are they called foundation models, because they have a broad knowledge,
  1327. 1:29:12broad knowledge across a wide range of domains.
  1328. 1:29:15The point is that in most real world applications,
  1329. 1:29:19we are not going to use a foundation model because
  1330. 1:29:22we are not dealing with a wide range of tasks, a wide
  1331. 1:29:26range of domains, and we usually have a specific task, and we have a
  1332. 1:29:31specific use case in mind. So we need to somehow make the model more specific,
  1333. 1:29:37and we need adapt its behavior to our requirements.
  1334. 1:29:43And this is really important.
  1335. 1:29:45and you saw just one example of it
  1336. 1:29:48that you provided a system prompt and you changed the model's behavior
  1337. 1:29:53and more accurately we can say this falls under the category of prompt engineering
  1338. 1:29:59we change the prompt in order to get the result that we want
  1339. 1:30:05and this is the first solution when we want to change the model behavior
  1340. 1:30:10in many cases that is enough
  1341. 1:30:13can solve the problem.
  1342. 1:30:15and there are other solutions as well for example RAG systems
  1343. 1:30:19fine-tuning and we go to those solutions when prompt engineering is is not enough
  1344. 1:30:25in that case first we try RAG, retrieval
  1345. 1:30:28augmented generation
  1346. 1:30:30RAG is useful when the model doesn't have that
  1347. 1:30:34information that is necessary for that task
  1348. 1:30:37for example we are designing a chat pod for our company and of course
  1349. 1:30:42A foundation model doesn't have access to our company's documentation.
  1350. 1:30:46So we need to provide that knowledge, that documentation for that chatbot.
  1351. 1:30:52And we do this by using a RAG system.
  1352. 1:30:56and if neither of these approaches solves our
  1353. 1:31:00problem we go to fine-tuning and this is
  1354. 1:31:03more complex and it requires additional infrastructure it needs compute resources it
  1355. 1:31:11needs training data and we need to be specialized in this field so these
  1356. 1:31:15are three levels of
  1357. 1:31:16customization first we try prompt engineering and if it's not enough if we need two
  1358. 1:31:24provide external resources.
  1359. 1:31:26If we need to provide additional knowledge, we consider RAG.
  1360. 1:31:30and if the problem is behavioral, if the behavior
  1361. 1:31:34of the model is a problem, we go to fine-tuning.
  1362. 1:31:38And here, as you can see, prompt engineering was enough.
  1363. 1:31:42We just added a system prompt and in
  1364. 1:31:44many scenarios that's the case.
  1365. 1:31:46Some developers try to make it over complicated, but we shouldn't do that.
  1366. 1:31:53First, we should go to prompt engineering and if
  1367. 1:31:55it's not enough, we consider other solutions.
  1368. 1:31:58And here our goal was to create a branded feeling.
  1369. 1:32:03I mean, we want the user to feel it is chatting with a branded AI assistant.
  1370. 1:32:08And that was our goal. And as you can see, we have fulfilled this goal here.
  1371. 1:32:15Ladies and gentlemen, a quick note about the local language.
  1372. 1:32:19On Planet Chatbot, nobody speaks in words, they speak in tokens.
  1373. 1:32:24A token is a small piece of text, sometimes
  1374. 1:32:27a whole word, sometimes just part of one.
  1375. 1:32:30Before any message reaches the LLM, it's split
  1376. 1:32:33into tokens and every token becomes a number.
  1377. 1:32:37Tokens are also the planet's currency.
  1378. 1:32:40Every token you send and every token you receive has a cost.
  1379. 1:32:45So learning to count your tokens is an essential survival skill
  1380. 1:32:48your captain will now explain.
  1381. 1:32:58Okay, the next very important concept that you need to understand is tokens.
  1382. 1:33:03Up until now, I've sometimes referred to tokens
  1383. 1:33:07as just words, just to keep things simple.
  1384. 1:33:09But from this point forward, I want to use the correct terminology.
  1385. 1:33:14In the context of LLMs, the unit we use to
  1386. 1:33:17measure the input and output is called a token.
  1387. 1:33:21You can think of tokens as the basic unit of text that an LLM processes.
  1388. 1:33:26In natural human language, we think in terms
  1389. 1:33:30of words, but for LLMs, we work with tokens.
  1390. 1:33:34Now, why is it important? Why is it to understand this concept?
  1391. 1:33:38There are several different reasons.
  1392. 1:33:40One of the most important reasons is cost management.
  1393. 1:33:44The provider you are using charges you based on the number of tokens you use,
  1394. 1:33:48the number tokens you send to the model and the number the model generates.
  1395. 1:33:53Another important reason is performance analysis
  1396. 1:33:56For example, if you want to compare several models
  1397. 1:33:59decide which one is the best option for you, for your application
  1398. 1:34:03and in these scenarios, you need to understand how token usage works
  1399. 1:34:07So now let's see an example, let's what a token actually is
  1400. 1:34:12Imagine we send something like this to our LLM I love AI
  1401. 1:34:17Before the model processes this piece of text, it gets broken down into tokens
  1402. 1:34:22the sentence I told you I love AI
  1403. 1:34:24for a simple example it could be something like this
  1404. 1:34:28I one token love one token and AI another token but this is the important part
  1405. 1:34:34a token is not necessarily a complete word a token can be a complete world
  1406. 1:34:40part of a word or even a punctuation mark
  1407. 1:34:43for example you can consider
  1408. 1:34:46a word like interesting it could
  1409. 1:34:49something like this inter one token esting another token so
  1410. 1:34:56don't think of them as exactly as as the same thing as words
  1411. 1:35:01and i can say as a rough approximation you can consider one token
  1412. 1:35:06around three quarters of a word in English language
  1413. 1:35:10but remember this is just an approximation
  1414. 1:35:13one thing that you need to have in your mind is that
  1415. 1:35:17a token can be a whole word,
  1416. 1:35:19part of a word, or a punctuation mark
  1417. 1:35:21you might be wondering why do I keep using can be, could be
  1418. 1:35:26why don't I just tell you how exactly a word is going to be spill it into tokens
  1419. 1:35:31the reason is that different models use different tokenizers
  1420. 1:35:35and each model has its own vocabulary of tokens
  1421. 1:35:39and the model uses this vocabulary when it processes the text every LLM
  1422. 1:35:46has a vocabulary, a collection of tokens. And suppose I give the model this input.
  1423. 1:35:52I am.
  1424. 1:35:53And now the model needs to predict the word that comes next.
  1425. 1:35:57And the model looks at the context of the sentence and
  1426. 1:36:02considers possible next tokens from its vocabulary.
  1427. 1:36:07For example, possible candidates might be tired, happy, sad, and went.
  1428. 1:36:14and each of these possible next tokens gets a probability
  1429. 1:36:18for example, purely for illustration, purely for example I can say
  1430. 1:36:22tired gets 20%, happy gets 21%
  1431. 1:36:28sad gets 10% and went gets 0.00001%
  1432. 1:36:34these numbers are completely made up just for example
  1433. 1:36:38but in this example you saw that I gave a very low probability to went
  1434. 1:36:44because it doesn't fit grammatically
  1435. 1:36:47but it still have a probability and it might be chosen
  1436. 1:36:53and this collection of possible tokens is called a probability distribution
  1437. 1:36:58the model uses this probability distribution to select the next token
  1438. 1:37:03so in this way, HAPPY has a much higher probability in comparison with WENT
  1439. 1:37:10but the point is that WENT still has a chance
  1440. 1:37:14and it's not necessarily impossible and
  1441. 1:37:18uh this probabilistic process this probabilisting nature is one of the
  1442. 1:37:24reasons LLMs can behave in a non-deterministic way
  1443. 1:37:28can be creative in other words the exact details are of course much more
  1444. 1:37:33complicated more complicated that than i simplified it for you but uh
  1445. 1:37:38conceptually this is what's happening the model has a vocabulary of
  1446. 1:37:42possible tokens and it processes the context
  1447. 1:37:46performs a lot of mathematical calculations
  1448. 1:37:50assigns probabilities to possible next tokens
  1449. 1:37:53and then it selects one of those tokens you need to
  1450. 1:37:57remember the parameters i talked about temperature
  1451. 1:38:01and top p these parameters influence this process
  1452. 1:38:04at exactly this stage where the model is working
  1453. 1:38:08with this probability distribution and it's
  1454. 1:38:11deciding which token to generate next okay
  1455. 1:38:14Now there is one more important thing to understand.
  1456. 1:38:18The tokens we talked about aren't what the model actually receives.
  1457. 1:38:22Each token is mapped to a numerical representation, a numerical identifier.
  1458. 1:38:27And it is called a token ID.
  1459. 1:38:29For example, again purely for our example, I can be converted to 101.
  1460. 1:38:36Love can be convert to, for example, 409.
  1461. 1:38:40AI782 and just these examples.
  1462. 1:38:44So ultimately the text we provide, for example, I love AI,
  1463. 1:38:48is converted into an array of token IDs. Finally, that array of
  1464. 1:38:53token IDs is passed to the model and
  1465. 1:38:56The component responsible for doing that is called tokenizer.
  1466. 1:39:01Tokenizer maps a piece of text into tokenize these.
  1467. 1:39:06So the overall process again looks something like this. We have text.
  1468. 1:39:11Then we have tokenization process
  1469. 1:39:14We have tokens Then we token IDs
  1470. 1:39:17And these token IDs are sent to the LLM
  1471. 1:39:21Now, the good news for you is that you usually
  1472. 1:39:24don't need to work with the tokenizer directly.
  1473. 1:39:27In most AI engineering applications, you rarely have to manually tokenize your text.
  1474. 1:39:32And even if you decide to run a model locally, the
  1475. 1:39:36tokeniser usually comes with the model itself.
  1476. 1:39:39And it is provided alongside it, so you don't have to be worried about it.
  1477. 1:39:46So, for now,
  1478. 1:39:47the main thing I want you to remember is
  1479. 1:39:50your text goes through tokenization process
  1480. 1:39:54it gets converted into tokens and these tokens are mapped to
  1481. 1:39:58token ids and that's a basic process and as i mentioned earlier one of the
  1482. 1:40:03first places where understanding tokens becomes really important is cost
  1483. 1:40:08management for example if you look at here look at the open ai pricing
  1484. 1:40:13page you will notice that pricing is based
  1485. 1:40:16on tokens it's saying that i'm charging you
  1486. 1:40:19based on tokens, you might see something like this
  1487. 1:40:22$10 per 1 million token and $50 for output tokens
  1488. 1:40:28So, input tokens are the tokens contained in the text you're sending to the LLM
  1489. 1:40:34your input, your query, and of course, if you're designing your application
  1490. 1:40:39the history is taken into account
  1491. 1:40:42and the response that is generated, the tokens that are used by the LLM to generate
  1492. 1:40:48The response is called output tokens.
  1493. 1:40:51And as you can see here on this pricing page, output
  1494. 1:40:55tokens are more expensive than input tokens.
  1495. 1:40:58And as an AI engineer, tokens need to be important to you.
  1496. 1:41:02You need to take care of things like cost management,
  1497. 1:41:06model limits, context window, performance analysis.
  1498. 1:41:10All these things depend on keeping track of the number of tokens we use.
  1499. 1:41:16And from this point forward, whenever we talk
  1500. 1:41:19about the amount of text, the LLM processes,
  1501. 1:41:22I want you to think in terms of tokens, not just words.
  1502. 1:41:27Oh
  1503. 1:41:33When it comes to counting tokens, there are two
  1504. 1:41:35different places where we might want to count them.
  1505. 1:41:38One is before, when it's before we send the request
  1506. 1:41:41and the other is after we get the response.
  1507. 1:41:44Okay, so let's start with the latter one, with the second one.
  1508. 1:41:49After we receive a response from the LLM, we
  1509. 1:41:52might want to know how many tokens were used.
  1510. 1:41:54For example, we want to how many input tokens did we
  1511. 1:41:58use, how many output tokens did the model generate,
  1512. 1:42:02and finally how many token were used in total. Input tokens plus output tokens.
  1513. 1:42:08Okay, this is actually the easier case because you send a query to an LL,
  1514. 1:42:13the LLL returns that number, it returns a response and
  1515. 1:42:18in the response metadata, the API tells you how many tokens were used
  1516. 1:42:22so this is pretty straightforward, you only need to extract that number
  1517. 1:42:27and as I told you, the other case is when we want to count the number of tokens
  1518. 1:42:33before sending the request and there can be different reasons
  1519. 1:42:37maybe we want to check how many tokens we have before sending a request
  1520. 1:42:41maybe we wanna set a limit for example, in cases that the input is too large
  1521. 1:42:47we may want to prevent the user from sending that request
  1522. 1:42:51in these cases we need to know how many token we have before sending request
  1523. 1:42:58and this is a little more complicated
  1524. 1:43:01because as I mentioned, different models have different tokenizers
  1525. 1:43:05so to count the tokens before sending the request
  1526. 1:43:09we need to have that tokenizer and we need tokenize the text
  1527. 1:43:13so you have many additional steps
  1528. 1:43:16you need to take your text, pass it through the proper tokenizer
  1529. 1:43:21and then after tokenizing it you need to count the number of
  1530. 1:43:25tokens and store it somewhere and use it however you need
  1531. 1:43:29anyway in this session we are going to focus on the second case the easier one
  1532. 1:43:34I'm going to show you where this token where
  1533. 1:43:36this metadata is located in the response
  1534. 1:43:38and then we are gonna keep track of the number tokens
  1535. 1:43:43that is consumed throughout our conversation okay
  1536. 1:43:47and later I will show you how we can calculate the actual cost
  1537. 1:43:52because it was one of the reasons that we wanted to keep track of used tokens
  1538. 1:43:58and at that point I will you how to keep input and output tokens separately
  1539. 1:44:04because the cost is different for input tokens and output token
  1540. 1:44:08in this session we only want to keep the total
  1541. 1:44:12tokens that is used during our conversation
  1542. 1:44:15So first I need to have a variable, have a
  1543. 1:44:20variable to keep track of this number.
  1544. 1:44:23to add the number of used tokens on each iteration of our conversation loop.
  1545. 1:44:29So here I'm going to add a new global variable alongside our other global variables,
  1546. 1:44:35and I'm gonna call it total tokens.
  1547. 1:44:41and I set it to zero.
  1548. 1:44:44now here inside our conversation loop here when we
  1549. 1:44:48get the response i'm going to add the number of
  1550. 1:44:51tokens the number tokens used to this variable
  1551. 1:44:55so here that we are getting the response there
  1552. 1:44:59is a field inside called usage
  1553. 1:45:01and inside that we have total tokens and it represents the number
  1554. 1:45:05of tokens that are used in total so every time we receive the response from the llm
  1555. 1:45:11We extract this value and we add it.
  1556. 1:45:14So here, before returning the response, I say total tokens
  1557. 1:45:20is equal to total tokens plus this number.
  1558. 1:45:25response.usage.
  1559. 1:45:29total tokens and as you can see it is suggesting that
  1560. 1:45:32now I need to tell you that we have a small programming issue here
  1561. 1:45:37Python sees that we are assigning a value to total tokens inside this function.
  1562. 1:45:45and by default
  1563. 1:45:46Python assumes that total tokens is a local variable
  1564. 1:45:51something different from this total tokens that we have up here
  1565. 1:45:55what happens is that on the right hand side we are trying to access total tokens
  1566. 1:46:00before this local variable has been assigned a value
  1567. 1:46:04okay that is a problem
  1568. 1:46:06and that's the reason
  1569. 1:46:08Python will complain about it and this is one of the reasons
  1570. 1:46:12it is told that using global variables is not a good approach.
  1571. 1:46:18But later when we refactor our application, when we make our
  1572. 1:46:23application object oriented, we will solve this problem properly.
  1573. 1:46:27But for now, as a temporary solution, we can explicitly tell Python that we
  1574. 1:46:34want to use the global variable that we defined up here.
  1575. 1:46:39And to do that, it's enough to say
  1576. 1:46:42global before our global variable and this way.
  1577. 1:46:47Python won't create a new local variable called total tokens inside this function.
  1578. 1:46:52Instead, total tokens
  1579. 1:46:54inside this functions refers to the same global
  1580. 1:46:58total tokens variable that we defined earlier.
  1581. 1:47:01Okay, now I want to use it.
  1582. 1:47:03I want print the total number of consume tokens
  1583. 1:47:07after the user decides to end the conversation.
  1584. 1:47:11So after the chat function is called,
  1585. 1:47:15I print and here I use a formatted string and I say consumed tokens
  1586. 1:47:21colon space and here
  1587. 1:47:24I use the placeholder and I put
  1588. 1:47:27total tokens inside. Now let's run our program.
  1589. 1:47:32Okay, I say hello.
  1590. 1:47:34What is the capital?
  1591. 1:47:40France.
  1592. 1:47:42And now I terminate the program and as you can see, the program tells
  1593. 1:47:47us the total number of tokens we consumed during the conversation.
  1594. 1:47:52So now we are not only having a conversation with the LLM, we
  1595. 1:47:56are also tracking how many tokens that conversation is using.
  1596. 1:48:02Oh
  1597. 1:48:07Okay, as you may have realized, our program is growing.
  1598. 1:48:11We are adding new features to it. And if we continue this pattern, I mean, if we
  1599. 1:48:16continue growing the application with the same architecture,
  1600. 1:48:19it's eventually going to become difficult to maintain.
  1601. 1:48:22And you actually saw an example of it in the previous session.
  1602. 1:48:26We reached a point where even though we didn't really want to,
  1603. 1:48:30we had to use a global variable. So now,
  1604. 1:48:33Let's move on to the proper solution and that
  1605. 1:48:37solution is object-oriented programming.
  1606. 1:48:40and at the scale of this project object-oriented programming is enough
  1607. 1:48:45this structure solely is enough, it's more than enough to solve our problem
  1608. 1:48:50and give us the structure that we need and good for our program
  1609. 1:48:54So let's start prefactoring our application. First, I'm going to add a class here.
  1610. 1:49:01I add a new class called
  1611. 1:49:03Chatbot
  1612. 1:49:05and inside I'm going to
  1613. 1:49:07define a constructor for it. So I say def
  1614. 1:49:12in it and
  1615. 1:49:13here we go and here as you can see we have three
  1616. 1:49:19global variables and these were for
  1617. 1:49:23these were our previous implementation and in new
  1618. 1:49:27implementation we need to take these three variables
  1619. 1:49:31and we want to move them inside this constructor and we
  1620. 1:49:36are going to make them the attributes of this class so
  1621. 1:49:41let me fix the problem with indentation now it is okay
  1622. 1:49:46and the only thing left is to add self
  1623. 1:49:50at the beginning of each one oneself here another
  1624. 1:49:54one and another one here now
  1625. 1:49:57i can access these
  1626. 1:49:58attributes from the other methods that i'm going to define inside this class
  1627. 1:50:03what other methods do we need exactly the same functionality we had
  1628. 1:50:07we want that we had a bot function and we had the chat function
  1629. 1:50:13okay what's wrong here
  1630. 1:50:15all messages. Now it's okay.
  1631. 1:50:17It's complaining about that we don't have such a
  1632. 1:50:20variable and that's great because you're refactoring.
  1633. 1:50:25Okay, we have a bot function here and we have the chatbot function here.
  1634. 1:50:28And we are going to move these two functions inside
  1635. 1:50:31our class and make them methods of this class.
  1636. 1:50:35So first I do that for the bot method.
  1637. 1:50:44Okay, let's
  1638. 1:50:46fix the problem
  1639. 1:50:48again with indentation okay now it's okay
  1640. 1:50:52okay I'm
  1641. 1:50:52not going to need this line anymore
  1642. 1:50:55we did all of
  1643. 1:50:56it to get rid of this global variable
  1644. 1:51:00okay and I add self here
  1645. 1:51:04And one here.
  1646. 1:51:05places that it's complaining self.
  1647. 1:51:09dot oh we need to put
  1648. 1:51:13this self as a first parameter of this method as well oneself here
  1649. 1:51:19ok and
  1650. 1:51:21I guess now it's okay.
  1651. 1:51:24yes we need another one here because here it's trying to define
  1652. 1:51:30a
  1653. 1:51:31local total tokens variable here so
  1654. 1:51:34we don't want that so
  1655. 1:51:38and instead of this syntax I use plus equal. Now it's much better.
  1656. 1:51:44Okay, after this I'm going to move
  1657. 1:51:48the chat function inside this class, okay?
  1658. 1:51:53ok the indentation is ok and now it's a method
  1659. 1:51:57the only thing is to add self keyword here and here
  1660. 1:52:03we need to use self before calling this method of the class
  1661. 1:52:08okay now
  1662. 1:52:09it's perfect
  1663. 1:52:11okay now we have these methods inside our class bot method and chat method
  1664. 1:52:19bot handles communication with the LLM and chat handles the conversation loop
  1665. 1:52:24now to actually use these methods first we need to create an object we need
  1666. 1:52:31to create a object from our chatbot class so
  1667. 1:52:35inside the main function where it is complaining about
  1668. 1:52:39this function that no longer exists
  1669. 1:52:44instead of this line we want to create
  1670. 1:52:48an object from this way so I say chatbot here
  1671. 1:52:52parenthesis and now we have an object and I store it in a
  1672. 1:52:56variable called bot. Now I can access and now I have an
  1673. 1:53:02object that I can call these methods on it these two methods that I just defined
  1674. 1:53:10OK, here, messages again, we need a self.
  1675. 1:53:14Save that messages.
  1676. 1:53:17okay yes as i told you i call the chat method on this object and my program
  1677. 1:53:26will work like before hopefully we will test it and we will see
  1678. 1:53:32Now the only thing left is that we still have one line
  1679. 1:53:36here and this is from our previous implementation
  1680. 1:53:40and we no longer need such a variable.
  1681. 1:53:43We no longer have this total tokens here and it is complaining about this matter.
  1682. 1:53:50So we need to do this somehow else.
  1683. 1:53:53I mean rather than keeping this logic outside the class it's better to add a
  1684. 1:53:59new method in our chatbot class whose job is to generate a report.
  1685. 1:54:03By doing this we can also follow an important
  1686. 1:54:07principle in software engineering better.
  1687. 1:54:10This principle is called separation of concerns.
  1688. 1:54:15And the idea is that each part of our application
  1689. 1:54:18should have a clear responsibility.
  1690. 1:54:21For example, here as I told you we have chat
  1691. 1:54:24method that handles the conversation and
  1692. 1:54:27We have bot method that handles the communication.
  1693. 1:54:31And our new report method is going to handle reporting and observability.
  1694. 1:54:38Observability is very important. It's very important in real world applications.
  1695. 1:54:42Whether you are developing the system in the development phase,
  1696. 1:54:47whether you are running it in production.
  1697. 1:54:49In both cases, you want to be able to understand
  1698. 1:54:53what's happening inside your system.
  1699. 1:54:55What's going on.
  1700. 1:54:56so that's really important and you want to be able
  1701. 1:55:01to inspect things like requests responses errors
  1702. 1:55:05token usage and other important metrics so that really important
  1703. 1:55:10and let's do it let's add a new method to this class whose responsibility is
  1704. 1:55:17reporting and observability so i call
  1705. 1:55:21it report i put uh i put self as its
  1706. 1:55:25first parameter and first I'm going to print
  1707. 1:55:29each message in our conversation history. So I say for
  1708. 1:55:34message in self that
  1709. 1:55:37messages
  1710. 1:55:39I'm going to print
  1711. 1:55:41the message
  1712. 1:55:44and after I'm going to print the number of tokens that is used so I just take it and
  1713. 1:55:53paste it here
  1714. 1:55:55Okay, and I put a self at the beginning. Great.
  1715. 1:56:00So now we have a dedicated method responsible for generating the report,
  1716. 1:56:05the token number and the messages that we have in conversation history.
  1717. 1:56:11And finally, we call this method after the conversation loop.
  1718. 1:56:16I say bot.
  1719. 1:56:18report okay,
  1720. 1:56:19so now the overall structure of our application is much better
  1721. 1:56:25it is much cleaner we have a chatbot class and it contains
  1722. 1:56:31three methods and the state of our
  1723. 1:56:34application, I mean the attributes that our application has
  1724. 1:56:39this is much, this is better because each method has a clear responsibility
  1725. 1:56:44and if somewhere needs to be
  1726. 1:56:47changed we go to that specific part of our
  1727. 1:56:50application and now we have a better foundation
  1728. 1:56:53for
  1729. 1:56:54continuing to add new features to our application
  1730. 1:56:57but before finishing this session let's
  1731. 1:57:01test our program to see whether it uh whether
  1732. 1:57:04it works like before or not so i say hello
  1733. 1:57:09What is it?
  1734. 1:57:10Capital of
  1735. 1:57:12France.
  1736. 1:57:14what is its
  1737. 1:57:16population
  1738. 1:57:18Okay, great. It works like before and I terminate the program and
  1739. 1:57:22yeah, the new functionality that we just added.
  1740. 1:57:26Conversation history is printed here.
  1741. 1:57:30messages with different roles, first user, then assistant, first user,
  1742. 1:57:34then assistant. And, um,
  1743. 1:57:37Finally, the number of used tokens is printed here, exactly what we wanted.
  1744. 1:57:44Okay, great.
  1745. 1:57:46you
  1746. 1:57:52okay in this session we are going to add another cool feature to our application
  1747. 1:57:57to the report method we just created at the end of the last session
  1748. 1:58:01and so far our report is able to count the total number of tokens we have consumed
  1749. 1:58:08now we want to take it one step further
  1750. 1:58:12and we want it capable of calculating the cost
  1751. 1:58:16we want calculate the cost as well
  1752. 1:58:19and for this example we will use some hypothetical
  1753. 1:58:23pricing of course if you are working on a real
  1754. 1:58:27project if you're working with a real model you
  1755. 1:58:30need to you need check the pricing and it depends
  1756. 1:58:33on your provider so it's a responsibility for you
  1757. 1:58:36you need find out how much you are charged per
  1758. 1:58:39million input tokens and per million output tokens the
  1759. 1:58:43first step that we need is to track these two separately
  1760. 1:58:47Currently, we are counting only total tokens,
  1761. 1:58:50but we need to keep track of input tokens and
  1762. 1:58:53output tokens separately because as you saw,
  1763. 1:58:57input tokens, and output tokens usually have
  1764. 1:59:00different prices.
  1765. 1:59:02So at the first step, I'm going to add two new attributes to the constructor.
  1766. 1:59:09Here I'm going to add a new
  1767. 1:59:11attribute called input tokens
  1768. 1:59:16And another one.
  1769. 1:59:17called
  1770. 1:59:19output
  1771. 1:59:22tokens
  1772. 1:59:23and I set it to 0 as well.
  1773. 1:59:26Okay now when we get the response from the LLM
  1774. 1:59:29we need to keep track of the input and output
  1775. 1:59:32tokens as well so here
  1776. 1:59:37when we are receiving total tokens let me just duplicate this line
  1777. 1:59:44two times and
  1778. 1:59:46modify the left hand side. I
  1779. 1:59:48change it to Input Tokens.
  1780. 1:59:52and I change this one to output tokens the attribute the attributes we just defined
  1781. 2:00:00and in these cases pay close attention i mean when you copy paste
  1782. 2:00:05always pay close the attention it's very easy to accidentally to accidentally
  1783. 2:00:10introduce a small mistake to a program so you should be really cautious
  1784. 2:00:14now the question is that which attribute
  1785. 2:00:17inside response.usage contains the number of
  1786. 2:00:21input tokens and output tokens as well for input tokens it's prompt tokens
  1787. 2:00:27So here we should say prompt tokens.
  1788. 2:00:31And of course, the exact attribute names can be different depending on the
  1789. 2:00:35wrapper or the library you are using.
  1790. 2:00:37But in the OpenAI wrapper, the number of
  1791. 2:00:40tokens sent to the model is available through
  1792. 2:00:44prompt tokens and usually you don't need to memorize it.
  1793. 2:00:47And after
  1794. 2:00:48using a wrapper or a library, you will get used to it.
  1795. 2:00:52So don't worry about it.
  1796. 2:00:54and for output tokens we have completion
  1797. 2:00:58completion tokens in other words these are the
  1798. 2:01:02tokens that were generated to produce your response
  1799. 2:01:06now we have everything that we need to calculate the
  1800. 2:01:10cost and as i told you we are going to use some
  1801. 2:01:14made up pricing
  1802. 2:01:15let's assume for this example that we are charged five cents five cents per
  1803. 2:01:22million input tokens and eight cents
  1804. 2:01:24twice i mean
  1805. 2:01:26per million output tokens. So we can calculate the cost like this.
  1806. 2:01:32Here I say self.input
  1807. 2:01:36tokens times
  1808. 2:01:390.04 I said 4 or 5, okay
  1809. 2:01:44it's not important, 5, ok plus self
  1810. 2:01:50Output tokens times
  1811. 2:01:530.08 Okay, a
  1812. 2:01:55tiny fraction.
  1813. 2:01:58And we want to print this. So we say print
  1814. 2:02:04and I put it inside and let's add a dollar sign before
  1815. 2:02:10so I use a formatted string
  1816. 2:02:13Okay.
  1817. 2:02:16And here I add dollar sign and
  1818. 2:02:19This needs to be inside a placeholder. Great.
  1819. 2:02:24But there is one important detail here.
  1820. 2:02:26Remember that these rates are per 1 million tokens. So we need to convert it.
  1821. 2:02:31We need to divide it by 1 million.
  1822. 2:02:35So I put the whole expression inside parentheses.
  1823. 2:02:41And then I divide it by 1,000,000.
  1824. 2:02:45Okay, 6.
  1825. 2:02:47And to increase its readability, I add underscores here.
  1826. 2:02:52I'm using these underscores to simply make it easier to read.
  1827. 2:02:57Python allows these unders scores inside numeric literals.
  1828. 2:03:03So this is exactly the same as writing what we had before.
  1829. 2:03:11It's just for reading your number better.
  1830. 2:03:14And now, we need nothing else.
  1831. 2:03:16Let's just
  1832. 2:03:17test our program. I run it one more time.
  1833. 2:03:21Hello?
  1834. 2:03:23How are you?
  1835. 2:03:26OK and it
  1836. 2:03:28enter quit,
  1837. 2:03:30press and as you can see the cost here is calculated for us
  1838. 2:03:35and as we can see, the result is displayed in scientific notation
  1839. 2:03:41this negative e
  1840. 2:03:44This simply means the number is extremely small.
  1841. 2:03:47In this example, we are only using a few tokens.
  1842. 2:03:50So the actual cost is just a tiny fraction of a cent, which is really, really low.
  1843. 2:03:57you
  1844. 2:04:02Now let's move on to the first approach, to the first approach I mentioned.
  1845. 2:04:06I mentioned that there are two places we want to count the token number.
  1846. 2:04:11One of which is before, before we send the user's query to the LLM and one is after.
  1847. 2:04:17In the two previous sessions, we covered the second one.
  1848. 2:04:21And now we want cover the first one, the situation when we want count
  1849. 2:04:26the tokens before we send the user's query to the LLM.
  1850. 2:04:29And there are certain situations we need that.
  1851. 2:04:32We need to track that, track that number.
  1852. 2:04:35And the important question is how can we take a
  1853. 2:04:38piece of text and break it down into tokens.
  1854. 2:04:41As you remember, I mentioned that every model has its own tokenizer.
  1855. 2:04:45For example, OpenAI has its tokenizer, okay.
  1856. 2:04:48another model may use a different tokenizer and currently we are using
  1857. 2:04:53an open AI model so we are going to use the open AI tokenizer but
  1858. 2:04:59what if what if we were using a different model of
  1859. 2:05:02course in that case we would we would need that's
  1860. 2:05:05model tokenizer and we would to go to hugging face
  1861. 2:05:09find the tokenizer associated with that model
  1862. 2:05:12and use it so for now for open AI models I use
  1863. 2:05:15approach appropriate for that I mean using
  1864. 2:05:18the tokenized are associated with open AI and later I will show you
  1865. 2:05:23how to find and access the tokenizer that is for another model that you might use
  1866. 2:05:28so now how can we do that for OpenAI?
  1867. 2:05:31if you're working with OpenAI models you can use TIKTOKEN
  1868. 2:05:34there is a package called tiktoken
  1869. 2:05:36you install it and you can tokenize any piece of text
  1870. 2:05:41so first let me check yeah I'm connected to the internet
  1871. 2:05:45so I'm going to install this new package open terminal and I say
  1872. 2:05:51pipenv.
  1873. 2:05:52install
  1874. 2:05:54tick token okay the package is installed and now
  1875. 2:05:59i'm going to import it so up here i import
  1876. 2:06:04tick token and then at the beginning of my chat method we are going to use it we are
  1877. 2:06:09going to tokenize the piece of text that the user has entered the tick token
  1878. 2:06:14package provides a function called get encoding
  1879. 2:06:17we need an encoding and which encoding
  1880. 2:06:20do we need for this example we are gonna use
  1881. 2:06:23cl100k base and the encoding depends on the
  1882. 2:06:27model the exact model from openai that that you are using so
  1883. 2:06:31up here at the beginning of this method i'm going to use this function
  1884. 2:06:37get encoding function and I'm going to pass what I said cl100k
  1885. 2:06:45base
  1886. 2:06:46okay and we need to keep the results somewhere so i store it in a variable called
  1887. 2:06:52encoding now this encoding object that you are seeing here
  1888. 2:06:57provides me with a method called encode and we can call encode
  1889. 2:07:01pass it the text that we want to tokenize and it will give
  1890. 2:07:05us a token ids and we only need to count them it gives us
  1891. 2:07:09an array and we just need to account the number of elements inside so here
  1892. 2:07:14Before sending that text to the bot method, I'm going to tokenize it.
  1893. 2:07:19So I say.
  1894. 2:07:22encode and I pass
  1895. 2:07:24user input
  1896. 2:07:26and here I just print it
  1897. 2:07:30So I print it
  1898. 2:07:31Now let's
  1899. 2:07:34Let's run the program.
  1900. 2:07:36Okay, I say, now instead of hello, let's say
  1901. 2:07:40which is the capital
  1902. 2:07:43of France.
  1903. 2:07:46and now as you can see before the response we are seeing the token IDs
  1904. 2:07:53for example this token ID that you are seeing here is associated with what
  1905. 2:07:59and this is for is this is there
  1906. 2:08:02now instead of just seeing token IDs we want to know how many tokens we have
  1907. 2:08:08that's pretty easy as a result
  1908. 2:08:10that you are seeing here is just a regular Python list.
  1909. 2:08:13So we can use Python's built-in LEN function to
  1910. 2:08:17count the number of elements inside this list.
  1911. 2:08:20So let's terminate and
  1912. 2:08:22here instead of printing the list itself, I'm going to put it inside
  1913. 2:08:28the len function.
  1914. 2:08:31Okay, now let's run our program one more time. I say hello.
  1915. 2:08:37It says one. Okay, so we made it.
  1916. 2:08:40We can count the number of tokens before we send our request.
  1917. 2:08:44Now, the question is, the question is what is the benefit? What is the
  1918. 2:08:48benefit of knowing the number of tokens before sending the request to the LLM?
  1919. 2:08:52There are actually several possible use cases.
  1920. 2:08:54For example, you can use this as a guard.
  1921. 2:08:58we could say if the number of tokens in the
  1922. 2:09:01user's input is greater than a certain limit
  1923. 2:09:03don't send a request to the LLM.
  1924. 2:09:06For example, let's say we only want to allow prompts within
  1925. 2:09:10a maximum of 10 tokens. For instance,
  1926. 2:09:14So let's do that. First, I'll store this number in a variable called token count.
  1927. 2:09:22Okay.
  1928. 2:09:23Great. And then we can
  1929. 2:09:27add an if statement so i say if
  1930. 2:09:30token count is greater than 10. if it exceeds the limit
  1931. 2:09:37i print something like this i can say your prompt
  1932. 2:09:42exceeded the limit.
  1933. 2:09:46and can say try
  1934. 2:09:49a shorter one.
  1935. 2:09:51please and after exceeding this limit we don't want the program to continue and
  1936. 2:09:57send this prompt to the LLM with this implementation
  1937. 2:10:01it does but we don' t want to do that so here
  1938. 2:10:05we need to use the continue keyword here I say continue
  1939. 2:10:10and by doing this it moves to the next iteration
  1940. 2:10:14and this gives the user another opportunity to enter a shorter prompt
  1941. 2:10:19so let's run our program and test it
  1942. 2:10:23okay here I say hello
  1943. 2:10:24I didn't exit the limit now let me add some gibberish
  1944. 2:10:32to shortly test it. Something more than 10 tokens.
  1945. 2:10:35Okay, now it's definitely more than 10 tokens, okay. As you can see, it says here,
  1946. 2:10:40prompt exceeded the limit and
  1947. 2:10:42That was exactly what we wanted.
  1948. 2:10:45Oh
  1949. 2:10:51Okay, in the previous session, our model
  1950. 2:10:54happened to be from OpenAI, from Open AI family
  1951. 2:10:57and I showed you how to use tiktoken to tokenize your text.
  1952. 2:11:01But what happens if the model we're using isn't OpenAI? Isn't from OpenAi family.
  1953. 2:11:06For example, what if it's Llama or any other model you found on HuggingFace?
  1954. 2:11:11In these situations, the first approach is to relax our requirements a little bit.
  1955. 2:11:17It's true that different models use different tokenizers,
  1956. 2:11:20but in many cases, the difference is not significant.
  1957. 2:11:23It's negligible. For most tasks, we don't really need the exact token number.
  1958. 2:11:29You only need an approximation.
  1959. 2:11:31For example, the OpenAI tokenizer might tell you a
  1960. 2:11:34piece of text contains 10 tokens, for example,
  1961. 2:11:38while the actual tokenizer for your model you're using might count.
  1962. 2:11:448 tokens instead of 10 tokens or 12 tokens so it's not significant
  1963. 2:11:48and that's perfectly fine for most cases so if your task is not sensitive
  1964. 2:11:54if you don't need the exact token number you can simply use TIKTOKEN
  1965. 2:11:58because it's convenient and
  1966. 2:12:00it's easy to work with
  1967. 2:12:02but let's assume that for whatever reason you really need access the tokenizer
  1968. 2:12:08for the exact model you're using for example, let's say our model is Llama
  1969. 2:12:13and we need the exact number. So in these cases, what do we need to do?
  1970. 2:12:19Well, in these situations, first we need transformers,
  1971. 2:12:22and transformers is a library from HuggingFace.
  1972. 2:12:26Not only for this case, I mean tokenization, but for many other
  1973. 2:12:29things, for example, downloading a model, running a model.
  1974. 2:12:32We also need transformes for many tasks.
  1975. 2:12:36It's really worth becoming familiar with it.
  1976. 2:12:38And
  1977. 2:12:38one important thing here is that you need an access token.
  1978. 2:12:42I'm not completely sure whether an access token is
  1979. 2:12:45required here for this tokenization task or not
  1980. 2:12:48but for many other tasks it is required and in some cases it warns you
  1981. 2:12:53it warn you that you are not using a token
  1982. 2:12:56and if you use a token you can have higher
  1983. 2:12:58limits or better access and things like that
  1984. 2:13:01besides some models require you to be authenticated
  1985. 2:13:06otherwise you cannot access that
  1986. 2:13:09model uh it's really good
  1987. 2:13:11create your access token right now go to settings
  1988. 2:13:14create access token and have an access token
  1989. 2:13:16and i've already done that this is my access token
  1990. 2:13:20okay so for the reason i told you i copy this and
  1991. 2:13:24i put it inside my dot n file here i create another
  1992. 2:13:29environment variable called hf token and i set it to
  1993. 2:13:35my HuggingFaceAccessToken and make sure you use exactly this name
  1994. 2:13:39I mean hf underscore token because we are not
  1995. 2:13:42going to read this environment variable directly
  1996. 2:13:45we are NOT going to write it ourselves in our python code
  1997. 2:13:48the Hugging Face Transformers library is going to
  1998. 2:13:51read it and it does it automatically so the approach we read this token is not like
  1999. 2:13:56something like this os.get.env rather than the transformers is going
  2000. 2:14:01to do that so um
  2001. 2:14:04First, I'm going to install this library, this library that I told you.
  2002. 2:14:08So I say pipenv
  2003. 2:14:10install
  2004. 2:14:12transformers. Once that's installed we need to import it.
  2005. 2:14:17So I'm going to import AutoTokenizer class from a transformer. So I say from
  2006. 2:14:24Transformers import
  2007. 2:14:27Auto tokenizer
  2008. 2:14:30now instead of using tick token instead of
  2009. 2:14:33using tick to get here i'm i'm going to use
  2010. 2:14:36this auto tokenizer class that i just
  2011. 2:14:40imported impo i just imported and instead of
  2012. 2:14:44get encoding i'm used i'm gonna use from
  2013. 2:14:48pre-trained, okay, let's remove it
  2014. 2:14:51entirely.
  2015. 2:14:53from pre-trained
  2016. 2:14:55Okay, this method takes the name of the model you want to use.
  2017. 2:14:59So I head over to here, I copy the model name, I paste it and that's it.
  2018. 2:15:06and pay close attention the model identifier needs to be
  2019. 2:15:10the correct identifier that you copy from here
  2020. 2:15:13don't just guess it don't copy it from the provider you're using for example
  2021. 2:15:18Groq it needs to this exact identifier because you're
  2022. 2:15:22using transformers and transformers is from
  2023. 2:15:25hugging face so pay attention to that okay
  2024. 2:15:28now there is nothing left to change and here we can use it like before here i'm
  2025. 2:15:34I print it, I print.
  2026. 2:15:36token count.
  2027. 2:15:39Okay, I run my code to test it. Okay, as you can see, it displays a warning here.
  2028. 2:15:46It's telling that PyTorch should also be installed if you want to use the models.
  2029. 2:15:51But for now, for our purpose, that's perfectly fine
  2030. 2:15:54because we are not trying to load or run the model. We are just using the tokenizer.
  2031. 2:16:00We only want the tokenized. Let's say hello. And as you see, it's working correctly.
  2032. 2:16:07What is it?
  2033. 2:16:08Capital.
  2034. 2:16:10Oh, fuck.
  2035. 2:16:11France
  2036. 2:16:13And as you can see, it's counting token number.
  2037. 2:16:16Here we are using an open AI model, but we are using here, um,
  2038. 2:16:22Llama's tokenizer to count tokens.
  2039. 2:16:25So as you saw, we successfully learned how to use the
  2040. 2:16:29exact tokenizer for the exact model we are using.
  2041. 2:16:32Although here we are an open AI model, we are not using LamaBot.
  2042. 2:16:37I showed you how to use Llama's tokenizer if you wanted to use Llama model.
  2043. 2:16:41And now before finishing the session, let's do something a little more interesting.
  2044. 2:16:47Let's add a limit for the total total tokens.
  2045. 2:16:50total number of tokens our chatbot is allowed to consume
  2046. 2:16:54and as you remember here
  2047. 2:16:57we have an attribute called total tokens so
  2048. 2:17:01as the conversation
  2049. 2:17:03continues and more messages are exchanged this number keeps increasing
  2050. 2:17:08and we can put
  2051. 2:17:09a limit on it in practice you will find that in
  2052. 2:17:12many services that you may use many ai services
  2053. 2:17:15that you might use and you can in those services
  2054. 2:17:19you can consume a certain number of tokens within
  2055. 2:17:22a certain period of time and let's implement that let's a simple version of that
  2056. 2:17:28we only need to add a condition so I say
  2057. 2:17:33if self.
  2058. 2:17:36Why isn't suggesting anything? I don't know
  2059. 2:17:39total tokens
  2060. 2:17:43Great here, then.
  2061. 2:17:44300 or we can just say the 300 is okay okay we will test it
  2062. 2:17:50and in that case I'll print
  2063. 2:17:54Unfortunately
  2064. 2:17:57you
  2065. 2:17:59Exceeded.
  2066. 2:18:01the
  2067. 2:18:02token limit.
  2068. 2:18:04for today.
  2069. 2:18:05and here instead of using the continue keyboard I'm going to use
  2070. 2:18:09the break keyboard because I want to terminate the program
  2071. 2:18:13because we want to terminate our program and
  2072. 2:18:16we don't want to give the user another chance
  2073. 2:18:20and now here as you can see we have two different limits
  2074. 2:18:24the first one checks the number of tokens in the current user prompt
  2075. 2:18:29and if it exceeds 10 tokens, reject that prompt and give the user another chance
  2076. 2:18:37the second one checks the total number consumed
  2077. 2:18:41if it exceed 300, we stop the chat
  2078. 2:18:47and
  2079. 2:18:49Let's test our program. Let's say hello.
  2080. 2:18:53Okay.
  2081. 2:18:55What is the...
  2082. 2:18:58Capital of
  2083. 2:18:59France.
  2084. 2:19:02What is its population?
  2085. 2:19:06okay as you can see here it says unfortunately
  2086. 2:19:10we exceeded the total token limit for today and
  2087. 2:19:15it automatically terminated the program and after
  2088. 2:19:19that we see the report so that's okay and let's
  2089. 2:19:24remove this line that we kept printing the number of token in each query okay
  2090. 2:19:32So this is a simple example, but the idea is the same.
  2091. 2:19:36The idea is same for more sophisticated scenarios.
  2092. 2:19:39And we learned how to have a simple token management system,
  2093. 2:19:44and we can extend it for real world applications.
  2094. 2:19:50you
  2095. 2:19:56Okay, in this session I want to show you how to
  2096. 2:19:58stream the response we receive from the LLM
  2097. 2:20:01So far our application waits for the LLL to generate the entire response
  2098. 2:20:06And then we wait for it to generate entire response
  2099. 2:20:11and we return the complete answer to the user
  2100. 2:20:14But now we want to change this behavior
  2101. 2:20:17Instead of waiting for the entire response, I want
  2102. 2:20:21the LLM to send us what it has generated so far
  2103. 2:20:24and we want to display it to the user as it arrives
  2104. 2:20:27so to do that it's not difficult at all
  2105. 2:20:31we need to set another argument called stream and i set it to true
  2106. 2:20:36by default it is false which means we don't want to stream the response
  2107. 2:20:40but we want do it once we do it the response is no longer returned as before
  2108. 2:20:46previously we could access a response through something like this
  2109. 2:20:51choices.message.content. But when we enable streaming, response becomes an iterable.
  2110. 2:20:58So we have an iterable that we can loop over.
  2111. 2:21:01So for now, I'm going to comment out
  2112. 2:21:05these lines here. And as I mentioned,
  2113. 2:21:09response that we get now is an iterable we can
  2114. 2:21:12we can iterate over it and in each iteration it
  2115. 2:21:17it gives us a chunk and a chunk basically contains a piece of a piece
  2116. 2:21:22of the content that the LLM has generated and
  2117. 2:21:25sent back to us so since response is iterable
  2118. 2:21:29i'm going to loop over it. and i do this using a for loop so i say for
  2119. 2:21:36and I call the loop variable chunk.
  2120. 2:21:40Say chunking.
  2121. 2:21:42response and for each chunk we still have this part i mean choices
  2122. 2:21:48however we no longer have
  2123. 2:21:50message instead we have delta so
  2124. 2:21:53Let's copy it.
  2125. 2:21:56here.
  2126. 2:21:59paste it here instead of response I say chunk and instead of message I say delta
  2127. 2:22:07okay great here i'm also going to add a guard
  2128. 2:22:11to make sure that choices is not mpt choices
  2129. 2:22:15actually exists
  2130. 2:22:16before before trying to access its first item if it doesn't exist i don't want to do
  2131. 2:22:23anything but if it exists i want to access it's first element so here let's say if
  2132. 2:22:31chunk
  2133. 2:22:33choices.
  2134. 2:22:36in that case i'm going to i'm gonna access that and i'm
  2135. 2:22:40going to store it in a variable called delta okay so this delta represents the
  2136. 2:22:47new piece of content generated by the llm it might be a token part of a word a word
  2137. 2:22:53a longer piece of text whatever it really depends on that moment that the ll m is
  2138. 2:22:59generating and now here i'm gong to add another guard
  2139. 2:23:04I'm going to say if at this delta
  2140. 2:23:07It's not empty.
  2141. 2:23:10in that case i'm going to print that chunk of data
  2142. 2:23:12that i received but here we have a problem by default
  2143. 2:23:16python's print function adds a new line after it
  2144. 2:23:19prints something we don't want that new line we
  2145. 2:23:22want the pieces of response that we receive to appear next to each other so
  2146. 2:23:28Delta.
  2147. 2:23:29so i'm going to add another argument called end and
  2148. 2:23:33i set it to an empty string the default value
  2149. 2:23:37is new line backslash n
  2150. 2:23:41And for this reason, it goes to the next line by default.
  2151. 2:23:44So setting it to an empty string prevents Python from moving to the next line.
  2152. 2:23:49And now there is one more thing that is setting flush to true.
  2153. 2:23:55flash to true and
  2154. 2:23:56What does that mean?
  2155. 2:23:57Because Python
  2156. 2:23:58may sometimes buffer the output instead of displaying
  2157. 2:24:02it immediately and we don't want that behavior.
  2158. 2:24:05We want every piece of generated content to be displayed as soon as we receive them.
  2159. 2:24:11Okay?
  2160. 2:24:12But again, we have another problem.
  2161. 2:24:14I also need to store the complete response because we need it for history.
  2162. 2:24:19As you remember, to add history to our conversation, we need to
  2163. 2:24:22store the messages and then with every new query, we
  2164. 2:24:25need to send the entire history.
  2165. 2:24:28But with the code that we have, we are only printing the response.
  2166. 2:24:32We are not actually storing it. So how we can store the completed response manually.
  2167. 2:24:39I mean, here.
  2168. 2:24:40before our loop starts, I can add a new variable called response text.
  2169. 2:24:49points.
  2170. 2:24:50text and I set it to an empty string.
  2171. 2:24:53Now every chunk that arrives I added to this variable. So here
  2172. 2:24:59I say response
  2173. 2:25:01text.
  2174. 2:25:03plus equal delta
  2175. 2:25:07So with this little trick, we solve this problem in this way.
  2176. 2:25:11The user sees the response being generated in
  2177. 2:25:14real time while we also build the complete
  2178. 2:25:17response as well. And there is still one more edge case that we need to handle.
  2179. 2:25:23Remember that we wanted to keep track of how many tokens were consumed.
  2180. 2:25:28And when we use this streaming mode, you don't see the usage information.
  2181. 2:25:33the user info the usage information is typically
  2182. 2:25:36provided in the final chunk with the code we have
  2183. 2:25:39in first iteration in first chunk we have nothing in second and third ones we have
  2184. 2:25:44nothing and eventually when the stream is finished when the final
  2185. 2:25:49chunk arrives the final trunk will contain the information the usage
  2186. 2:25:54information that we need so we need another if statement here and i say if
  2187. 2:26:01chunk.usage
  2188. 2:26:03if in the chunk that has arrived there is this usage field then I'm going to add
  2189. 2:26:12token I'm going to extract
  2190. 2:26:16that information and add them to the attributes
  2191. 2:26:20that we define for tracking token usage.
  2192. 2:26:24Now after the loop, I'm going to use an empty print statement.
  2193. 2:26:30It simply moves us to the next line. After that empty line, we will have our report.
  2194. 2:26:35Okay, and it will be more visually appealing.
  2195. 2:26:39And finally, we need to add the complete
  2196. 2:26:41generated response to the conversation history.
  2197. 2:26:45I don't need these lines anymore. I copied them. This one as well.
  2198. 2:26:49And I comment it out. Okay, this is exactly
  2199. 2:26:53what we want at this point we need to
  2200. 2:26:57add the complete response okay the same name okay that's okay
  2201. 2:27:02and finally we return the response it's not bad to return the
  2202. 2:27:05response but we don't need it anymore because as you can see we are printing
  2203. 2:27:10the response for the user as the chunks arrive
  2204. 2:27:15and here we need actually remove this line we don t want to
  2205. 2:27:20print it anymore
  2206. 2:27:21because if we do that the user will see the response twice
  2207. 2:27:25I only need to call this bot method
  2208. 2:27:30Okay, finally let's run our program and test it
  2209. 2:27:35Okay.
  2210. 2:27:36What happened?
  2211. 2:27:38there is a problem here. Oh, we forgot the comma.
  2212. 2:27:46And here we have a typo.
  2213. 2:27:49okay great I run the program one more time okay I say hello okay
  2214. 2:27:55what's the problem here oh the
  2215. 2:27:58problem is that instead of chunk
  2216. 2:28:03We are using response
  2217. 2:28:07and this is why I told you that pay attention when you copy paste your code
  2218. 2:28:12okay one more time let's
  2219. 2:28:15say hello
  2220. 2:28:17Okay, what is the capital? Let's say, tell me a story.
  2221. 2:28:25okay as you can see the response appears very
  2222. 2:28:28quickly it is actually being being streamed piece
  2223. 2:28:31by piece but it's happening so fast that you may
  2224. 2:28:34not even notice and notice individual chunks and
  2225. 2:28:37to make the streaming behavior more smooth and
  2226. 2:28:41easier to see i'm going to do another little
  2227. 2:28:44visual trick here i'm gonna import time module
  2228. 2:28:49okay and um
  2229. 2:28:52Here right after we receive a new piece of
  2230. 2:28:55response, I'm going to introduce an artificial
  2231. 2:28:58delay.
  2232. 2:28:59So here I'm going to say time
  2233. 2:29:03sleep for 0.05
  2234. 2:29:07seconds this is obviously not something that
  2235. 2:29:11you want to do in real application it's just
  2236. 2:29:14for visual purpose to see that it is actually being
  2237. 2:29:18streamed and let's run our program one more time
  2238. 2:29:24Okay.
  2239. 2:29:27I just say
  2240. 2:29:29Tell me a story.
  2241. 2:29:32okay as you can see this time we can clearly see
  2242. 2:29:35the response being generated piece by piece and
  2243. 2:29:39that's exactly what we were looking for we have
  2244. 2:29:42successfully implemented the streaming for our
  2245. 2:29:44LLM response and you saw that it wasn't difficult at all okay let's
  2246. 2:29:51terminate it consume tokens and calculated costs
  2247. 2:29:55okay now we have everything we wanted to
  2248. 2:29:59Ladies and gentlemen, a message from your captain.
  2249. 2:30:03We've noticed that some passengers are getting
  2250. 2:30:05tired of our long exploration of planet Chatbot
  2251. 2:30:08and can't wait to set off for the next world.
  2252. 2:30:12That's completely understandable.
  2253. 2:30:14Planet Chatbot is the longest stop on this entire journey.
  2254. 2:30:18But please know this, what you learn here is essential
  2255. 2:30:21for your survival on every planet that follows.
  2256. 2:30:24So please be patient, the next world is closer than you think.
  2257. 2:30:37From this session onward, I want to talk to you about a very important concept.
  2258. 2:30:41It's called context window. This is very important.
  2259. 2:30:44Every LLM has a context window and its size is a hard limit.
  2260. 2:30:49It's a hard-limit imposed by the model. In other words, you cannot change it.
  2261. 2:30:54You cannot change the model's context window yourself.
  2262. 2:30:57So when you're using a model, you can check its context window
  2263. 2:31:01and decide whether it's sufficient whether it's good for your application or not.
  2264. 2:31:07Besides checking this limit, you also need to
  2265. 2:31:10implement some strategies to manage this,
  2266. 2:31:13to manage the context window.
  2267. 2:31:15and there are different strategies or different ways to manage the context window.
  2268. 2:31:20This is something that honestly you need to do, you need
  2269. 2:31:23to take care of because that has a lot to do with the cost and other metrics that
  2270. 2:31:28are important to you and your application.
  2271. 2:31:30So first let's discuss what exactly is context window?
  2272. 2:31:35Okay, let's say that we have a model and this
  2273. 2:31:39model has a context window of 128,000 tokens.
  2274. 2:31:44What does that mean?
  2275. 2:31:45It means that in a single request, the model can only process up to approximately
  2276. 2:31:51128, 000 tokens. And what does that include? What does this number include?
  2277. 2:31:58It includes both the input and the output.
  2278. 2:32:02both the input tokens and the output tokens generated by the model.
  2279. 2:32:06So we can say it includes the user's last query, our conversation
  2280. 2:32:11history,
  2281. 2:32:13and the input that the model generates itself.
  2282. 2:32:17So again, the tokens
  2283. 2:32:19that occupy the context window come from three main components.
  2284. 2:32:25New query, or history, and the model's output.
  2285. 2:32:29And for sure, if you have any system messages, those are also taken into account.
  2286. 2:32:34So as long as the total token count fits within this limit, I
  2287. 2:32:39mean fits within the model's context window, we are fine.
  2288. 2:32:43We don't have any problem.
  2289. 2:32:44But if we exceed the available capacity, we run into a problem.
  2290. 2:32:49So we need to be careful, we need implement some
  2291. 2:32:52strategies in our application to handle this limit.
  2292. 2:32:54Even if we choose a model that has a very high context
  2293. 2:32:58window we still need to apply these strategies.
  2294. 2:33:02And even when you open conversation in ChatChipity or cloud,
  2295. 2:33:06don't assume that every message you have exchanged is
  2296. 2:33:09necessarily sent to the model exactly the same form.
  2297. 2:33:12Exactly the same for every time.
  2298. 2:33:14No, these messages, I mean, your conversation history is summarized.
  2299. 2:33:19Some of them may be deleted, maybe summarized in some way.
  2300. 2:33:22And there are different strategies, I'm in.
  2301. 2:33:24And different systems use different strategies.
  2302. 2:33:26But in this tutorial, I am going to introduce two main strategies.
  2303. 2:33:31and the second approach is more common in real world application.
  2304. 2:33:35But first, we will see the first approach. And that approach is sliding window.
  2305. 2:33:40and when you understand this strategy first, it gives us a very good foundation
  2306. 2:33:45and we won't have any difficulty to understand the second approach.
  2307. 2:33:50Okay, let's start with the first approach, sliding window. What does that mean?
  2308. 2:33:56In this strategy we decide how many messages
  2309. 2:33:59we want to keep in our conversation history.
  2310. 2:34:01For example, we might decide to keep only the last five
  2311. 2:34:06messages and ignore the rest, ignore the early messages.
  2312. 2:34:10So as a new message arrives, all their messages are removed.
  2313. 2:34:14One new message comes and one daily message goes.
  2314. 2:34:18And with this strategy, we are somehow sliding a
  2315. 2:34:21window and because of this, these strategies
  2316. 2:34:24often called the sliding window. So the whole point of this strategy is that
  2317. 2:34:29we no longer keep the entire conversation.
  2318. 2:34:32Instead, we will only keep a selected portion of our conversation.
  2319. 2:34:36For example, the last five messages. Okay. And why do we do that?
  2320. 2:34:41Why the last 5 messages? Why the LAST messages? I mean, the assumption is that
  2321. 2:34:47the recent messages are often more relevant, more important.
  2322. 2:34:50And if we keep these messages, our LLM is more likely to have the
  2323. 2:34:55information that is needed to keep track of the conversation.
  2324. 2:34:59And of course, this approach has a limitation.
  2325. 2:35:02We may lose some important information, but we cannot do anything about it.
  2326. 2:35:07We should accept that we are losing some information.
  2327. 2:35:10And if the user refers to something discussed
  2328. 2:35:13a long time ago in messages that we deleted,
  2329. 2:35:16the model doesn't have enough context to understand and answer that query anymore.
  2330. 2:35:22So that's a limitation.
  2331. 2:35:24And that's why the second approach can be
  2332. 2:35:27better because instead of removing messages,
  2333. 2:35:30it summarizes those messages. So we lose less information.
  2334. 2:35:36But first, let's implement the sliding window approach.
  2335. 2:35:42So first I'm going to add a new method for it.
  2336. 2:35:46I'm gonna add a, a new, a method called, uh, let's set it
  2337. 2:35:50up here. Okay. Here.
  2338. 2:35:53I add a new method, I call it
  2339. 2:35:56Trim
  2340. 2:35:58messages.
  2341. 2:36:00OK, self.
  2342. 2:36:02pass okay first I don't implement it so I use
  2343. 2:36:06the pass keyword so Python won't complain
  2344. 2:36:09about it and then I'm going to add another parameter for it which is max number
  2345. 2:36:16and I set a default value for it for example 4
  2346. 2:36:21and this is the number of messages that I want to keep in conversation history okay
  2347. 2:36:27now we're going to call this method we'll call this message inside our bot method
  2348. 2:36:33after the LLM has generated its response and
  2349. 2:36:36we have added the response to our conversation history so at the end of this method
  2350. 2:36:43I'm going to called this method trim messages. Okay.
  2351. 2:36:47This means after each response, we trim the conversation history.
  2352. 2:36:52We are moving that sliding window.
  2353. 2:36:54And with this approach, the window moves forward
  2354. 2:36:58and this way, the conversation progresses.
  2355. 2:37:02This way, we keep the most recent messages and we discard older ones.
  2356. 2:37:07Now let's go back to our method we just defined and let's implement it.
  2357. 2:37:14the first thing that we need to check is whether
  2358. 2:37:18the number of messages is already less than max
  2359. 2:37:21number or not if that's the case we don't need to do
  2360. 2:37:25anything if we already have less than this number
  2361. 2:37:28we don' t need to anything so here i put an if statement and i say if the length of
  2362. 2:37:36or our history.
  2363. 2:37:38is less than this max number that I have here so in that case I return
  2364. 2:37:43and this prevents us from doing unnecessary work
  2365. 2:37:46when the history is still short, is less then four messages
  2366. 2:37:51and there is also another edge case we need to consider
  2367. 2:37:54if you remember our conversation history does not
  2368. 2:37:57only contain user messages and assistant messages
  2369. 2:38:01it also contains system messages and as i
  2370. 2:38:04explained earlier system messages are important
  2371. 2:38:08these messages contain instructions and we need
  2372. 2:38:12these instructions to guide the LLM's behavior
  2373. 2:38:15so we don't want to accidentally or deliberately remove these instructions.
  2374. 2:38:21So the first step is to separate our messages. We need two lists.
  2375. 2:38:27The first list will contain only system messages. And the second list will
  2376. 2:38:33contain all the other messages containing user messages and assistant messages.
  2377. 2:38:38Here we can do this using list comprehensions.
  2378. 2:38:42And this is a very concise syntax and this way
  2379. 2:38:46can create a new list based on an older list. So let's do that.
  2380. 2:38:51First, I'm going to use square brackets and this way I'm telling Python that, uh, I
  2381. 2:38:57am, I am going to this syntax. Then I'm gonna iterate over my conversation history.
  2382. 2:39:03So I say for every single message in my conversation history.
  2383. 2:39:09and then I'm going to add a condition. I'm going to add a
  2384. 2:39:13condition to check if that message has a role of system or not.
  2385. 2:39:18So I say if that message.
  2386. 2:39:23is equal to
  2387. 2:39:24system.
  2388. 2:39:26In that case, I want to have that message. Okay.
  2389. 2:39:30Then I'm going to store the returning list in a variable called system messages.
  2390. 2:39:38Now for the other list I'm going to do the opposite thing.
  2391. 2:39:42I only duplicate this line.
  2392. 2:39:45However, this time I only need to change equality to inequality.
  2393. 2:39:51And this way we get all messages whose role is not system.
  2394. 2:39:56Which means user messages and assistant messages.
  2395. 2:40:00So I change its variable name to other messages.
  2396. 2:40:07okay now we have two lists now how the next step
  2397. 2:40:11is to keep only the last max number messages from
  2398. 2:40:15other messages we need to take these messages
  2399. 2:40:18out and we can do this using slicing syntax
  2400. 2:40:21so here i say other messages
  2401. 2:40:25I add brackets and then I say negative max number to the end.
  2402. 2:40:32So here because we are using a negative index,
  2403. 2:40:36this selects the last max number elements.
  2404. 2:40:39In this case it is four. And we go until the end, so we go all the way to the
  2405. 2:40:45end. So this gives us the last four non-system messages.
  2406. 2:40:49And I store this list inside a variable called trimmed messages.
  2407. 2:40:57Okay, so far so good.
  2408. 2:40:59Now we have all we need.
  2409. 2:41:01We have system messages and we have the messages that we want to keep.
  2410. 2:41:07And here is a great place to use Python's unpacking syntax.
  2411. 2:41:12We want to create a new list, a new conversation history, which consists, uh,
  2412. 2:41:18system messages and trimmed messages so here I unpack all my system messages
  2413. 2:41:26along with
  2414. 2:41:28The messages I want to keep.
  2415. 2:41:31Okay, and then we want to update our conversation history.
  2416. 2:41:36So I say self.messages.
  2417. 2:41:40okay great and that's it
  2418. 2:41:42now the only thing left is to test it
  2419. 2:41:46i run my program okay great
  2420. 2:41:50i say i ask a series of questions
  2421. 2:41:56Great.
  2422. 2:41:57What is that?
  2423. 2:41:59Capital of...
  2424. 2:42:00France
  2425. 2:42:03What is
  2426. 2:42:04Uh...
  2427. 2:42:07what is its population
  2428. 2:42:09okay we had set a token limit and we exceeded that token
  2429. 2:42:15limit so let's increase it where was that here
  2430. 2:42:19Okay, 3000 is enough. Okay, or maybe 10,000. Okay, one more time.
  2431. 2:42:26Okay, again, what is the
  2432. 2:42:28capital of?
  2433. 2:42:30France
  2434. 2:42:32Okay, what is its
  2435. 2:42:35population
  2436. 2:42:37okay here we have a problem and the problem is that
  2437. 2:42:44it belongs to here up here problem is that we forgot to unpack this list and
  2438. 2:42:51we are creating a nested list and then we're
  2439. 2:42:55passing it to messages so it shows that it was
  2440. 2:42:59trying to to call this method it actually called
  2441. 2:43:03it and we had this problem so i run it one more
  2442. 2:43:06time okay say hello
  2443. 2:43:09you can fast forward this part if you like
  2444. 2:43:14Capital
  2445. 2:43:16France,
  2446. 2:43:17what is its population?
  2447. 2:43:21Okay.
  2448. 2:43:22What is its
  2449. 2:43:25president okay now let's terminate the program and see our conversation history
  2450. 2:43:31and as you can see our first message is a system message so we successfully
  2451. 2:43:38separated the messages and we are not removing our system message which is
  2452. 2:43:42important and as we can see our first
  2453. 2:43:45message was hello and here you see that here we have what is its population so
  2454. 2:43:51as you see early messages
  2455. 2:43:53are gone and we are only going to have the last
  2456. 2:43:56four messages okay so we have successfully
  2457. 2:44:00implemented the first approach and in the next
  2458. 2:44:04session we will move to to the next approach
  2459. 2:44:13All right, in this session we are going to look at the second approach.
  2460. 2:44:17The second approach to managing conversation history.
  2461. 2:44:20In the previous approach we removed the older messages
  2462. 2:44:23and kept only the most recent ones.
  2463. 2:44:27As I mentioned the main drawback of this approach is that we lose information.
  2464. 2:44:31Now with our new approach instead of simply deleting the older
  2465. 2:44:35messages we are gonna send them to an LLM and this way we get a summary and we can
  2466. 2:44:41replace those early messages with that summary. This way we lose less information.
  2467. 2:44:47This is a common strategy for managing conversation history.
  2468. 2:44:51And you see this pattern and you see
  2469. 2:44:53this approach, this strategy in many AI applications and projects.
  2470. 2:44:58So the question is, how are we going to implement this?
  2471. 2:45:01There are different approaches, sub approaches we can say here.
  2472. 2:45:05But one option is to summarize the conversation after every, every few messages.
  2473. 2:45:10For example, we could summarize it in every five, every five messages.
  2474. 2:45:15So the first step is to keep track of which iteration, to keep
  2475. 2:45:20track of each iteration of conversation we are currently in.
  2476. 2:45:23I'm going to add a new attribute to my class and that attribute can be
  2477. 2:45:28called iteration, just iteration is good.
  2478. 2:45:32and I set it to 0 as its initial
  2479. 2:45:36value
  2480. 2:45:38Now I'm going to go
  2481. 2:45:41I'm going to go to my...
  2482. 2:45:43conversation loop and here at the beginning
  2483. 2:45:46at the begining of the loop I'm going to increment this attribute
  2484. 2:45:50so I say self.iteration
  2485. 2:45:54plus equal
  2486. 2:45:57plus equal one. I increment it. I increment it by one.
  2487. 2:46:02Now I'm going to define another method
  2488. 2:46:07below this method.
  2489. 2:46:09Summarize.
  2490. 2:46:12messages.
  2491. 2:46:16Okay, and I just put pass here because first I just
  2492. 2:46:20want to define this method without implementation.
  2493. 2:46:24And then here instead of calling the previous
  2494. 2:46:27method, I'm going to call this new method.
  2495. 2:46:31So I say self.
  2496. 2:46:34summarize messages and i remove this one now let's
  2497. 2:46:38implement it first i want summarization to happen
  2498. 2:46:43only every five messages so we can do this
  2499. 2:46:46using a modulus operator if you remember the
  2500. 2:46:49modulus operator gives you the reminder after the division so here i can say
  2501. 2:46:57if self.iteration
  2502. 2:47:01and I say modulus as I told
  2503. 2:47:04you
  2504. 2:47:055 if it's not equal to 0
  2505. 2:47:08I'm going to
  2506. 2:47:10I'm going to return.
  2507. 2:47:11So this means that if the current iteration isn't divisible by five, we return.
  2508. 2:47:19We return immediately without doing anything.
  2509. 2:47:22and summarization will happen on iterations like 5, 10, 15, 20 and so on.
  2510. 2:47:30Next, we need to separate the system messages
  2511. 2:47:33from the other messages, just as we did in
  2512. 2:47:36the previous session. So we can just copy the logic that we had for that.
  2513. 2:47:47Okay, let's fix
  2514. 2:47:48indentation. No, it's okay.
  2515. 2:47:51Now at this point, I want to split other messages into two parts.
  2516. 2:47:55The first part contains the older messages.
  2517. 2:47:59We want to summarize these older messages and the second part contains
  2518. 2:48:03the more recent messages. And we want to keep these messages.
  2519. 2:48:08We want preserve them exactly as they are. How many messages we want summarize.
  2520. 2:48:14It depends on ourselves. It depends our project. For this example,
  2521. 2:48:20I'm going to summarize the first five messages and I will keep the rest as they are.
  2522. 2:48:26So here to create these sub lists, I'm
  2523. 2:48:30going to use bracket notation. So I save other messages.
  2524. 2:48:34I want to summarize the first five
  2525. 2:48:39messages and I also slice
  2526. 2:48:42the rest so from 0 to 5 from index 5 to the end
  2527. 2:48:47here we have two lists and we want to store
  2528. 2:48:50these two inside two variables so a better
  2529. 2:48:54syntax here is to convert these two into a
  2530. 2:48:57tuple now we can unpack this tuple on the left side so here i say
  2531. 2:49:06Early
  2532. 2:49:08messages
  2533. 2:49:11and
  2534. 2:49:12last
  2535. 2:49:14messages.
  2536. 2:49:15Okay, now
  2537. 2:49:16on the right hand side, we have a top hole.
  2538. 2:49:19You're unpacking it into two variables, early messages and last messages.
  2539. 2:49:25I want to leave these last messages unchanged
  2540. 2:49:30and I want summarize these. So we need an LLM call.
  2541. 2:49:35So here I
  2542. 2:49:37just,
  2543. 2:49:40Copy.
  2544. 2:49:41this piece
  2545. 2:49:42of code
  2546. 2:49:44and i paste it here we need to change some
  2547. 2:49:47arguments for example we no longer need the
  2548. 2:49:50streaming key here and here instead of passing the
  2549. 2:49:54entire conversation history i just want to pass
  2550. 2:49:58early messages but we need we need guide we need the llm we need
  2551. 2:50:02to specify its task so here i also add a system message so here instead
  2552. 2:50:07of just this list i create a new list and I put a message as its first element
  2553. 2:50:14and then I unpack this list here
  2554. 2:50:19So here first we have a system message, then we have our messages.
  2555. 2:50:25So here I specify the role.
  2556. 2:50:29I say it's a system message and then
  2557. 2:50:32we have our content
  2558. 2:50:35So I say, summarize these messages concisely.
  2559. 2:50:43keeping
  2560. 2:50:44key facts
  2561. 2:50:46and
  2562. 2:50:48decisions.
  2563. 2:50:51Okay.
  2564. 2:50:52okay as you can see for this example i'm using the same model the same
  2565. 2:50:57model that we used as our main model for conversation but you could also use another
  2566. 2:51:04another model a smaller model a more lightweight
  2567. 2:51:08model or a model that is specifically trained
  2568. 2:51:11trained for summarization it totally depends on you
  2569. 2:51:15and it's not a difficult thing just change it
  2570. 2:51:18and use another model
  2571. 2:51:20and then here we receive a response we need to convert it into a message
  2572. 2:51:26I prefer to use a system message some other people have a different preference
  2573. 2:51:31they save it as an assistant message but I prefer save it
  2574. 2:51:35as a system so here I create a message I specify the role
  2575. 2:51:41I want it to be a system message and then I specify its content
  2576. 2:51:46I say Let me use formatted string and I use triple.
  2577. 2:51:53triple quotes.
  2578. 2:51:55Here I say this is a summary of the five early messages.
  2579. 2:52:03and then down here I put a placeholder
  2580. 2:52:06and then I extract the content of my response I say response.
  2581. 2:52:13choices, choices zero,
  2582. 2:52:15message and content.
  2583. 2:52:18the same as before
  2584. 2:52:19we did it a lot and I store it in a variable called
  2585. 2:52:22summarized message
  2586. 2:52:27Okay, great. Now
  2587. 2:52:28we have converted the LLM's response into a message, into a
  2588. 2:52:32message that can put inside our conversation history.
  2589. 2:52:37and the final step is that we need to construct our new conversation history
  2590. 2:52:43first we need put all system messages at the beginning
  2591. 2:52:48then we want the summarized message and then we
  2592. 2:52:52want the last messages the last
  2593. 2:52:55messages that we that we left here to use them
  2594. 2:53:00so here i create a new array
  2595. 2:53:04with the same syntax like before. I mean unpacking. I have a new array.
  2596. 2:53:09First I put system messages.
  2597. 2:53:14system messages, then I have this single summarized message and then last messages.
  2598. 2:53:23the messages that I didn't want to summarize. Okay,
  2599. 2:53:27that's okay now.
  2600. 2:53:29I need to update the conversation history. So I say self that
  2601. 2:53:34messages
  2602. 2:53:37and I set it to this new array. And that's it.
  2603. 2:53:40We have replaced the older messages with a summary.
  2604. 2:53:43We are also preserving the more recent messages. That's a very good strategy.
  2605. 2:53:49And at the end, I put a print statement.
  2606. 2:53:53This way we can realize that the summarization is taking place.
  2607. 2:53:59I say summarization was
  2608. 2:54:02successful.
  2609. 2:54:03Okay, now the final step is to run our program and test it to see
  2610. 2:54:09whether it works. So here I say hello, create
  2611. 2:54:15What is the capital of France?
  2612. 2:54:21What is...
  2613. 2:54:23its population
  2614. 2:54:27Okay.
  2615. 2:54:28What is its
  2616. 2:54:32Uh,
  2617. 2:54:34with its
  2618. 2:54:36President
  2619. 2:54:39Okay, still waiting for the summarization. Okay. What is its...
  2620. 2:54:45famous dish
  2621. 2:54:49now here we need to have a summary yes summarization was successful and let's
  2622. 2:54:54quit and see the conversation history so here i can we can see that we have our
  2623. 2:55:00system message that we specified ourselves and now here uh we have a system
  2624. 2:55:05message that is the summary that the LLM generated for us and yes we have
  2625. 2:55:11key information and then we have the last messages after that summary so
  2626. 2:55:18As you can see in this strategy, instead of
  2627. 2:55:20simply deleting the earlier messages, we are
  2628. 2:55:23using an LLM to summarize them.
  2629. 2:55:26And this way we lose less information in comparison with the previous approach.
  2630. 2:55:31Welcome back aboard Neon Learn. Planet Chatbot is conquered.
  2631. 2:55:35One world down, nine to go.
  2632. 2:55:39Fasten your seat belt. Next stop, plan a classifier.
  2633. 2:55:42Here you'll teach AI to classify anything
  2634. 2:55:45using lightweight models from Hugging Face.
  2635. 2:55:48No LLM required.
  2636. 2:55:50Your next world awaits. Enjoy the journey.
  2637. 2:56:00Alright, in the next project we are going to talk about sentiment analysis.
  2638. 2:56:04This is one of the most commonly used AI tasks.
  2639. 2:56:08And it means we give our model a piece of text
  2640. 2:56:12and it tells us whether it's negative, positive, or neutral.
  2641. 2:56:18This is something that you will see in many
  2642. 2:56:20different applications, in many AI tasks.
  2643. 2:56:22For example, we can use an AI model to analyze customer comments.
  2644. 2:56:28and this way it can determine whether each comment is positive, negative or neutral.
  2645. 2:56:35Maybe we need this information for ourselves, maybe we
  2646. 2:56:38want to help other customers to make a better decision.
  2647. 2:56:42The application is honestly endless.
  2648. 2:56:45But the important question for us is that how can we implement this task?
  2649. 2:56:50Well, one way is to use an LLM. I mean,
  2650. 2:56:55the same LLM that we used in the previous project, in the chatbot project.
  2651. 2:57:00And another approach is to use models that are specifically designed, that are
  2652. 2:57:05specifically trained for sentiment analysis.
  2653. 2:57:08You might ask, what's the advantage of the second one?
  2654. 2:57:12Because obviously the first one is simpler when we use an LL.
  2655. 2:57:15But what is the advantage the second to use a specialized model?
  2656. 2:57:20The answer is that these models are much smaller,
  2657. 2:57:23they are lighter and as a result, cheaper.
  2658. 2:57:26They are so lightweight that maybe you can run them on your own personal computer
  2659. 2:57:32because they are lightweight parameters are not too many.
  2660. 2:57:35So that's the advantage and that's exactly what
  2661. 2:57:38we are going to do in this second project.
  2662. 2:57:40So first we use an LLM to implement this task
  2663. 2:57:44and then we move on to the second approach.
  2664. 2:57:47which we will use a specialized model and you will see how we do that.
  2665. 2:57:52So let's focus on the first approach and let's implement it.
  2666. 2:57:55Let's go to into code
  2667. 2:57:57and start creating our project.
  2668. 2:58:00So first I create a new file called
  2669. 2:58:03sentiment.
  2670. 2:58:05Analysis
  2671. 2:58:07high and
  2672. 2:58:08here I will follow the same steps that we used in the previous project.
  2673. 2:58:13First, I'm going to import.env because I want
  2674. 2:58:16to access to my environmental variables,
  2675. 2:58:20because I'm, I am going to have an LLM call and then I import OS module to access
  2676. 2:58:27to my environment.
  2677. 2:58:29variables and then I'll import OpenAI because we want to create a client and
  2678. 2:58:34communicate with the LLM so I say from
  2679. 2:58:38OpenAI
  2680. 2:58:40import
  2681. 2:58:41OpenAI class
  2682. 2:58:44So, and finally, I'm going to load the environment variables. So I say.env.
  2683. 2:58:50load.env and then I'll create my main function.
  2684. 2:58:58and finally i'll add our starting point i say if
  2685. 2:59:02double underscore name double underscore is equal to
  2686. 2:59:06double underscore main double underscore in that
  2687. 2:59:10case i want the main function to be executed and
  2688. 2:59:14from the beginning i want to use object-oriented
  2689. 2:59:18structure so here i add a new class called analyzer
  2690. 2:59:23along with its constructor so init and that's it and now
  2691. 2:59:28i go to our chatbot project and i'm going to borrow
  2692. 2:59:34these two.
  2693. 2:59:36paste them here okay fix indentation okay this is
  2694. 2:59:41my api key and that's the same and here instead of
  2695. 2:59:46using this system message i'm going to use another
  2696. 2:59:49system message because it's not a chatbot anymore
  2697. 2:59:52i'm gonna give it an instruction specifically
  2698. 2:59:56for sentiment analysis so here i say classify
  2699. 3:00:00the sentiment of the text as positive negative or neutral okay
  2700. 3:00:08And now we need a method that receives a piece of text as its
  2701. 3:00:13input and sends it to the LLM and returns the model's response.
  2702. 3:00:17So here I say def.
  2703. 3:00:20sentiment analysis
  2704. 3:00:24self parameter should be always self in a class in Python
  2705. 3:00:27and the second parameter is the text that we
  2706. 3:00:30want to figure out what is its sentiment
  2707. 3:00:33now again I go to chatbot and again borrow some code
  2708. 3:00:40okay I copy this and I paste
  2709. 3:00:43it here
  2710. 3:00:45and of course we need to make a few changes first I'm going to add temperature
  2711. 3:00:51and I set the temperature to 0.1 a very low temperature
  2712. 3:00:57and the reason is that for sentiment analysis we generally don't need creativity
  2713. 3:01:02we want the model to behave as deterministically as possible so that's the reason
  2714. 3:01:08then here for the messages we no longer need these two
  2715. 3:01:13we need to unpack our messages here our conversation history that we have
  2716. 3:01:18as an attribute so first i unpack it so i say self.messages
  2717. 3:01:24and then
  2718. 3:01:26we are going to put the text that this function,
  2719. 3:01:29this method more accurately receives.
  2720. 3:01:32But instead of just putting a piece of text here, we
  2721. 3:01:36need to wrap it with that role content structure.
  2722. 3:01:39So here I put curly braces and I say role is user and then I say
  2723. 3:01:47content is
  2724. 3:01:50what I have received
  2725. 3:01:52here okay that's it and finally I get the response and return
  2726. 3:01:56so I say return
  2727. 3:01:58response.choices
  2728. 3:02:00exactly like before choices 0
  2729. 3:02:06Message, Content.
  2730. 3:02:09Now we need to go to our main function and use it.
  2731. 3:02:13First, I need to create an object.
  2732. 3:02:15So I say analyzer and then
  2733. 3:02:17I store this object inside the variable.
  2734. 3:02:22Say analyzer.
  2735. 3:02:24and then I can call the sentiment analysis method
  2736. 3:02:29on this object so I say analyzer.sentiment
  2737. 3:02:33analysis and the piece of text that I want to give it is for example the movie
  2738. 3:02:39is perfect.
  2739. 3:02:42Okay, now let's run our program and see what happens. Run the program.
  2740. 3:02:48Oh, we forgot to print it actually.
  2741. 3:02:50So let's print it or maybe it's better to store the
  2742. 3:02:55result in a variable called sentiment and then print it.
  2743. 3:03:02Okay, one more time.
  2744. 3:03:05And as you can see, it has correctly labeled the
  2745. 3:03:08sentiment of this piece of text as positive.
  2746. 3:03:12Ladies and gentlemen, the captain has switched on the feedback sign.
  2747. 3:03:17If this flight has been worth it, please rate it with a like.
  2748. 3:03:21Want a window seat for every new launch? Join our frequent flyer program.
  2749. 3:03:25Subscribe and turn on notifications.
  2750. 3:03:29And tell us in the comments where should our next
  2751. 3:03:31flight go, the captain is already planning the route.
  2752. 3:03:35Thank you for choosing Neon Learn.
  2753. 3:03:45Okay, now in this session, I want to talk about
  2754. 3:03:48an important concept called prompt engineering.
  2755. 3:03:51This probably isn't a new concept for you.
  2756. 3:03:53We have actually been doing it throughout the course a lot. So what
  2757. 3:03:57does that mean?
  2758. 3:03:58What does prompt engineering mean?
  2759. 3:04:00In simple terms, it means controlling the behavior
  2760. 3:04:04of the model through the input we provide.
  2761. 3:04:07Okay, for example, in the previous project, we
  2762. 3:04:10use the same model to summarize messages and
  2763. 3:04:13In this project you saw that we used that model
  2764. 3:04:16again, but this time for sentiment analysis.
  2765. 3:04:19The model is the same, but the behavior is completely different.
  2766. 3:04:22So where does this different behavior come from?
  2767. 3:04:25It comes from the prompt that we provide.
  2768. 3:04:29So, as a category of prompt engineering, we can refer to few-shot
  2769. 3:04:33prompting, and that's what I'm going to talk about now.
  2770. 3:04:37it means that before the model generates its answer
  2771. 3:04:41we provide it with a few examples of how we expect it to behave
  2772. 3:04:45let me give you an example
  2773. 3:04:47for example we can tell the model that the movie was terrible
  2774. 3:04:51and then we show it that it should classify it as negative
  2775. 3:04:56then again we say that nothing special and we show its label neutral
  2776. 3:05:01terrific positive the movie is terrible
  2777. 3:05:04negative and by giving the model these examples
  2778. 3:05:08we are showing it what the task looks like
  2779. 3:05:11and how we expect it to respond and one of the
  2780. 3:05:14best patterns for doing this in these scenarios
  2781. 3:05:18is using user assistant pattern in other words
  2782. 3:05:21we can put several examples in the conversation
  2783. 3:05:24history with the roles of user and assistant in other
  2784. 3:05:28words again even though these are not actual messages
  2785. 3:05:33we put them and this way we want to kind of control the behavior of the model
  2786. 3:05:38so let me show you here in our conversation history
  2787. 3:05:43after putting the system prompt we add some user resistant messages
  2788. 3:05:50So here I say role is user.
  2789. 3:05:53And here we specify I absolutely loved this restaurant.
  2790. 3:06:00OK, and then we have a message with the assistant role.
  2791. 3:06:07and we say here positive okay another example for for the negative case okay
  2792. 3:06:15Here I copy two of them.
  2793. 3:06:17okay again user has said the wait time was too long was way too long
  2794. 3:06:26So it's definitely negative.
  2795. 3:06:30One more time.
  2796. 3:06:32We can say
  2797. 3:06:35it was an average experience
  2798. 3:06:39nothing special and here we can say neutral okay these are
  2799. 3:06:45not real messages as you can see we didn't
  2800. 3:06:49we didn send this message to the llm and get the
  2801. 3:06:52corresponding response they are made up these are
  2802. 3:06:55for controlling models behavior to show it how it
  2803. 3:06:59should behave what is the structure of output what is
  2804. 3:07:03what are for example possible answers and instead of providing a long explanation we
  2805. 3:07:09just provide some examples now let's run our program
  2806. 3:07:14Okay, as you can see, our program is working like
  2807. 3:07:16before, but let me show you something interesting.
  2808. 3:07:19Imagine we expected a slightly different output format.
  2809. 3:07:22For example, I expected that the model put a
  2810. 3:07:26sentiment plus colon before each response.
  2811. 3:07:30So here I add sentiment
  2812. 3:07:32colon and I repeat that again.
  2813. 3:07:36Okay,
  2814. 3:07:38here.
  2815. 3:07:39and again here now
  2816. 3:07:40let's run our program one more time
  2817. 3:07:43as you can see this time the model has learned that
  2818. 3:07:46we want this part here before the actual answer
  2819. 3:07:50so you can say this is quite useful this is very useful in these scenarios
  2820. 3:07:55in scenarios that explanation is less useful than
  2821. 3:07:59providing some examples it is useful both for us and for the model itself
  2822. 3:08:04the model can better understand
  2823. 3:08:07what you want.
  2824. 3:08:08And one important question you might ask is how many examples are enough?
  2825. 3:08:13Honestly, in many cases, around three to five examples are fine.
  2826. 3:08:18And pay attention when you
  2827. 3:08:20choose your examples, try to cover different scenarios.
  2828. 3:08:24For example, in our example, you
  2829. 3:08:26saw that we include a clear example of positive
  2830. 3:08:30case, negative case and neutral case.
  2831. 3:08:33So you can also add
  2832. 3:08:35couple of more challenging or ambiguous examples.
  2833. 3:08:38For example, this one was a little bit vague because it could be also negative.
  2834. 3:08:44But we showed the model that in these scenarios it should give this answer neutral.
  2835. 3:08:49But someone may say it is negative when there is nothing special
  2836. 3:08:53when an average experience is positive.
  2837. 3:08:57Sorry, it is positive, but it totally depends on us and few-shot prompting is great.
  2838. 3:09:04to specify our tendency, our attitude.
  2839. 3:09:07For example, for this kind of sentiment analysis
  2840. 3:09:10we want it to be neutral instead of negative.
  2841. 3:09:14But keep in mind that generally adding more
  2842. 3:09:17and more examples is not necessarily better.
  2843. 3:09:20In many cases, going beyond 10 examples doesn't provide much additional benefit.
  2844. 3:09:26It only burns your tokens.
  2845. 3:09:28So choose your examples carefully and the goal isn't
  2846. 3:09:32to give the model as many examples as possible.
  2847. 3:09:35The goal is to give a small number of representative
  2848. 3:09:39examples that clearly demonstrate the behavior
  2849. 3:09:42We expect. We want.
  2850. 3:09:45Ladies and gentlemen, congratulations. You've just learned to run sentiment analysis
  2851. 3:09:50with a large language model.
  2852. 3:09:53But before we move on, your captain has a bonus for you.
  2853. 3:09:57You'll learn to solve the very same task with natural language inference.
  2854. 3:10:01Instead of a giant LLM, a small model simply
  2855. 3:10:04checks whether a statement follows from the text.
  2856. 3:10:08Is this review happy or unhappy?
  2857. 3:10:11It's dozens of times lighter, runs on an
  2858. 3:10:14ordinary server, and needs no API calls at
  2859. 3:10:17all.
  2860. 3:10:18Consider it a free upgrade, your captain will now show you how.
  2861. 3:10:29Alright, in the previous session we worked on sentiment analysis
  2862. 3:10:32and we used an LLM to perform this task Now in this session we are going to use
  2863. 3:10:39a model that's much more specifically suited to this kind of task.
  2864. 3:10:43If you want to be precise, sentiment analysis is a well-defined task
  2865. 3:10:47and there are many many models specifically trained for this task.
  2866. 3:10:52For example, these models might be trained on data sets containing
  2867. 3:10:56movie reviews, product reviews, or other types of comments or other information.
  2868. 3:11:01And during training, each example is associated with a label,
  2869. 3:11:06such as positive, negative or neutral.
  2870. 3:11:08And after training, the model learns to classify
  2871. 3:11:11new pieces of text based on those labels.
  2872. 3:11:15So let's take a quick look at HuggingFace and
  2873. 3:11:19search for sentiment analysis or just sentiment.
  2874. 3:11:23Okay.
  2875. 3:11:24As you can see here, there are many models with the word sentiment in their names.
  2876. 3:11:29Previously, Huggingface had a dedicated filter task for that, but
  2877. 3:11:35at some point it was removed under a broader category
  2878. 3:11:39broader category called text classification
  2879. 3:11:42and changes like this happen a lot happen frequently in this community
  2880. 3:11:47and this has two reasons first this field is relatively young changes happen a
  2881. 3:11:52lot and second it's evolving incredibly quickly
  2882. 3:11:56so if you notice a task category has changed has removed has uh whatever
  2883. 3:12:02Don't be surprised and even libraries you use, for example
  2884. 3:12:06You can see that Langchain sees a lot of changes
  2885. 3:12:09I know it can be a little frustrating especially when
  2886. 3:12:13you are following a course, following a tutorial
  2887. 3:12:16and you see that interface is not like what you see in the videos
  2888. 3:12:21but you should be adaptive as developers you need to get used to it
  2889. 3:12:26now the question is that what are we going to do in this session
  2890. 3:12:32honestly i don't want to look at this problem only as a sentiment analysis task
  2891. 3:12:37i want you to learn something more powerful
  2892. 3:12:41something that you can use not only for sentiment
  2893. 3:12:44analysis but also for many other use cases.
  2894. 3:12:47And that brings us to NLI or Natural Language Inference.
  2895. 3:12:51The goal of NLI is to understand the logical
  2896. 3:12:55relationship between two pieces of text.
  2897. 3:12:58These two pieces have specific names.
  2898. 3:13:02The first one is called premise
  2899. 3:13:05and the second one is call hypothesis.
  2900. 3:13:08The premise is the information we start with
  2901. 3:13:12and the hypothesis is a piece of information we
  2902. 3:13:16want to evaluate in relation to the premise.
  2903. 3:13:20And there are three outcomes. The first is entailment.
  2904. 3:13:25The second is contradiction. And the third is neutral.
  2905. 3:13:30Let me give you an example this way you will learn better.
  2906. 3:13:32Suppose our premise is a man. is playing guitar on stage.
  2907. 3:13:39Now let's look at three different hypotheses that each one has a different outcome.
  2908. 3:13:45The first example is a person is performing music. What do you think?
  2909. 3:13:51What do think the result should be? The premise tells us that a man is playing
  2910. 3:13:55guitar on a stage. The hypothesis says the person is
  2911. 3:13:59performing music, so these two statements are consistent and the
  2912. 3:14:04premise supports the hypothesis. So the result is definitely entailment.
  2913. 3:14:09Now, Example 2. A man is sleeping.
  2914. 3:14:12The premise tells us that a man is playing
  2915. 3:14:15guitar while the hypothesis says he's sleeping.
  2916. 3:14:18So in this example, the two statements contradict each other.
  2917. 3:14:23A man cannot be sleeping while playing music. So that doesn't make sense.
  2918. 3:14:27So the result is contradiction.
  2919. 3:14:30And the third example, a man wearing a red T-shirt.
  2920. 3:14:35Does the premise confirm this statement? No. Does it contradict it? Not necessarily.
  2921. 3:14:40So, the result is simply neutral.
  2922. 3:14:44And that's the basic idea behind natural language inference.
  2923. 3:14:48Now,
  2924. 3:14:49the question is, why am I saying that NLI is very powerful?
  2925. 3:14:55because it can be used for many many different applications and use cases
  2926. 3:15:01for example NLI can be use for zero shot classification
  2927. 3:15:06fact checking, question answering, duplicate detection and many other scenarios
  2928. 3:15:12and the use case we are going to focus on in the following sessions
  2929. 3:15:16is zero shot classification so let's understand what it means well traditionally
  2930. 3:15:23If we wanted to build a model for a particular classification task,
  2931. 3:15:28we would usually need to train a model on a labeled dataset.
  2932. 3:15:32Let's give you an example.
  2933. 3:15:34Let's say we want to automatically tag articles on our website.
  2934. 3:15:38We have different articles. We want to tag them.
  2935. 3:15:41And suppose these articles are about programming and
  2936. 3:15:45our labels are JavaScript, Python, Ruby and CSS.
  2937. 3:15:49I mean programming languages and technologies.
  2938. 3:15:53Traditionally, if we wanted such a model, we would collect
  2939. 3:15:57a data set containing perhaps a thousand articles.
  2940. 3:16:01Each article would have a label, a label indicating its topic.
  2941. 3:16:05For example, one article might be labeled JavaScript, another Python, another Ruby.
  2942. 3:16:11And we would then use this label data set to train a model.
  2943. 3:16:16and after training a model at the end we would have a model
  2944. 3:16:20that had learned to classify articles into those categories
  2945. 3:16:25so
  2946. 3:16:25we could give it a new article without a label
  2947. 3:16:29and it would predict the category that article belonged to
  2948. 3:16:32but now let's look at the approach that I want to talk about
  2949. 3:16:36zero shot classification
  2950. 3:16:37with a suitable pre-trained NLI model we don't
  2951. 3:16:40need to have that training process anymore
  2952. 3:16:44and we can use a good NLI model to classify different kinds of label data
  2953. 3:16:51so we can take a general purpose model that has already grasped
  2954. 3:16:55that has understood relationships between text statements
  2955. 3:17:00and give it a set of candidate labels
  2956. 3:17:02so the model can evaluate the input text against those labels
  2957. 3:17:07and determine which ones are the best match
  2958. 3:17:10For example, today I have a set of candidate
  2959. 3:17:13labels like JavaScript, Python, Ruby, and CSS.
  2960. 3:17:16But tomorrow, I might need to classify completely different categories.
  2961. 3:17:20For example health, sports, economics, technology, and so on.
  2962. 3:17:26Now, without changing the model, I only change the labels, the candidate labels.
  2963. 3:17:32And that's really powerful that we don't need a new model.
  2964. 3:17:36We can use the same model.
  2965. 3:17:38So that makes zero shot classification and natural language
  2966. 3:17:42inference extremely useful in a wide range of applications.
  2967. 3:17:47and that's exactly why I wanted to introduce NLI
  2968. 3:17:50instead of focusing only on sentiment analysis
  2969. 3:17:54another advantage is that it's much smaller and I mean
  2970. 3:17:58these kinds of models are smaller in comparison with
  2971. 3:18:02something like the generation models that we used
  2972. 3:18:07for our chatbot we use an LLM these LLMs have
  2973. 3:18:11billions of parameters but these kinds of models
  2974. 3:18:14that we are going to use are much smaller
  2975. 3:18:16for example the model we are
  2976. 3:18:18going to is under 1 GB we can even run it
  2977. 3:18:22locally on our personal computer and we will
  2978. 3:18:26do it in the following sessions and the question is which model are we going to
  2979. 3:18:31use exactly the model I'm going to for this project is distilbert-base-uncased-mnli
  2980. 3:18:40Let's search it more accurately, type form.
  2981. 3:18:44Okay, so that's the model we are going to use.
  2982. 3:18:47And as you can see, it has 67 million parameters, which is
  2983. 3:18:53really lightweight.
  2984. 3:18:58Okay, the model we are going to use belongs to the BERT family. So what is BERT?
  2985. 3:19:04BERT stands for Bidirectional Encoder Representations from Transformers.
  2986. 3:19:09That's quite a long name, I know.
  2987. 3:19:11But BERT was introduced by Google in 2018 and it became an influential model in NLI.
  2988. 3:19:19And many subsequent models and approaches
  2989. 3:19:22were built on top of it, on ideas from BERT.
  2990. 3:19:25So here the important thing to understand is
  2991. 3:19:29that BERT is an encoder-based transformer
  2992. 3:19:32and it learns conceptual representations of language
  2993. 3:19:36and one of its key characteristics is that it processes text bidirectionally
  2994. 3:19:41and because of that it has this term in its name
  2995. 3:19:45Now let's look at what does that mean in total Consider these two sentences
  2996. 3:19:50I see the blue sky and I feel blue In the first sentence, blue refers to color.
  2997. 3:19:57In the second sentence, blue describes a feeling of sadness.
  2998. 3:20:01So, the word is the same but the meaning is different.
  2999. 3:20:04The meaning changes depending on the context.
  3000. 3:20:07And the word context you hear a lot refers to that.
  3001. 3:20:10If you see, if you hear that a model learns the context of the text,
  3002. 3:20:15the term context means that. So, to summarize, context means.
  3003. 3:20:21means the surrounding information that helps us to understand
  3004. 3:20:26what a word or sentence actually means.
  3005. 3:20:28So we should consider other words when we want to grasp the meaning of a word.
  3006. 3:20:33And that's really important.
  3007. 3:20:35Earlier approaches often struggle to capture this information.
  3008. 3:20:40They couldn't do that effectively.
  3009. 3:20:42But the transformer architecture revolutionized
  3010. 3:20:45everything and introduced the powerful
  3011. 3:20:47technique a powerful mechanism called attention
  3012. 3:20:51and it was described in the influential paper
  3013. 3:20:54attention is all you need and as i told you attention helps a model to understand
  3014. 3:21:00the context better and it determines which parts
  3015. 3:21:04of the input are relevant to one another and the
  3016. 3:21:08BERT model builds on the transformer architecture
  3017. 3:21:12and uses information from both sides from both
  3018. 3:21:15directions and this can help the model understand the context much better.
  3019. 3:21:21So that's the meaning of BERT and what does
  3020. 3:21:25Distilbert mean? As you can see it's not only BERT, it's
  3021. 3:21:30DistilBERT. We should know that what distil mean actually in
  3022. 3:21:35in the world of AI. It refers to a technique. We have a technique called
  3023. 3:21:39Knowledge Distillation.
  3024. 3:21:41Knowledge Distillation is a technique used to
  3025. 3:21:44train a smaller model to reproduce the behavior of a larger model.
  3026. 3:21:49To replicate capacities of a large model.
  3027. 3:21:52For example, imagine we have a model with billions
  3028. 3:21:55of parameters and we want to create a smaller one.
  3029. 3:21:58And this way we can train a small model using the
  3030. 3:22:01information that the larger model produces.
  3031. 3:22:05And it enables the smaller model, to learn to approximate some of its capacities and
  3032. 3:22:11This way we can reduce the model size and this way we need
  3033. 3:22:15less computational power, fewer computational requirements
  3034. 3:22:20and we can kind of retain as much useful performance as possible
  3035. 3:22:25and this DistilBERT here is a small version derived from the BERT model we talked about
  3036. 3:22:30and it uses this approach, knowledge distillation
  3037. 3:22:34and this DistilBERT was introduced by Hugging Face
  3038. 3:22:37and the goal was obvious to be more lightweight and it's lighter than the BERT model
  3039. 3:22:44so it was about this part, this DistilBERT
  3040. 3:22:46and another important thing, another important concept I want you to understand
  3041. 3:22:51is about three different categories that we have
  3042. 3:22:54which are encoder, decoder and cross-encoder the
  3043. 3:22:58BERT model is an encoder
  3044. 3:23:01encoder model, encoder-based model. But what exactly does an encoder mean?
  3045. 3:23:06In simple terms, an encoders processes input
  3046. 3:23:10and it tries to take out representations. Okay?
  3047. 3:23:14And a decoder, on the other hand, is used to generate output.
  3048. 3:23:19For example, a GPTS style model that you see are decoder only
  3049. 3:23:23and are designed to generate information, to generate text data in general.
  3050. 3:23:28And the key point here is that BERT is not designed to generate text.
  3051. 3:23:33It is designed to understand text. And because of that, it's an encoder model.
  3052. 3:23:39And for our NLI task, that's exactly what we need.
  3053. 3:23:42We don't need a model that can generate text, we needed to
  3054. 3:23:46understand the relationship between two pieces of text.
  3055. 3:23:49And help us determine whether one is entailment, contradiction, neutral, and so on.
  3056. 3:23:55So here we can use an encoder based model
  3057. 3:23:58rather than a generative model, a generative
  3058. 3:24:00element. Of course, different AI tasks require different architectures.
  3059. 3:24:04For example, if we had machine translation,
  3060. 3:24:07in that case, we needed both because we need
  3061. 3:24:10encoder to process the source language and take out the context.
  3062. 3:24:15And then we use decoder to use that information to generate in the target language.
  3063. 3:24:21That was an example where we need both components.
  3064. 3:24:25And there's still one category called cross encoder.
  3065. 3:24:29A cross encoders processes two pieces of text together.
  3066. 3:24:32That's important. It does it together.
  3067. 3:24:34And it allows the model to compare the two inputs directly
  3068. 3:24:38and learn how they are related to one another. This is very useful in many tasks.
  3069. 3:24:45For example,
  3070. 3:24:46In RackSystems, we want to know whether a
  3071. 3:24:49document is relevant to search query or not.
  3072. 3:24:52Or many other situations that's really important that we need to know
  3073. 3:24:56whether a piece of text answers a question or no.
  3074. 3:25:01And in these cases, what matters to us is the
  3075. 3:25:03relationship between two pieces of text.
  3076. 3:25:06And in this situations, cross-encoder is one option that helps us.
  3077. 3:25:16Now that we understand the theory behind this task,
  3078. 3:25:20let's see how we can use this model in our code.
  3079. 3:25:23If you go on the model's page on Hugging Face and click on Use this model and click
  3080. 3:25:29on Transformers, you can see here we have two options, Pipeline and Auto Classes.
  3081. 3:25:34Auto Class give you more control,
  3082. 3:25:36more control over how the model is loaded and used.
  3083. 3:25:40However, they also require more code and involve a little more complexity.
  3084. 3:25:46The pipeline option on the other hand is much simpler.
  3085. 3:25:49As you can see, we can specify the task and the model
  3086. 3:25:53and the pipeline takes care of most of the implementation for us
  3087. 3:25:57which is really great.
  3088. 3:25:58But one important point is that the pipeline
  3089. 3:26:01itself also uses the underline model classes.
  3090. 3:26:05It simply provides a higher level interface that makes things easier for us.
  3091. 3:26:10So in this project, which approach are we going to use? we are going to pipeline
  3092. 3:26:15because it's simpler and that's enough that's for our project and it's
  3093. 3:26:19simpler for you now what we are
  3094. 3:26:22going to do now is to replace the LLM we used in our previous session
  3095. 3:26:27with our pre-trained HuggingFace model what you saw here
  3096. 3:26:30and to use models from Huggingface we need to install transformers library
  3097. 3:26:35which we have already installed in an earlier session We also need PyTorch.
  3098. 3:26:39We also needed PyTorch, which is one of the
  3099. 3:26:42libraries that Transformers uses to run model.
  3100. 3:26:44So we need it. And before the session, I installed it.
  3101. 3:26:49So you can install it using this command, pipenv install torch.
  3102. 3:26:54By running this command you can't install PyTorch. Okay.
  3103. 3:26:58now let's go back to our code and first i'm going
  3104. 3:27:02to import pipeline i imported it before starting
  3105. 3:27:05recording the session i imported pipeline from
  3106. 3:27:08transformers and now let's find the part of our
  3107. 3:27:11code which was for the llm so now we are going
  3108. 3:27:15to replace it with our classifier so here
  3109. 3:27:17instead of having this lln i'm gonna have a classifier
  3110. 3:27:22and I set it to pipeline exactly what you see here we need a pipeline
  3111. 3:27:28the first argument we need to provide is the type of task that we want to perform
  3112. 3:27:34in our case the task is zero shot classification so here you see it is suggesting us
  3113. 3:27:39I'm going to have this task zero shot classification
  3114. 3:27:43next we need specify which model we want to use so I go here and I copy it
  3115. 3:27:49to avoid any typo
  3116. 3:27:51So I paste it here and this way you specify the model that you're going to use.
  3117. 3:27:57And that's pretty much it.
  3118. 3:27:59The pipeline handles the main steps required to load
  3119. 3:28:02the model and perform zero shot classification.
  3120. 3:28:05We don't need to implement those steps ourselves, which is really good.
  3121. 3:28:08As a beginner, these types of things are
  3122. 3:28:11really scary, but it makes it easier for us.
  3123. 3:28:14So that's really great.
  3124. 3:28:15now let's move let's move to our sentiment analysis method here we need to we need
  3125. 3:28:22to change this LLM call that we have here i'm going to remove the code that sends
  3126. 3:28:27the request to an LL and instead we are going to use the model that we chose
  3127. 3:28:32first we need
  3128. 3:28:33to define our candidate labels which labels do we want to use positive negative
  3129. 3:28:38and neutral these are all labels so here i define a new variable called
  3130. 3:28:44labels it's a list of three labels positive
  3131. 3:28:48negative and neutral okay next we'll use the
  3132. 3:28:52classifier we initialized in our constructor and
  3133. 3:28:56as a first argument we pass the text that was
  3134. 3:28:58provided to our model and as the second argument we
  3135. 3:29:02passed the candidate labels we just defined here
  3136. 3:29:05let's do it here i say self.classifier okay and as first argument i say
  3137. 3:29:13text, the text that I'm receiving in this method.
  3138. 3:29:16And as a second argument, I specify the candidate label.
  3139. 3:29:21So I say candidate
  3140. 3:29:23labels, and I set it to these labels.
  3141. 3:29:27And I store the result in a variable called result.
  3142. 3:29:35And finally, I will return the result.
  3143. 3:29:40and that's all we need to change notice that we have kept the same method and the
  3144. 3:29:45same overall structures we have simply changed the implementation behind it
  3145. 3:29:49that's the only thing we do previously we used an LLM and we had an LLL call
  3146. 3:29:54now we are using a pre-trained model and we do it locally we run a model
  3147. 3:29:59locally and we get it from HuggingFace and run it and use it now let's run our
  3148. 3:30:03program and see what happens okay here I run the program
  3149. 3:30:08One thing to keep in mind is that the first
  3150. 3:30:10time you run the program, the model needs to
  3151. 3:30:12be downloaded. So the first run will take longer than usual.
  3152. 3:30:16And once the model has been downloaded and cached,
  3153. 3:30:19subsequent runs generally won't need to download
  3154. 3:30:22it again. Okay. You can see the result here.
  3155. 3:30:25It says that with this probability 0.98% this
  3156. 3:30:29text you see here, the movie is perfect,
  3157. 3:30:33positive.
  3158. 3:30:33And there is one important detail here.
  3159. 3:30:36When you run the program, the model's weights
  3160. 3:30:39are loaded into your computer's memory.
  3161. 3:30:42So this way the model can perform the task and can perform inference.
  3162. 3:30:47And when the program finishes, the memory can be released.
  3163. 3:30:51So that adds to
  3164. 3:30:52the time you're waiting here but when you run this program on a server
  3165. 3:30:57it's up 24 7 and you don't have to wait the same time that you wait here
  3166. 3:31:02now let me show you something interesting i'm going to change the labels
  3167. 3:31:07but i won't change the model or the implementation of our method
  3168. 3:31:12just instead of specifying these labels i say finance
  3169. 3:31:18health,
  3170. 3:31:20and sort.
  3171. 3:31:23And here, instead of saying the movie is perfect, I say
  3172. 3:31:28Fitness activity.
  3173. 3:31:31It's good for our
  3174. 3:31:34body okay that's it and everything is like before
  3175. 3:31:38Okay, I run the program. Okay, notice what we have done here.
  3176. 3:31:43We have changed the task from sentiment analysis
  3177. 3:31:45to classifying text into broader categories
  3178. 3:31:48related to fitness, health, sports. And as you can see, it works correctly.
  3179. 3:31:53This piece of text is related to health and it correctly labels it as health.
  3180. 3:31:59As you saw, we didn't train a new model and we
  3181. 3:32:01didn't change the underlying classification
  3182. 3:32:04implementation. We only changed. the candidate labels and the input. That's it.
  3183. 3:32:08And that's the power of zero shot classification.
  3184. 3:32:11We can use the same model for different classification
  3185. 3:32:15tasks, simply by providing a different set
  3186. 3:32:17of candidate labels.
  3187. 3:32:19And that is one of the reasons NLI-based models
  3188. 3:32:23are so useful and powerful in AI engineering.
  3189. 3:32:26Welcome back aboard Neon Learned.
  3190. 3:32:29Planet Classifier is conquered, two worlds down, eight to go.
  3191. 3:32:33Fasten your seat belts. Next stop, Planet Embeddings.
  3192. 3:32:37Here, you'll master embeddings and build search
  3193. 3:32:39that understands meaning, not just keywords.
  3194. 3:32:45Your next world awaits. Enjoy the journey.
  3195. 3:32:48Oh
  3196. 3:32:54Okay, in our next project, we are going to explore embeddings.
  3197. 3:32:57Embeddings are one of the fundamental concepts you need to understand.
  3198. 3:33:01You need to understand when working with LLMs and AI engineering in general.
  3199. 3:33:05They are not only an important concept in many AI
  3200. 3:33:08engineering applications, such as RAG, RAG systems,
  3201. 3:33:11but they can also be used as a foundation for tasks in their own right.
  3202. 3:33:17and that's something we are going to explore in this project.
  3203. 3:33:20So let's start by understanding the idea behind embeddings.
  3204. 3:33:24Let me start with an example.
  3205. 3:33:26As humans, we receive information in many different forms.
  3206. 3:33:29It can be text, images, audio, and many other types of data.
  3207. 3:33:33Now, when we hear a sentence, when when we read something or receive information
  3208. 3:33:39in some other way, our brain processes that information and forms
  3209. 3:33:44a mental representation of what it means.
  3210. 3:33:47We still don't fully understand every detail of what happens inside the human brain.
  3211. 3:33:52But the important point is that we form an
  3212. 3:33:55understanding of the information we receive.
  3213. 3:33:58For example, when I tell you I like black cats, you
  3214. 3:34:01might immediately imagine a black cat in your mind.
  3215. 3:34:05The concept of embeddings aims to achieve something similar for computers.
  3216. 3:34:10The idea is to take an input.
  3217. 3:34:12and convert it into a numerical representation that
  3218. 3:34:16captures important information about its meaning.
  3219. 3:34:19And
  3220. 3:34:20question is that how do
  3221. 3:34:22we represent that meaning?
  3222. 3:34:24We represent it using numbers. Computers only deal with numbers.
  3223. 3:34:30And for embedding the way we do it, we use a pre-trained model.
  3224. 3:34:34This model receives a piece of data as input and converts it into vector.
  3225. 3:34:40which is simply a list of numbers.
  3226. 3:34:42For now, imagine that this vector has only two dimensions. What does that mean?
  3227. 3:34:48It means we can represent the vector as a
  3228. 3:34:51point on a two-dimensional coordinate system
  3229. 3:34:54with an x-axis and y-axis.
  3230. 3:34:57Now, suppose we give the model the word cat.
  3231. 3:35:01The model converts it into a point in this two-dimensional space.
  3232. 3:35:06If we give it the word dog, we get another point.
  3233. 3:35:09We can do the same thing with ship and car. Now, here's the important part.
  3234. 3:35:15Because cats and dogs are semantically related,
  3235. 3:35:19a well-trained embedding model will generally represent them
  3236. 3:35:23in a way that makes their vectors relatively close.
  3237. 3:35:27Similarly, the representations of ships and cars
  3238. 3:35:31may be closer to each other than they are to the representation of cats and dog.
  3239. 3:35:36In other words, the model learns to represent
  3240. 3:35:39relationships between concepts in a numerical space.
  3241. 3:35:43Of course, this is a simplified illustration.
  3242. 3:35:46Real embedding spaces are much more complex,
  3243. 3:35:49and the relationships between vectors depend on how the model was trained.
  3244. 3:35:54But this gives you the basic idea of embedding models. And you should know that
  3245. 3:36:00Having this concept, I mean having a model that can convert data
  3246. 3:36:04into meaningful numerical representations is extremely powerful
  3247. 3:36:08because it allows us to build many different types of applications.
  3248. 3:36:12For example, embeddings are useful for semantic search, which is finding
  3249. 3:36:17information based on meaning rather than just matching exact keywords.
  3250. 3:36:23Or recommendation systems to find products, articles or
  3251. 3:36:27other items that are similar to something a user likes, a user prefers.
  3252. 3:36:32Spam detection, which is representing messages in a way that can help a
  3253. 3:36:37classifier distinguish spam from legitimate messages.
  3254. 3:36:41Or fraud detection, using embeddings as features
  3255. 3:36:45to help identify suspicious patterns or
  3256. 3:36:48RAG systems, the application that we want to have.
  3257. 3:36:51Finding relevant documents to provide as context to an LLM.
  3258. 3:36:56Of course, building a high-quality embedding model from scratch isn't easy.
  3259. 3:37:01I simplified it.
  3260. 3:37:02It requires substantial mathematical knowledge, large amounts
  3261. 3:37:07of training data, computing resources, and time of course.
  3262. 3:37:11But here's the good news.
  3263. 3:37:13We are working in AI engineering, and we are not
  3264. 3:37:16going to train a foundation model from a scratch.
  3265. 3:37:19Our goal is to use these models, to learn how to use
  3266. 3:37:23these models effectively. So we don't need to train them, we just want to use them.
  3267. 3:37:28And there are many, many commercial and freely available embedding models
  3268. 3:37:33that we can use directly in our application. So, we don't need to build
  3269. 3:37:37an embedding model ourselves.
  3270. 3:37:39Another point is that, I mean, another thing that I simplified was about
  3271. 3:37:43dimensions. I told you that we had a two-dimensional space.
  3272. 3:37:48In reality, embedding models typically represent text in
  3273. 3:37:52spaces with hundreds or sometimes thousands of dimensions.
  3274. 3:37:56Each input is converted into a vector containing
  3275. 3:38:00many numerical values, several hundreds.
  3276. 3:38:02And again, another thing is that don't assume
  3277. 3:38:06that the input has to be a single word.
  3278. 3:38:08It could be a singled word, of course.
  3279. 3:38:11It could be a sentence, a paragraph, or even a longer document,
  3280. 3:38:15several pages, depending on the model and its limits.
  3281. 3:38:19Embedding models have maximum input lengths, just like other language models.
  3282. 3:38:24As long as the input fits within the model's supported
  3283. 3:38:28length, we can generate an embedding for it.
  3284. 3:38:31For longer documents, we may need to split
  3285. 3:38:34the text into smaller chunks first and then
  3286. 3:38:37we do the task. And embeddings aren't limited to text.
  3287. 3:38:41There are models designed to generate embeddings
  3288. 3:38:44for other types of data, including images,
  3289. 3:38:46audio, and some multimodal models can even represent different types of
  3290. 3:38:51data in a shared embedding space, making it
  3291. 3:38:54possible to compare information across modalities.
  3292. 3:38:59to compare a video with a piece of text to find
  3293. 3:39:02out whether that video is related to that
  3294. 3:39:05text.
  3295. 3:39:06And as I mentioned earlier, there are many embedding models available.
  3296. 3:39:11For example, OpenAI offers
  3297. 3:39:12embedding models in addition to its generation models.
  3298. 3:39:15Different embedding model in OpenAI produce
  3299. 3:39:18vectors with different numbers of dimensions.
  3300. 3:39:20For example, there is a model that produces 1500 dimensional
  3301. 3:39:25vectors and there is another larger model that
  3302. 3:39:28produces 3000 dimensional vectors.
  3303. 3:39:31And generally speaking, larger embedding vectors
  3304. 3:39:34require more storage and computational resources.
  3305. 3:39:38A model with more dimensions. may capture useful distinctions more effectively.
  3306. 3:39:43So another way we can say that it's stronger,
  3307. 3:39:46but more dimensions don't automatically mean
  3308. 3:39:49better results.
  3309. 3:39:50The quality of the model depends on many factors
  3310. 3:39:53like the way it is trained and so when you
  3311. 3:39:56want to use it in your tasks, there are some consequences. considerations.
  3312. 3:40:00And in addition to OpenAI, another provider you will often hear is Coheer.
  3313. 3:40:06Its embedding models are used in applications
  3314. 3:40:09such as Semantic Search and Retrieval Systems.
  3315. 3:40:12And for enterprise projects where quality and
  3316. 3:40:15reliability are important, commercial embedding
  3317. 3:40:18models can be a very good option. It is worth investing.
  3318. 3:40:23but if you want to keep your costs down there are
  3319. 3:40:26many excellent open source embedding models
  3320. 3:40:29some of them are small enough to run directly on
  3321. 3:40:33your personal computer and in this project we will
  3322. 3:40:36use a model that we can do we can download it and use it on our own computer
  3323. 3:40:49For this project we are going to use a very popular embedding
  3324. 3:40:52model called All Mini LM and let's search for it in Hugging Face.
  3325. 3:40:56And now let me explain where this name comes from.
  3326. 3:40:59First let's look at this part, All.
  3327. 3:41:02This refers to the model being designed as a general purpose sentence embedding
  3328. 3:41:07model and it is trained on a broad collection of data sets and that's why
  3329. 3:41:13It can be used for a broad range of tasks, including
  3330. 3:41:17semantic search, center and similarity, and clustering.
  3331. 3:41:21And you can use it for texts about health, technology,
  3332. 3:41:25sports, and many other subjects and domains.
  3333. 3:41:29And next we have Mini-LM.
  3334. 3:41:31Mini-lm is a family of compact transformer models associated with Microsoft.
  3335. 3:41:38The model we are using builds on another model called Mini LM L6 H384
  3336. 3:41:46and it's a smaller encoder model designed to
  3337. 3:41:49be effective and small at the same time.
  3338. 3:41:51And then we have L6.
  3339. 3:41:54The L6 indicates that the underlined transformer encoder has six layers
  3340. 3:42:00and you should know that neural network models
  3341. 3:42:03are made up of layers that progressively transform the input to the output.
  3342. 3:42:09And finally, what does V2 mean?
  3343. 3:42:12It identifies this particular version of the model.
  3344. 3:42:17So we have a general purpose embedding model based on a compact six layer
  3345. 3:42:22transformer encoder with version 2.
  3346. 3:42:25And one more useful detail is that this model produces 384 dimensional embeddings.
  3347. 3:42:32It's relatively small and fast, which makes it a good choice for learning
  3348. 3:42:37and for even many real world applications.
  3349. 3:42:40So,
  3350. 3:42:46To work with embedding models, we can use a
  3351. 3:42:48Python library called sentence transformers.
  3352. 3:42:51So let's start by installing it by installing sentence-transformers.
  3353. 3:42:55So on the terminal, I say pipenv install sentence-transformers.
  3354. 3:43:03Okay, great.
  3355. 3:43:04And while it is installing it, I'm going to
  3356. 3:43:08create a new Python file called embedding.py.
  3357. 3:43:12So here I say embedding dot py.
  3358. 3:43:15hi okay here first i'm going to import dot env so
  3359. 3:43:19i say import dotenv and then i load environment
  3360. 3:43:24variables from our dotenv file so i'll say dotenv
  3361. 3:43:27dot load dotenv notice that we don't need to
  3362. 3:43:31import os module here we are not going to read the token ourselves
  3363. 3:43:36The sentence transformers library is going to do that.
  3364. 3:43:39Next, we are going to import sentence transformer itself.
  3365. 3:43:43As you can see, it was successfully installed and now I can import it.
  3366. 3:43:48I can say from a sentence transform. Import.
  3367. 3:43:53sentence transformer it's a it's a class and we are going to create an
  3368. 3:43:58instance of this class let's do it let's create an instance of
  3369. 3:44:02this class and tell it which model we are
  3370. 3:44:04going to use to avoid any typo i'm going to copy this name from here i paste it
  3371. 3:44:12now let's have a few sentences a few
  3372. 3:44:16sentences that we want to convert into embeddings i guess or some
  3373. 3:44:21examples here so yes all I need are these examples
  3374. 3:44:26so I copy these examples and I paste them here
  3375. 3:44:31and I store the result of this line in a
  3376. 3:44:33variable called model because it's going to
  3377. 3:44:36be
  3378. 3:44:36the model that we use to embed our query or our sentences
  3379. 3:44:41Now let's create our main function.
  3380. 3:44:44I define my main function as keyword and our usual entry point.
  3381. 3:44:49I say if double underscore name
  3382. 3:44:52double underscore is equal to double underscore main double underscore.
  3383. 3:44:58In that case, I want the main function to be executed,
  3384. 3:45:01something that we have had a lot in this course.
  3385. 3:45:05Now inside main we can call the model and pass in our list of sentences.
  3386. 3:45:10These sentences I mean.
  3387. 3:45:12So here I use this model.
  3388. 3:45:14This model provides a method called encode and
  3389. 3:45:17it gets the sentences that we want to embed.
  3390. 3:45:20And it's nicer if we move these sentences
  3391. 3:45:24inside this function because it would be the only
  3392. 3:45:28place that we are going to use these sentences.
  3393. 3:45:32And I'm going to store the embeddings in a variable called embeddings.
  3394. 3:45:36So they say embeddings now. I'm gonna print embeddings shape.
  3395. 3:45:42I don't want to see the embedding themselves. I only want to the dimension.
  3396. 3:45:48So I run the program and the first time you run it, it may take a little
  3397. 3:45:52longer because the model needs to be downloaded, loaded and initialized.
  3398. 3:45:57And on subsequent subsequent runs, it will usually be faster.
  3399. 3:46:01I'm waiting for it to be executed. Okay. Weights are loaded. Okay.
  3400. 3:46:06And you can see this number here, four and 384. What does that mean?
  3401. 3:46:11The first number, I mean four, tells us the
  3402. 3:46:14model generated four, four embeddings, one
  3403. 3:46:18embedding for, for each sentence.
  3404. 3:46:19Here we had four sentences and now we have four embedding.
  3405. 3:46:24And the second number 384 tells us that each embedding contains three
  3406. 3:46:30384 numerical values.
  3407. 3:46:33In other words,
  3408. 3:46:34each sentence has been converted into a vector with 384 dimensions.
  3409. 3:46:42You can think of each vector as a point in a 384 dimensional space.
  3410. 3:46:47Since we have four sentences, now we have 4 points in that space.
  3411. 3:46:53And that brings us to the next question.
  3412. 3:46:56What interesting things can we actually do with these embeddings? Let's find out.
  3413. 3:47:02Oh
  3414. 3:47:08In this session, we are going to learn the basics of Semantic Search.
  3415. 3:47:11Let's say we have a collection of sentences and
  3416. 3:47:14we receive a new sentence called a query.
  3417. 3:47:16Our goal is to find the sentence in our collection
  3418. 3:47:20that is most similar in meaning to our query.
  3419. 3:47:23One way to do this is to calculate the similarity
  3420. 3:47:26between the query and each of those sentences in our
  3421. 3:47:30collection and then return the most similar one. But the question is, how
  3422. 3:47:35How do we do this? How do measure this distance? How do we calculate the similarity?
  3423. 3:47:41In the previous lesson, we converted each of our four
  3424. 3:47:46sentences into a point in a 384 dimensional space.
  3425. 3:47:51And we did it using an embedding model. Now we can do the same thing with our query.
  3426. 3:47:57We convert it into a vector in the same embedding space. And then in that space,
  3427. 3:48:03we can compare it with the embeddings that we generated for our sentences,
  3428. 3:48:08from our sentences from our collection. So the idea is simple.
  3429. 3:48:13find the sentence whose embedding is most similar
  3430. 3:48:17to the query embedding and return that sentence.
  3431. 3:48:21But this raises an important question.
  3432. 3:48:23How do we measure similarity between two vectors?
  3433. 3:48:27There are several metrics we can use but one
  3434. 3:48:30of the most common is cosine similarity.
  3435. 3:48:33Cosine similarity measures the cosine of the angle between two vectors.
  3436. 3:48:38In a simplified geometric picture, vectors pointing in
  3437. 3:48:42similar directions tend to have higher cosine similarity.
  3438. 3:48:46And
  3439. 3:48:46the score ranges from minus 1 to 1.
  3440. 3:48:50A score of 1 means the vectors point in the same direction.
  3441. 3:48:55A score means they are orthogonal, meaning there is no directional similarity.
  3442. 3:49:02and sq means they point in opposite direction.
  3443. 3:49:06And for a purpose, a higher cosine similarity
  3444. 3:49:10means a better match, means those two sentences
  3445. 3:49:13are closer. Alright, let's see how this works in code.
  3446. 3:49:18First, I should import util again from sentence transformers and the
  3447. 3:49:23This module provides several useful functions. One of which is cosine similarity.
  3448. 3:49:29It calculates cosine similarity between vectors. Let's remove these lines here.
  3449. 3:49:33The lines that we have for encoding the sentences and printing their shape.
  3450. 3:49:38Because we are going to handle the embeddings inside another function.
  3451. 3:49:43So I create another function for finding similarity.
  3452. 3:49:48Let's call this function find closest. closest. So I define a new function called
  3453. 3:49:54find
  3454. 3:49:56closest and this
  3455. 3:49:57function takes a query and a list of sentences and returns the sentence that is most
  3456. 3:50:05similar to that query so first we have query
  3457. 3:50:10and second we
  3458. 3:50:12have sentences
  3459. 3:50:13okay
  3460. 3:50:15pass and then inside my main function i'm gonna call this function i am gonna
  3461. 3:50:22say find closest
  3462. 3:50:24I'm calling it, I give it a query or we can put the query inside the variable.
  3463. 3:50:30I say I define
  3464. 3:50:32a new variable called query and I set it to I am
  3465. 3:50:36feeling great today.
  3466. 3:50:40So I pass a query here as its first argument, and then I pass my collection
  3467. 3:50:47of sentences.
  3468. 3:50:49And then I store the result in a variable called close a sentence, and
  3469. 3:50:55then I print it.
  3470. 3:50:57I print closest sentence. Okay, that's pretty much it.
  3471. 3:51:01Now the only thing left is to implement our find closest function here.
  3472. 3:51:07So here we need to embed both query and sentences.
  3473. 3:51:12So here I say model.encode.env.
  3474. 3:51:15and I send sentences and I do the same for the query.
  3475. 3:51:22and I store these two into separate variables, I call it.
  3476. 3:51:29embeddings and
  3477. 3:51:30I call the second one query
  3478. 3:51:32embedding.
  3479. 3:51:34query embedding.
  3480. 3:51:36okay that's it now we have the embedding for every
  3481. 3:51:39sentence in our collection as well as the embed
  3482. 3:51:42for our query our next task is to find the sentence
  3483. 3:51:46with the highest cosine similarity score to do
  3484. 3:51:49that we need to keep track of two things the index
  3485. 3:51:52of the closest sentence and its similarity score
  3486. 3:51:56so here we need two variables first closest index
  3487. 3:52:00and we need another variable called closest score
  3488. 3:52:05First, we initialize closest index to zero.
  3489. 3:52:08In other words, we are assuming that the first
  3490. 3:52:11item is the item that we are looking for.
  3491. 3:52:13And we also initialize the closest score to
  3492. 3:52:16the lowest possible cosine similarity score,
  3493. 3:52:19which is negative one. Now, this is the procedure.
  3494. 3:52:22We need to iterate through all sentences.
  3495. 3:52:25And if we find a closer embedding, so we update these two variables.
  3496. 3:52:31I mean index and the score. So we need to iterate over it.
  3497. 3:52:35Let's use enumerate function because we also need the index.
  3498. 3:52:39So I call enumerated and I give it my embeddings
  3499. 3:52:44and it gives me index as well as each vector
  3500. 3:52:48in my embedding. So I save for index as there as vector.
  3501. 3:52:54in my embeddings.
  3502. 3:52:57Okay, inside this loop, we'll calculate the cosine
  3503. 3:53:00similarity between the current sentence and our query.
  3504. 3:53:04So here I use the util module that we imported and I call
  3505. 3:53:08cosine similarity function and I give it the vector that we
  3506. 3:53:12have in this iteration as well as the query embedding.
  3507. 3:53:16Then I use the item method and I store the result in a variable called similarity.
  3508. 3:53:25okay this item method you see here extracts the
  3509. 3:53:29scalar similarity score from the returned tensor
  3510. 3:53:33yes it returns a tensor but we need a score and
  3511. 3:53:37this is a conversion next we need to check we need
  3512. 3:53:41to check whether the current sentence has a higher
  3513. 3:53:44similarity score than the best score we have seen
  3514. 3:53:47so far or not so if it does we update both closest index and closest score.
  3515. 3:53:54So here we need an if statement.
  3516. 3:53:56So I say if similarity
  3517. 3:53:58is greater than closest score, pay attention.
  3518. 3:54:03In that case we're gonna update both these two
  3519. 3:54:08i copy and paste them here here i put similarity and
  3520. 3:54:12then i say index the index that we have in this
  3521. 3:54:16iteration and the similarity that we calculated
  3522. 3:54:20okay so with this logic in our code as we continue
  3523. 3:54:24through the loop we keep comparing each sentence with our query and we keep updating
  3524. 3:54:30these variables and when we are done we have a closest index and index not
  3525. 3:54:37in text.
  3526. 3:54:38we have closest index and closest core and then we can return
  3527. 3:54:42so here at the end after this loop i'm going to return
  3528. 3:54:47sentences but the index that we found we calculated so here i use
  3529. 3:54:54closes index okay that's pretty much it we have implemented a basic semantic search
  3530. 3:55:00function from scratch and let's run and see what happens okay i run the program my
  3531. 3:55:05query was i'm feeling great today let's see what will happen okay here we have a
  3532. 3:55:11problem what is a problem oops my bad here we need to say index and here we
  3533. 3:55:17need to save similarity one more time i hope this time everything is all right
  3534. 3:55:24Okay, as you can see, the closest sentence to my query is today is a sunny day.
  3535. 3:55:31Okay, we implemented this function manually, so you
  3536. 3:55:34could understand what's happening behind the scenes.
  3537. 3:55:38But I should tell you, we don't have to do that.
  3538. 3:55:40I mean, Sentence Transformers already provides a utility
  3539. 3:55:44function called semantic search that can do the same for us.
  3540. 3:55:48There is an important difference, however.
  3541. 3:55:51semantic search i mean this utility function that
  3542. 3:55:54i'm telling you doesn't take raw sentences and you
  3543. 3:55:57need to embed your query and your sentences your
  3544. 3:56:00collection of sentences first and then you can
  3545. 3:56:03pass them as the argument of this function
  3546. 3:56:07this utility. function so here first i say
  3547. 3:56:11model.encode and you can borrow some code from here yes that's better
  3548. 3:56:19Okay.
  3549. 3:56:21I paste it here.
  3550. 3:56:23Okay.
  3551. 3:56:24I embed both my query and my sentences.
  3552. 3:56:29Then everything is ready to use this utility function. So I say util.
  3553. 3:56:34semantic search and first i send query query embedding then i send
  3554. 3:56:39embeddings sentence embeddings and we could we could use better names
  3555. 3:56:45here but it doesn't matter and then we specify the top k and i set it
  3556. 3:56:50to two and the top K you see here tells the function how many of the
  3557. 3:56:55most similar results we want the functionality that we didn't have
  3558. 3:57:01in our function in our own function it always
  3559. 3:57:04returns the most similar sentence but here you can
  3560. 3:57:08receive the top k most similar sentences
  3561. 3:57:12Anyway, let's store the result in a variable called results.
  3562. 3:57:16Then I print it and let's run our program to see what happens.
  3563. 3:57:21Okay, as you can see here, the output is a little
  3564. 3:57:24different from our manually implemented function.
  3565. 3:57:27Instead of returning the sentences themselves,
  3566. 3:57:30SemanticSearch returns a list of result dictionaries.
  3567. 3:57:34Each result indicates an ID, which is the
  3568. 3:57:37index of the matching sentence and a score.
  3569. 3:57:41which represents its similarity to the query.
  3570. 3:57:43For example, you can see here that it says the closest
  3571. 3:57:47one has the index 3, which is the last item.
  3572. 3:57:50And that is the similarity score.
  3573. 3:57:54So that brings us to the core idea behind Semantic Search,
  3574. 3:57:59which is encoding text into embeddings, compare
  3575. 3:58:03those embeddings using a similarity metric,
  3576. 3:58:07and retrieve the text that best matches the meaning of our query.
  3577. 3:58:13Welcome back aboard NeonLearn.
  3578. 3:58:15Planet Embeddings is conquered, three worlds down, seven to go.
  3579. 3:58:20Fasten your seat belts. Next stop, climate red.
  3580. 3:58:24Here, you'll build an AI support agent that answers from real company data
  3581. 3:58:28and begin your journey with Langchain.
  3582. 3:58:32Your next world awaits. Enjoy the journey.
  3583. 3:58:41Okay, from this point on, we are going to start
  3584. 3:58:44learning about RAG systems, which stands
  3585. 3:58:46for Retrieval Augmented Generation.
  3586. 3:58:49RAG is one of the most important topics in AI engineering.
  3587. 3:58:53You will see it mentioned frequently in job
  3588. 3:58:56descriptions, and many companies are looking
  3589. 3:58:58for engineers who know how to build RAG-based applications. But what exactly is RAG?
  3590. 3:59:05What problem does it solve and how does it work?
  3591. 3:59:08Okay, let's start with a problem we discussed
  3592. 3:59:11earlier and it relates to the limitations of
  3593. 3:59:14LLMs.
  3594. 3:59:15Remember when I described an LLM as something like a person, a person who has read
  3595. 3:59:22a huge amount of information available on the internet.
  3596. 3:59:26That analogy helps us understand
  3597. 3:59:28why LLMS can answer so many general questions.
  3598. 3:59:32They have been trained on enormous amounts of data.
  3599. 3:59:36they can often provide useful answers across a wide range of topics.
  3600. 3:59:41But, of
  3601. 3:59:42course, they have limitations.
  3602. 3:59:44And the first limitation is something called the knowledge cutoff.
  3603. 3:59:49A model's knowledge cut off refers to the point in time beyond which
  3604. 3:59:53its training data may not provide reliable information about events.
  3605. 3:59:58For example, if a model's training data only covers information up to 2018,
  3606. 4:00:04it might not know about events that happened in 2019 or later.
  3607. 4:00:10So if you go beyond knowledge cut off and if you ask a
  3608. 4:00:14model about a recent event it hasn't learned about,
  3609. 4:00:18it may not have the information it needs to answer the question correctly.
  3610. 4:00:23And another limitation is private access.
  3611. 4:00:27Models don't have access to private or company-specific information.
  3612. 4:00:31For example,
  3613. 4:00:33the model doesn't know about your company's internal policies.
  3614. 4:00:37It doesn't have access to
  3615. 4:00:38your customer records or the details of your business.
  3616. 4:00:42So as a result, there are many situations
  3617. 4:00:45in which an LLM cannot give us a good answer and that's because it doesn't
  3618. 4:00:51have the relevant information and that is
  3619. 4:00:54exactly the problem RAG is designed to solve.
  3620. 4:00:57And
  3621. 4:00:57One way to think about RAG is as a method for improving an LLM's response
  3622. 4:01:02by giving it access to relevant information at the time we ask a question.
  3623. 4:01:07If the model doesn't have up-to-date information, we can do that.
  3624. 4:01:11We can retrieve that information for the model and provide it as context.
  3625. 4:01:15If it doesn't know our company's internal policies, we
  3626. 4:01:19retrieve the relevant policies and provide them to the model.
  3627. 4:01:23This way the model can use information that wasn't necessarily part of its training
  3628. 4:01:28and we kind of compensate for that. And let's look at a more tangible example.
  3629. 4:01:35Imagine that I own a gym and I want to build a chatbot assistant for my business.
  3630. 4:01:40My gym has a website and I'd like to add a
  3631. 4:01:43chat bot button to the corner of the page.
  3632. 4:01:46So visitors can click on and ask a question, something that you have seen a lot.
  3633. 4:01:52Now imagine a customer asks, I attended one session, but I wasn't satisfied.
  3634. 4:01:59I'd like a refund. What should I do?
  3635. 4:02:01So the LLM for sure doesn't know about our gym's cancellation or refund policies.
  3636. 4:02:08So we need to provide that information somehow.
  3637. 4:02:11Otherwise, the model might give a generic answer.
  3638. 4:02:14And this brings us to another important problem.
  3639. 4:02:18Hallucinations.
  3640. 4:02:19In these scenarios, hallucination happens a lot.
  3641. 4:02:23And as we discussed earlier, LLMs are trained to predict and generate text.
  3642. 4:02:28And they are very good at this task.
  3643. 4:02:31And they try to give natural and convincing answers.
  3644. 4:02:36But sounding convincing doesn't necessarily mean being correct.
  3645. 4:02:41When a model doesn't have enough information, it might still generate
  3646. 4:02:46an answer instead of clearly acknowledging that it doesn't know.
  3647. 4:02:50For example, our gym chat pod may respond.
  3648. 4:02:54I'm sorry, you had a bad experience, we know.
  3649. 4:02:57You can contact the management for a refund.
  3650. 4:02:59And that's terrible because our gym might not have a refund policy.
  3651. 4:03:04So, chatbot is offering something that doesn't exist.
  3652. 4:03:08So, it's giving incorrect information and this is an example of hallucination.
  3653. 4:03:13The model generates information that isn't supported by facts.
  3654. 4:03:17And RAG systems are going to help reduce this problem.
  3655. 4:03:21And it is done by retrieving relevant information, of course,
  3656. 4:03:25from a trusted knowledge base like our policy documentation.
  3657. 4:03:29And provided to the model before it generates the response.
  3658. 4:03:33For example, instead of asking the LLM to
  3659. 4:03:36answer the customer's question in first place,
  3660. 4:03:39we can first retrieve our gym's actual cancellation policy.
  3661. 4:03:44Then we can provide both the customer question and the relevant policy.
  3662. 4:03:48And now the model has the information that it
  3663. 4:03:52needs to generate a more accurate answer.
  3664. 4:03:55And before going any further, there is something important to understand.
  3665. 4:04:01I mean Retrieval Augmented Generation is a general concept.
  3666. 4:04:06There isn't just one way to implement it.
  3667. 4:04:09The best implementation depends on the task,
  3668. 4:04:12on project, the data we are working with.
  3669. 4:04:14And a RAG system contains several components and
  3670. 4:04:18those components need to work together effectively.
  3671. 4:04:23And it brings us to another important thing about RAG systems, which is monitoring.
  3672. 4:04:29It's really important to monitor the system. It's very important to evaluate it.
  3673. 4:04:34Simply implementing something that runs without errors
  3674. 4:04:37doesn't necessarily mean we have built a good RAG system.
  3675. 4:04:41For example, one factor that can have a significant
  3676. 4:04:44impact is chunk size, and we will explore it.
  3677. 4:04:47But at this point, I'm telling you that observability and
  3678. 4:04:51evaluation is an important part of implementing RAG systems.
  3679. 4:04:55Now, before finishing this lesson, I'm going to elaborate on rag words.
  3680. 4:05:00I mean, let's break down the three words that make up rag.
  3681. 4:05:04Retrieval, augmented, and generation.
  3682. 4:05:07Let's start with the first word, retrieval.
  3683. 4:05:10Retrieval means finding and collecting the
  3684. 4:05:13information that's relevant to a user's question.
  3685. 4:05:16Let's return to our example, our gym example.
  3686. 4:05:19Suppose we have a 100 page PDF containing all the gym's guidelines and policies.
  3687. 4:05:26Okay, a customer asks about cancellations.
  3688. 4:05:30And here's a point where we don't need every single page of that PDF.
  3689. 4:05:34We only need the sections related to cancellation and refund policies.
  3690. 4:05:39So the process of finding those relevant sections is called information retrieval.
  3691. 4:05:45And in a RAG system, the component responsible for
  3692. 4:05:49finding that information is called retriever.
  3693. 4:05:52Okay.
  3694. 4:05:53The retriever searches our knowledge base and returns the
  3695. 4:05:57information that is most relevant to the user's query.
  3696. 4:06:01So the retriever's job is to find the right information and bring it back.
  3697. 4:06:06And as we will see how well it performs, this job has a
  3698. 4:06:10significant impact on the quality of the final answer.
  3699. 4:06:14So this component is extremely important. The second word is augmented.
  3700. 4:06:20Augmented means enhanced or supplemented.
  3701. 4:06:23Before we implement our RAG, our LLM might not
  3702. 4:06:27have had access to our GIMS' private policies.
  3703. 4:06:31But now, we have retrieved the relevant information and can provide it to the model.
  3704. 4:06:36We are kind of augmenting the model's input with
  3705. 4:06:39additional context, giving it information that it needs.
  3706. 4:06:43So, as a result, the model has a better basis for generating its response.
  3707. 4:06:49And finally, we have generation. This is the part that the LLM generates an answer.
  3708. 4:06:54So let's put the three parts together. First, we receive a query from the user.
  3709. 4:06:59And in our example, the user asks what they should do if
  3710. 4:07:03they want to cancel a gym membership or request a refund.
  3711. 4:07:06We send the query to a retriever.
  3712. 4:07:08The retriever searches our knowledge base, which contains the gym's PDF guidelines.
  3713. 4:07:15and then returns the relevant information about cancellations and reforms.
  3714. 4:07:20Now we have two things, the original query from the user and the relevant
  3715. 4:07:24information retrieved from our knowledge base. And then we combine these two pieces
  3716. 4:07:30and we send it as a prompt to our LLM. So the LL M uses the query and the
  3717. 4:07:35retrieved context to generate a response. Here's a complete flow.
  3718. 4:07:40Before we had a RAG system, we simply send the user's query to the LLM.
  3719. 4:07:44But as we have seen, that approach can fail when
  3720. 4:07:47the model doesn't have enough information.
  3721. 4:07:49However, with RAG, we retrieve the relevant information.
  3722. 4:07:53We provide it to the model as a context.
  3723. 4:07:56And this gives the model a better chance of producing a correct and useful answer.
  3724. 4:08:02And that's the whole idea behind retrieval augmented generation.
  3725. 4:08:17Okay, now that we understand the overall idea,
  3726. 4:08:20let's take a closer look at the retriever component and how it works.
  3727. 4:08:25Imagine our knowledge base contains the same 100 page PDF.
  3728. 4:08:29You might ask, why do we need a retriever at all?
  3729. 4:08:32Why don't we simply send the entire PDF to the LLM along with the user's query?
  3730. 4:08:37Wouldn't it be better to give the model all the information
  3731. 4:08:41so it can find the answer itself? Well, not necessarily.
  3732. 4:08:45There are several catches, there are several important
  3733. 4:08:49reasons why this isn't always a good idea.
  3734. 4:08:51First, remember that every LLM has a context window.
  3735. 4:08:55The context window limits how much information
  3736. 4:08:58the model can process in a single interaction.
  3737. 4:09:01So, this includes the prompt, the conversation history, retrieved
  3738. 4:09:06information, and the space needed for the generated response.
  3739. 4:09:10So if we put the entire 100-page PDF into the prompt,
  3740. 4:09:15we could consume a large part of our context window.
  3741. 4:09:19In the worst case, the input could exceed the
  3742. 4:09:22model's limit and the request might fail.
  3743. 4:09:26even when the entire document fits, using so much context may be unnecessary.
  3744. 4:09:32So, it brings us to the second reason, cost.
  3745. 4:09:36When it's unnecessary, we are burning our tokens.
  3746. 4:09:39When we use a paid API, we're charged based on token usage.
  3747. 4:09:44So, if we send the entire knowledge base with query question,
  3748. 4:09:48we may repeatedly pay to process large amounts of
  3749. 4:09:51information that aren't relevant to the question.
  3750. 4:09:54even if we are running the model locally, we
  3751. 4:09:58are wasting hardware, time and computational
  3752. 4:10:01resources in general and it increases latency
  3753. 4:10:04in our system so we cannot respond to other
  3754. 4:10:07users.
  3755. 4:10:08To make it more clear, let's look at an example.
  3756. 4:10:13our knowledge base contains 100,000 tokens.
  3757. 4:10:16Now, if 10 users ask questions and we send the entire knowledge base with query,
  3758. 4:10:22that could mean processing around 1 million input tokens.
  3759. 4:10:26So ignoring other stuff, it could be 1 million
  3760. 4:10:30input tokens, but what if each question only needs a small section of documents?
  3761. 4:10:35For example, it only needs around
  3762. 4:10:37100 tokens instead of 100,000 tokens.
  3763. 4:10:40So in that case, across 10 requests, that's only around
  3764. 4:10:441,000 token.
  3765. 4:10:46So you can compare 1 million input tokens and 1, 000 input tokens.
  3766. 4:10:51That's a big difference.
  3767. 4:10:53So the principle is clear.
  3768. 4:10:55Retributing only the relevant information can greatly reduce
  3769. 4:11:00the amount of unnecessary context.
  3770. 4:11:03So it was the second reason. And the third reason is answer quality.
  3771. 4:11:08Giving the model more information doesn't
  3772. 4:11:11automatically mean giving it better information.
  3773. 4:11:14Imagine our gym has a swimming pool, a weight
  3774. 4:11:17training area, and several other facilities.
  3775. 4:11:20Our PDF contains information about all these stuff.
  3776. 4:11:23Now a customer asks about the cancellation policy as I told you.
  3777. 4:11:28So if we send the entire PDF,
  3778. 4:11:30The model also receives information about pool depth, swimming rules,
  3779. 4:11:35children's safety, gym equipment, and many other topics that are irrelevant.
  3780. 4:11:40So, I acknowledge that modern LLMs use attention mechanism, and this mechanism
  3781. 4:11:46helps them process relationships between different parts of the input.
  3782. 4:11:52But irrelevant information can still make it harder
  3783. 4:11:55to focus on the information that actually matters.
  3784. 4:11:59So that's really important and we don't just
  3785. 4:12:02want to give the model more information.
  3786. 4:12:05We want to get it the right information and
  3787. 4:12:08that's where retrieval becomes so important.
  3788. 4:12:12Now after knowing that retriever and retrieval augmented generation is important,
  3789. 4:12:17the question is that how do we solve this problem? If the problem is
  3790. 4:12:22We cannot give the model the entire document. How do we solve this problem?
  3791. 4:12:28What should we do?
  3792. 4:12:29One approach is to use the embeddings that we just learned in the previous project.
  3793. 4:12:34Let's say our knowledge base is that 100-page PDF.
  3794. 4:12:39The first thing we can do is to split it into smaller pieces.
  3795. 4:12:43For example, we might split it into paragraphs or sections.
  3796. 4:12:47And this process is called chunking.
  3797. 4:12:49For example a 100-page PDF might be converted
  3798. 4:12:53into hundreds or even thousands of chunks.
  3799. 4:12:56And remember, our knowledge base doesn't have to be a PDF.
  3800. 4:13:00It could contain Excel files, web pages, internal
  3801. 4:13:04documentation or many other types of data.
  3802. 4:13:08And once we have split the context into chunks,
  3803. 4:13:11we generate an embedding for each chunk.
  3804. 4:13:14In other words, we convert each chunk into a numerical vector in an embedding space
  3805. 4:13:21that we saw in the previous project.
  3806. 4:13:23Now our knowledge base is represented by a
  3807. 4:13:26collection of vectors instead of simple text.
  3808. 4:13:29And each vector is associated with its original text. Now what happens?
  3809. 4:13:35What happens when a user asks a question? That's really simple.
  3810. 4:13:38We take that question. and pass it through the embedding model that we have.
  3811. 4:13:42So we convert the query into a vector in the same space.
  3812. 4:13:47Then we compare the query embedding with the embeddings for chunks.
  3813. 4:13:52We can use a similarity measurement, such as the cosine that you saw.
  3814. 4:13:57So we can find which chunks are most relevant to the query.
  3815. 4:14:02And then the system receives the top matching chunks.
  3816. 4:14:06Another question that you might have is why do we
  3817. 4:14:09retrieve multiple chunks instead of just one chunk?
  3818. 4:14:13because the information might be spread across several parts of our knowledge base.
  3819. 4:14:19For example, one chunk might describe the cancellation
  3820. 4:14:23deadline while another explains the refund conditions.
  3821. 4:14:27So retrieving multiple relevant chunks can give the model
  3822. 4:14:31a more complete picture which is exactly what it needs.
  3823. 4:14:36So once retrieval is complete, we have the
  3824. 4:14:39original query and the relevant text chunks.
  3825. 4:14:42So we combine these two.
  3826. 4:14:44We combine retrieved chunks and the query in
  3827. 4:14:47a single prompt and send them to the LLM.
  3828. 4:14:50And the LLLM can use the retrieved information to generate an augmented answer.
  3829. 4:14:56So let's summarize the process into this. First we have chunking.
  3830. 4:15:00We split the knowledge base into smaller pieces. Then we have embedding.
  3831. 4:15:05We convert each chunk into a vector. Then we query embedding,
  3832. 4:15:10we convert the query into a Vector using the same model.
  3833. 4:15:14And we have retrieval, we use a similarity metric
  3834. 4:15:18and we find the chunks whose embeddings are most relevant to the query.
  3835. 4:15:23And finally we have generation.
  3836. 4:15:25We provide the query and retrieve chunks to the LLM.
  3837. 4:15:28so it can generate an answer.
  3838. 4:15:31And that's the core idea behind a basic embedding based RAG system.
  3839. 4:15:36And in upcoming lessons, we will take a closer look at the Retriever
  3840. 4:15:40and explore how to implement these steps in action.
  3841. 4:15:47Ladies and gentlemen, a quick update from the control tower.
  3842. 4:15:52Right now, AI engineer is the fastest growing job in the United States,
  3843. 4:15:56ranked number one on LinkedIn's jobs on the rise list, two years in a row.
  3844. 4:16:02And the top skills listed for this role, Lang Chain and Rag.
  3845. 4:16:06The exact skills you're building on this journey.
  3846. 4:16:09Companies are looking for people who can build with AI. The runway is ready.
  3847. 4:16:15So stay consistent, one lesson at a time, one project at a time.
  3848. 4:16:20Every hour you invest is fuel for your future. Your effort will pay off.
  3849. 4:16:27you
  3850. 4:16:33All right, from this point on, we are going to build a RAG project from scratch.
  3851. 4:16:39The first thing that we need is a knowledge base.
  3852. 4:16:41And for this project, we're going to use a PDF containing
  3853. 4:16:46information about the symptoms of different diseases.
  3854. 4:16:50The document is 25 pages long and contains information about approximately
  3855. 4:16:56100 different diseases, it has a paragraph describing each disease.
  3856. 4:17:02and I will upload this PDF and include a link
  3857. 4:17:05in the description of this YouTube video.
  3858. 4:17:08So you can download it and follow along with me. So that's our knowledge base.
  3859. 4:17:14And this is a great document for practicing RAG
  3860. 4:17:17because it gives us a collection of information
  3861. 4:17:21that we can split into chunks, converts them into embeddings,
  3862. 4:17:25and eventually used to retrieve relevant information.
  3863. 4:17:29After that, we will move on to vector databases and
  3864. 4:17:33I will explain why they are useful in RAG systems.
  3865. 4:17:37Not only useful, but also necessary.
  3866. 4:17:41So let's get started
  3867. 4:17:43and let's create a new Python file called rag.py.
  3868. 4:17:49And as usual, I want to have my main function.
  3869. 4:17:54and my entry point. So I say name is equal to main
  3870. 4:18:01in that case execute my main function so the
  3871. 4:18:05first task is to read the pdf file there are
  3872. 4:18:09several ways to do this but for this project
  3873. 4:18:13i'm going to use a package called py pdf
  3874. 4:18:16and we will also use langchain a very popular framework in ai engineering so first
  3875. 4:18:23let's install these two so i open up my terminal and here i say pipenv install
  3876. 4:18:33LangChain as well as PyPDF.
  3877. 4:18:36Before start coding, let me briefly explain what
  3878. 4:18:40LangChaine is and why we are going to use it.
  3879. 4:18:44LangChains is a widely used framework that helps us
  3880. 4:18:47build AI applications and it makes it easier for us.
  3881. 4:18:51We can easily build AI application in an organized way.
  3882. 4:18:55It provides reusable components that can help us write less code
  3883. 4:19:00and we can integrate different parts of our application
  3884. 4:19:04and it makes it easier to maintain and extend our application.
  3885. 4:19:08We are going to use LangChain to implement parts of
  3886. 4:19:11our RAG system, but keep in mind that it is not limited to RAG systems.
  3887. 4:19:15It can be used for many other AI engineering tasks.
  3888. 4:19:19And you will also hear about LangGraph. What is the difference?
  3889. 4:19:24LangGraph is particularly useful when we need more explicit control when
  3890. 4:19:29we are dealing with multi-step workflows including agentic system.
  3891. 4:19:34Later we will take a look at LangGraph as well.
  3892. 4:19:37Here LangChain is enough for us, is sufficient, is
  3893. 4:19:40all we need for implementing this RAG project.
  3894. 4:19:44And my general recommendation is to start with simpler tools.
  3895. 4:19:49As long as they meet your needs, that's okay.
  3896. 4:19:52You don't need to move to something more sophisticated.
  3897. 4:19:56I see that a lot, but some developers prefer to overcomplicate a problem.
  3898. 4:20:01So first use LangChain and when LangChains is not enough, you can go with LangGraph.
  3899. 4:20:07Okay, back to our code. The first thing we need to do is to read our PDF
  3900. 4:20:12file and
  3901. 4:20:12we want to extract its text. So, as I told you, I'm going to use PyPDF.
  3902. 4:20:18After that, I will use Langchain document object.
  3903. 4:20:22What exactly is a document object?
  3904. 4:20:25In Langchain, a document is a standard data structure
  3905. 4:20:29that contains two main things, page content and metadata.
  3906. 4:20:33Page content is the actual text or content of the document.
  3907. 4:20:38and metadata is additional information about the content.
  3908. 4:20:42For example, the metadata might include the source files name, the page number,
  3909. 4:20:47or other information that helps us identify where the content came from.
  3910. 4:20:52And this standard format is useful because it allows different
  3911. 4:20:56components to work with information in a consistent way.
  3912. 4:21:00It doesn't matter whether our original knowledge base comes
  3913. 4:21:04from a PDF, a spreadsheet, a web page, or another source.
  3914. 4:21:08We can extract the content.
  3915. 4:21:10represent it using the same document structure and this becomes particularly
  3916. 4:21:15useful when we start connecting components
  3917. 4:21:18together in a RAG pipeline the output from one
  3918. 4:21:22component can become the input to another component
  3919. 4:21:26and that's the purpose of document object
  3920. 4:21:29so let's start by reading our knowledge base which is a pdf
  3921. 4:21:34so first i import pdf reader from pypdf so i say from
  3922. 4:21:39pypdf import pdf reader then we need to import that document object that i told you
  3923. 4:21:46it's a class so i say from langchain core
  3924. 4:21:51documents okay import document okay that's
  3925. 4:21:56it
  3926. 4:21:57One thing worth mentioning here is that you may encounter older
  3927. 4:22:01tutorials that import components from the Langchain community.
  3928. 4:22:05Currently, the policy of Langchain is to provide standalone packages.
  3929. 4:22:10So instead of having Langchain Community which was previously contained many, many
  3930. 4:22:16tools and methods and functions and classes, now we have standalone packages
  3931. 4:22:22So when you are following older tutorials, always check the current
  3932. 4:22:26documentation to see which package provides the components you are using.
  3933. 4:22:31So if you see LangChain Community, at that point search for and Standalone Package.
  3934. 4:22:37So let's create a function to read our PDF.
  3935. 4:22:40I define a new function called readPDF file which takes
  3936. 4:22:44file name or more accurately it's the address of our file.
  3937. 4:22:48Next, inside this function, we're going to create an empty list called docs.
  3938. 4:22:54The process is that we'll create a PDFReader object and pass it
  3939. 4:22:58our file path and it gives us access to the pages of our PDF.
  3940. 4:23:03So first I create this object, I say PdfReader and I pass our path.
  3941. 4:23:09and we will get a reader, a reader object and then using this reader
  3942. 4:23:14we can access to pages of our document this way so reader.pages
  3943. 4:23:19it is an iterable, we can iterate over it so I'm going to loop over it
  3944. 4:23:26and since I also need index, I use enumerate it gives me both page and index
  3945. 4:23:32so I save for I and page in these pages
  3946. 4:23:37and inside the loop, we'll create a document object for each page, I mean.
  3947. 4:23:43As we discussed earlier, each document contains the page's content and its metadata.
  3948. 4:23:48I told you that it has two components.
  3949. 4:23:51So we can extract the text from the current page and we can use the name of our file
  3950. 4:23:56as well as its page using this index to be used as our metadata.
  3951. 4:24:01So I'm going to create an instance of this document object for each each page
  3952. 4:24:06and it has two components page content so first i
  3953. 4:24:10extract it using the extract text method and it
  3954. 4:24:14gives me the actual content of that page and here
  3955. 4:24:18i use an or logical operator a page might be
  3956. 4:24:22not so in that case i set it to an empty string and as it's metadata
  3957. 4:24:28we have two pieces of information first we have
  3958. 4:24:31source i give it the file name and then i specify
  3959. 4:24:34the page number that this page comes from so i
  3960. 4:24:38say page and i set it to i plus one because
  3961. 4:24:41indexes start from zero okay now we have all
  3962. 4:24:45we need and following that we can append
  3963. 4:24:49this document that we created to our list so first i set
  3964. 4:24:53store it in a variable called doc and then I
  3965. 4:24:56say docs.append and I add this doc to it.
  3966. 4:25:00And finally once we have processed every page, I mean after
  3967. 4:25:04this loop, we return this docs list that we created.
  3968. 4:25:08That's our PDF reader.
  3969. 4:25:10Now before we test the function, let's take a
  3970. 4:25:13moment to understand why we are storing metadata.
  3971. 4:25:16I mean source and this page number here.
  3972. 4:25:19Metadata is pretty useful when you are developing a RAG system.
  3973. 4:25:23You need it in many different stages, from development
  3974. 4:25:27to testing to evaluation and other stages.
  3975. 4:25:29So we kind of need to track where our information comes from.
  3976. 4:25:33For example, imagine that our RAG System answers a
  3977. 4:25:37question using information retrieved from page 14.
  3978. 4:25:41And if we preserve the source page number,
  3979. 4:25:44we can identify the original location of that information.
  3980. 4:25:48It can help us provide references to users.
  3981. 4:25:51So it makes it more factual and it makes debugging easier as well.
  3982. 4:25:56For example, if our chatbot produces an incorrect answer,
  3983. 4:26:01we can investigate which document and page were retrieved as context.
  3984. 4:26:06And that makes it easier to understand whether
  3985. 4:26:08the problem came from the retrieval process,
  3986. 4:26:11the source material, or the model's response.
  3987. 4:26:14So metadata isn't just extra information, it's an important part of
  3988. 4:26:19making our RAG system easier to inspect, evaluate and
  3989. 4:26:23maintain as well as providing user with factual information.
  3990. 4:26:27This way users can better trust us when we provide the reference.
  3991. 4:26:31Let's test our program up to this point. So here I'm going to call this.
  3992. 4:26:37I call this function, the function that we built.
  3993. 4:26:40And as its argument, I pass this PDF.
  3994. 4:26:44Okay, I say symptoms.pdf. Okay, and it gives me a list of documents.
  3995. 4:26:51So I store it in a variable called docs.
  3996. 4:26:54And
  3997. 4:26:54then I print the length of this list as well as its first item.
  3998. 4:27:01I say docs zero. Now let's run our program.
  3999. 4:27:06Okay.
  4000. 4:27:07As you can see here we have 25 pages.
  4001. 4:27:11Okay, it was the length and the first item in this
  4002. 4:27:15list, which is the first document actually we have.
  4003. 4:27:19So first we have its content and then we have the metadata.
  4004. 4:27:23It's the source that we have and this is the page number.
  4005. 4:27:27So here we had 25 document objects and each document
  4006. 4:27:32object contains both content and metadata.
  4007. 4:27:36So we have successfully read our PDF and convert
  4008. 4:27:39its pages into a LangChain document objects.
  4009. 4:27:42This is our first step toward building the RAG pipeline.
  4010. 4:27:52Alright, what have we done so far?
  4011. 4:27:55We started with a 25-page PDF file and converted into 25 document objects.
  4012. 4:28:01Now, you might think we can simply take these document
  4013. 4:28:04objects and pass them to our embedding model.
  4014. 4:28:07The model would convert each document into a
  4015. 4:28:10vector in our 300-dimensional embedding space.
  4016. 4:28:12But there is a problem.
  4017. 4:28:14A full page is too much information to represent as a single embedding.
  4018. 4:28:18and this is where domain expertise becomes important.
  4019. 4:28:23If you look at our PDF, you will notice that each page can contain information about
  4020. 4:28:29four or five different diseases. If we embed the entire page as a single unit,
  4021. 4:28:35the resulting vector has to represent all of that information at once.
  4022. 4:28:40And the problem is that the embedding might not
  4023. 4:28:42capture the specific information we are looking for.
  4024. 4:28:45For example, if a user asks about the symptoms of a particular disease,
  4025. 4:28:50we want to retrieve the section that describes that disease,
  4026. 4:28:54not an entire page containing several unrelated diseases.
  4027. 4:28:58And here's the important part.
  4028. 4:29:00If we don't store our information at the right level of detail,
  4029. 4:29:05we cannot expect our retrieval process to be accurate either.
  4030. 4:29:09So, instead of treating an entire page as one
  4031. 4:29:12document, we want to split it into smaller pieces.
  4032. 4:29:16Ideally chunks that contain a paragraph or a short section about a single disease.
  4033. 4:29:23And this process is called chunking. Now,
  4034. 4:29:28Splitting text isn't particularly difficult.
  4035. 4:29:31I mean, we could write a simple function to break a string into smaller sections.
  4036. 4:29:37but in real-world projects, developers usually rely on existing text splitters.
  4037. 4:29:42Langchain provides several text splitters that
  4038. 4:29:46work directly with its document objects.
  4039. 4:29:49So since we are already using Langchain and its document object,
  4040. 4:29:54it makes sense to use its built-in tools and follow the framework's conventions.
  4041. 4:29:59So to use these text split-ers, we need to install an additional Langchain package.
  4042. 4:30:05which is Langchain text splitters. So here,
  4043. 4:30:09I'm going to install pipenv install langchain text splitters.
  4044. 4:30:16Okay, and before we implement this, let's take a moment
  4045. 4:30:21to understand the basic approaches to text splitting.
  4046. 4:30:25The simplest approach is to split text based on a fixed size.
  4047. 4:30:29For example, we could say that every chunk should
  4048. 4:30:33contain 1000 characters or approximately 100 tokens.
  4049. 4:30:37But there is a problem with this approach.
  4050. 4:30:39It can split the text in the middle of a sentence or even
  4051. 4:30:43in the middle of a word. Imagine that a sentence starts at the end of one chunk
  4052. 4:30:49and continues in the next chunk. So the meaning can become harder to understand
  4053. 4:30:55when those chunks are retrieved separately. To address this problem,
  4054. 4:30:59we can use a strategy called recursive character splitting.
  4055. 4:31:04The idea is to try splitting the text using natural boundaries,
  4056. 4:31:09starting with larger sections, and gradually moving to smaller ones.
  4057. 4:31:14For example, the splitter can first try to separate the text at paragraph breaks.
  4058. 4:31:20If a resulting section is too large, it tries sentence-level separators like period,
  4059. 4:31:26and if the text is still too large, it can fall back to smaller separators,
  4060. 4:31:32such as spaces or individual characters.
  4061. 4:31:35In this strategy, the main principle is to preserve
  4062. 4:31:38the natural structure as much as possible,
  4063. 4:31:41while keeping each chunk within the target size.
  4064. 4:31:44So, this is why recursive splitting is a useful
  4065. 4:31:47general purpose strategy for many RAG applications.
  4066. 4:31:52And for this project, we'll use recursive characters. character text splitter.
  4067. 4:31:57So let's import it. I'm going to say from LangchainTextSplitter,
  4068. 4:32:03import recursive character textSplitters.
  4069. 4:32:06And despite the word character you see here,
  4070. 4:32:09this splitter can also work with token-based length measurements.
  4071. 4:32:14and in many cases, measuring chunk size in tokens is more useful,
  4072. 4:32:19more useful than measuring it in characters because language models process tokens
  4073. 4:32:24and have token-based context limits
  4074. 4:32:26and they describe everything with tokens rather than characters.
  4075. 4:32:30So it's better to use tokens instead of characters for splitting our chunks.
  4076. 4:32:35Now you might ask if you are splitting our chunk based on tokens,
  4077. 4:32:40which tokenizer should be used? Honestly,
  4078. 4:32:42For this task we don't necessarily need the
  4079. 4:32:45exact tokenizer used by our eventual LLM.
  4080. 4:32:48Here we need an estimate. We are looking for a practical estimate of chunk size.
  4081. 4:32:53So we can use a tokenizer such as tiktoken even if our model is not an open AI model.
  4082. 4:33:00So here inside our main function I'm going to use this text splitter
  4083. 4:33:07and
  4084. 4:33:08it gives me a method called from tick token encoder
  4085. 4:33:12which uses tick token tokenizer to split our chunks
  4086. 4:33:16and i set chunk size to 110 and chunk overlap to 10 i will explain don't
  4087. 4:33:23worry okay so let's look at these two parameters
  4088. 4:33:28that might have confused you the first one is chunk size
  4089. 4:33:33we have set it to 110 which means
  4090. 4:33:35We want each chunk to contain approximately 110 tokens.
  4091. 4:33:40The second parameter is chunk overlap, which we have set to 10.
  4092. 4:33:44This means that adjacent chunks can share approximately 10 tokens.
  4093. 4:33:50It helps us preserve some context across chunk boundaries.
  4094. 4:33:55For example, imagine a simplified case where the
  4095. 4:33:59first chunk contains tokens 0 through 109.
  4096. 4:34:03So the next chunk might begin around token 100, 10 tokens earlier.
  4097. 4:34:09So it shares the final 10 tokens of the previous chunk.
  4098. 4:34:14And the following chunk can then continue from around token 200.
  4099. 4:34:20And the main question is that why do we need chunk overlap?
  4100. 4:34:25Let's say an important sentence begins near the end
  4101. 4:34:29of one chunk and continues into the next junk.
  4102. 4:34:32and without this strategy, without overlapping,
  4103. 4:34:35the two pieces might be retrieved separately
  4104. 4:34:38and the relevant chunk could be incomplete.
  4105. 4:34:42However, with overlap, some of that information
  4106. 4:34:45is repeated in the neighboring chunk,
  4107. 4:34:48and this increases the chance that a retrieved chunk will
  4108. 4:34:52contain enough surrounding context to be useful to the LLM.
  4109. 4:34:57And of course, overlap isn't free.
  4110. 4:34:59It creates some duplicated text, which means additional storage and embedding work.
  4111. 4:35:05And too much overlap can also lead to redundant retrieval results.
  4112. 4:35:10So you should be careful, it depends on your task.
  4113. 4:35:13We need to find a reasonable balance and keep in mind that a
  4114. 4:35:18good retriever is built through experimentation and evaluation.
  4115. 4:35:22The values we are using here are starting points, not universal rules.
  4116. 4:35:27Depending on content of your documents and the kinds of questions users ask,
  4117. 4:35:33you might get better results with smaller
  4118. 4:35:36chunks, larger chunks or a different overlap.
  4119. 4:35:39The right way to decide is to inspect the chunks,
  4120. 4:35:42take the retrieval results and adjust these numbers based on what you observe.
  4121. 4:35:51Now let's store what it returns in a variable called splitter.
  4122. 4:35:56and now now that we have configured our splitter
  4123. 4:35:59and we have it we can use the splitter it gives
  4124. 4:36:02a method called split documents and we can pass
  4125. 4:36:06our documents to it so i call splitter.split
  4126. 4:36:09documents and it receives our docs and it returns our chunks
  4127. 4:36:17Here, notice that we are passing the entire
  4128. 4:36:19list of documents, not just a single document.
  4129. 4:36:22Langchain will split their contents into smaller
  4130. 4:36:26chunks and return a new list of document objects.
  4131. 4:36:29So that's really important. These chunks retain their associated metadata.
  4132. 4:36:34They retain their structure, which were document objects,
  4133. 4:36:39And that's really useful because as you can see, we simply pass 25
  4134. 4:36:43document objects and we will receive some chunks as document objects.
  4135. 4:36:48So here I print the length of this list to see how many chunks we have.
  4136. 4:36:55And then I print. The first chunk.
  4137. 4:36:59okay so let's run our program okay and as you can see
  4138. 4:37:05here we have 183 documents here and this is our
  4139. 4:37:10first chunk as and as we can see it's a document
  4140. 4:37:14object it has content and it has method metadata
  4141. 4:37:18Okay, let's summarize what we have done so far. We started with a 25 page PDF file.
  4142. 4:37:24We converted it into 25 document objects, one for each page.
  4143. 4:37:29Then we split those documents into smaller chunks,
  4144. 4:37:32creating a new list of document objects called chunks
  4145. 4:37:36that are better suited for embedding and retrieval.
  4146. 4:37:39And our pipeline now looks like this.
  4147. 4:37:42A 25 page pdf, 25 document object,
  4148. 4:37:47Then we have chunking, then we have a smaller document objects now.
  4149. 4:37:51After these steps, we are ready for the next step,
  4150. 4:37:55which is converting these chunks into embeddings.
  4151. 4:37:59Ladies and gentlemen, this is a flight update from the cockpit.
  4152. 4:38:03We've now been flying for about four hours and we have roughly four more hours
  4153. 4:38:08until our final destination.
  4154. 4:38:10We're currently exploring planet Rag, one of the
  4155. 4:38:13most important worlds on this entire journey.
  4156. 4:38:17So please return your seat to the upright position
  4157. 4:38:20and give your full attention to your surroundings.
  4158. 4:38:23What you learn on this planet is one of the most valuable skills in AI engineering.
  4159. 4:38:36Alright, we have reached the point where we have 183 document objects.
  4160. 4:38:41Now, we want to map the content of these documents
  4161. 4:38:45into our high dimensional embedding space.
  4162. 4:38:48More precisely, we wanted to convert each chunk into a vector in that space.
  4163. 4:38:54And this isn't anything new.
  4164. 4:38:56We have already done this in an earlier lesson using the AllMiniLM model.
  4165. 4:39:02And we are going to use the same model here. But there is a difference here.
  4166. 4:39:06Previously, I loaded the model using the sentence transformers library.
  4167. 4:39:11This time, I want to use LangChains integration instead.
  4168. 4:39:15So in order to keep our application consistent, I wanted to do that.
  4169. 4:39:20And this is one of the frustrating things about AI
  4170. 4:39:24engineering, especially when you are new to this field.
  4171. 4:39:28you might notice that different developers use different
  4172. 4:39:32libraries and approaches to accomplish exactly the same task.
  4173. 4:39:36It makes you confused and frustrated.
  4174. 4:39:39But honestly, there is usually a reason behind the approach that we choose.
  4175. 4:39:44You need to consider the architecture of your application first.
  4176. 4:39:48In our case, we are using Langchain. Okay, so the best option is this
  4177. 4:39:54bridge that is between Langchain and HuggingFace
  4178. 4:39:58and it makes our application more consistent.
  4179. 4:40:02So let's do it.
  4180. 4:40:03Let's first install the package that is for
  4181. 4:40:06using Huggingface models in our application,
  4182. 4:40:09in a Langchain application. So I say pipenv install langchainhuggingface.
  4183. 4:40:15Okay, so that's the whole point of this package.
  4184. 4:40:18It's a bridge between Huggingface and Langchain.
  4185. 4:40:23Now let's import the embedding class. There is a class called HuggingFaceEmbeddings.
  4186. 4:40:29And whenever you want to load an embedding model, you're going to use this class.
  4187. 4:40:34So I say from LangChainHuggingFaces, importHugingFaceEmbeddings.
  4188. 4:40:41This is a that we need. And then we can initialize our model using this class,
  4189. 4:40:47so I create an object using this glass.
  4190. 4:40:51and we need to specify some fields and the first one is our model
  4191. 4:40:57here I say model name which is allminiLM let's copy its name so
  4192. 4:41:03Okay, I copy its identifier and I paste it here.
  4193. 4:41:08Okay, this tells LangChain which embedding model we want to use.
  4194. 4:41:12And then I'm going to configure one additional
  4195. 4:41:15option, which is normalize embeddings.
  4196. 4:41:18Let's set it first and I will explain what is happening.
  4197. 4:41:21So I say encode keyword arguments and then it is a dictionary
  4198. 4:41:27and I say normalize Embeddings and I set it to true.
  4199. 4:41:33This thing that you're seeing here is related to linear algebra,
  4200. 4:41:37but I want to try my best to simplify it.
  4201. 4:41:40Okay, remember that each embedding is a vector in a high dimensional space.
  4202. 4:41:45You can think of a vector as an arrow that
  4203. 4:41:48starts at the origin of coordination system
  4204. 4:41:51and points toward a location in that space.
  4205. 4:41:54And different vectors can have different lengths.
  4206. 4:41:57And when we set normalize embeddings true,
  4207. 4:42:01The embedding model scales each vector so that its length becomes 1.
  4208. 4:42:07This process is called L2Normalization.
  4209. 4:42:10And the question is why would we want to do that?
  4210. 4:42:13Well, in our RAG application, we will eventually compare vectors
  4211. 4:42:17to find the text chunks that are most similar to the user's query.
  4212. 4:42:22And when vectors are normalized,
  4213. 4:42:25calculating cosine similarity becomes more convenient.
  4214. 4:42:29In fact, for unit length vectors, cosine
  4215. 4:42:32similarity is equivalent to their dot product.
  4216. 4:42:35So it would be more efficient if we normalize our vectors. Now we
  4217. 4:42:40have the model.
  4218. 4:42:41I store it in a variable called model. Now we can use our model.
  4219. 4:42:47Okay, let's get back to our main function.
  4220. 4:42:50I remove these print statements here that we use to inspect our chunks.
  4221. 4:42:55Now we want to use our model.
  4222. 4:42:57Langchain gives us a method called embedDocuments and we can use it
  4223. 4:43:02to generate embeddings in one call, which is really convenient.
  4224. 4:43:06However, there is one important detail.
  4225. 4:43:08EmbeddingDocuments expects a list of strings,
  4226. 4:43:11not a list a LangchainDocument objects.
  4227. 4:43:14our chunks variable contains a document object.
  4228. 4:43:17So first we need to extract the text from each one.
  4229. 4:43:22And we can do this using list comprehension. So I say for each
  4230. 4:43:27chunk in our chunks list, I'm going to access its page content.
  4231. 4:43:33So I say
  4232. 4:43:35chunk.
  4233. 4:43:36page.
  4234. 4:43:38content.
  4235. 4:43:39Okay. That's it.
  4236. 4:43:41And I store it in a variable called texts and
  4237. 4:43:44this creates a list containing the text content
  4238. 4:43:47of every chunk. Now we can pass it to the method that I mentioned.
  4239. 4:43:52I say model embed documents and I pass this list
  4240. 4:43:56that I just created and I store the result
  4241. 4:44:00in a variable called vectors and that's it!
  4242. 4:44:04The model will generate an embedding vector for each text chunk.
  4243. 4:44:07Now, you might be wondering, what happened
  4244. 4:44:10to the consistency we talked about earlier?
  4245. 4:44:13Didn't we say that we want to preserve LangChain's document format?
  4246. 4:44:18And now, what are we doing here? Yes, you are right.
  4247. 4:44:21For now, I'm extracting the text and generating
  4248. 4:44:24these embeddings separately to show you.
  4249. 4:44:27But in the next session, we will use a different approach.
  4250. 4:44:30let's print these vectors just like before i'm
  4251. 4:44:34going to first print the length of these vectors
  4252. 4:44:38and second we
  4253. 4:44:40are going to print the first item of this list
  4254. 4:44:45so I say vectors 0 let's run our program
  4255. 4:44:50everything is in order and
  4256. 4:44:52here we can see that we have 183 vectors the
  4257. 4:44:56same as the number of chunks that we had
  4258. 4:45:00and this is the embedding of the first chunk so the first
  4259. 4:45:05trunk which is something like these tokens these words
  4260. 4:45:09are converted to this vector that you're seeing here
  4261. 4:45:14at the end okay so now our pipeline looks like this
  4262. 4:45:19First we had PDF, we converted our PDF to document
  4263. 4:45:23objects, we went through chunking, and then we converted
  4264. 4:45:27our chunks into embedding vectors.
  4265. 4:45:29And in the next step, we look at how to store these vectors
  4266. 4:45:34and use them to retrieve the most relevant chunks
  4267. 4:45:37when a user asks a question.
  4268. 4:45:43Ladies and gentlemen were now cruising smoothly at full speed.
  4269. 4:45:48Every like and every subscription is fuel for this rocket
  4270. 4:45:51and helps us reach more travelers like you.
  4271. 4:45:55Know someone who'd love this journey, share it
  4272. 4:45:57with a friend, and bring a co-pilot on board.
  4273. 4:46:00Have a question, drop it in the comments. The captain is listening.
  4274. 4:46:04Now sit back, relax, and enjoy the rest of your flight.
  4275. 4:46:10you
  4276. 4:46:16Alright, we have now converted our entire knowledge
  4277. 4:46:19base into vectors in a high dimensional space.
  4278. 4:46:22The next step is to take a user's query, convert it into a
  4279. 4:46:26vector, compare it with the vectors in our knowledge base,
  4280. 4:46:30and retrieve the closest matches.
  4281. 4:46:33We have already implemented this approach in our
  4282. 4:46:35previous project, so the idea itself isn't new to you.
  4283. 4:46:38But here's the thing, in real-world drag applications, we generally don't store
  4284. 4:46:44all are vectors in a Python list and we don't manually
  4285. 4:46:48compare the query against every single vector.
  4286. 4:46:51Instead, we use a specialized database designed for this kind of work, this kind
  4287. 4:46:56of task, and it's called a vector database. A vector
  4288. 4:47:00database is designed to store vector embeddings
  4289. 4:47:04and support similarity search over them.
  4290. 4:47:07There are quite a few options available, including Pinecone, Weaviate, Chroma,
  4291. 4:47:13and also PGVector which adds vector search capabilities to PostgreSQL
  4292. 4:47:19and along with many other options that emerge these days.
  4293. 4:47:23And for this project I'm going to use ChromaDB
  4294. 4:47:26because it's really simple and it's a very good option for learning
  4295. 4:47:32and even in real-world projects.
  4296. 4:47:34You can develop real-world projects with Chroma and there is nothing wrong with it.
  4297. 4:47:38But before doing that, before seeing that in code, let me
  4298. 4:47:42explain what we need a vector database in the first place.
  4299. 4:47:46Why can't we simply keep all our vectors in a simple
  4300. 4:47:50Python list and search through them ourselves?
  4301. 4:47:54There are different reasons, or it's better
  4302. 4:47:57to say there are three main considerations.
  4303. 4:48:01The first one is memory usage. Second, persistence.
  4304. 4:48:05and search performance, the third one.
  4305. 4:48:07And let's start with memory.
  4306. 4:48:09In our previous example,
  4307. 4:48:11I'm in project, we stored all our vectors in a python list.
  4308. 4:48:15Those vectors occupy memory while our
  4309. 4:48:18program is running.
  4310. 4:48:19And when the program terminates, that in-memory list disappears.
  4311. 4:48:23When you don't use
  4312. 4:48:24a vector database, memory becomes a serious
  4313. 4:48:28thing, especially when your data set grows.
  4314. 4:48:31With a few hundred
  4315. 4:48:32or a few thousand vectors, a simple Python list is perfectly fine.
  4316. 4:48:37But imagine having 10 million vectors.
  4317. 4:48:39Our all-mini-LM model produces 384 dimensional vectors.
  4318. 4:48:44If we store 10 million vector using 32-bit floating-point numbers, the
  4319. 4:48:50raw vector values alone would take up approximately 14 GB of memory.
  4320. 4:48:55before accounting for metadata indexing and other stuff.
  4321. 4:48:58So, if you have a computer with 16 GB of RAM,
  4322. 4:49:02storing everything in memory could quickly become impractical.
  4323. 4:49:06A vector database can persist data on disk on the other hand,
  4324. 4:49:10which helps us manage datasets that don't fit entirely in memory.
  4325. 4:49:14Now, let's talk about search performance. If you have only a few hundred vectors,
  4326. 4:49:20comparing our query against every vector is usually fast enough.
  4327. 4:49:24But what happens when we have millions or tens of millions of vectors?
  4328. 4:49:28In that case, a brute force search compares a query against every vector.
  4329. 4:49:33And as the data set grows, that can become expensive.
  4330. 4:49:37Vector databases can use specialized indexing and search
  4331. 4:49:41techniques to make retrieval much more efficient.
  4332. 4:49:44So in those scenarios, we don't have a problem anymore.
  4333. 4:49:48And let's talk about the techniques that these databases use.
  4334. 4:49:53to improve performance. One widely used family of techniques is called
  4335. 4:49:57approximate nearest neighbor or ANN.
  4336. 4:50:01And that is, instead of calculating the exact distance to every single vector,
  4337. 4:50:09ANN algorithms search for vectors that are likely to be among the nearest neighbors.
  4338. 4:50:15This often provides a useful trade-off between speed and search accuracy.
  4339. 4:50:20Now that we understood why we are using vector databases
  4340. 4:50:24let's integrate chroma into our application since we are use Langchain
  4341. 4:50:30we'll install its dedicated chroma integration
  4342. 4:50:33so here first I say pipenv install langchain chroma
  4343. 4:50:39okay once it's installed we need to import it
  4344. 4:50:43so I say from langchainchroma import chroma
  4345. 4:50:48Okay, as you can see it is installed and it is a
  4346. 4:50:51class, so we need to create an object of it.
  4347. 4:50:54So I say Chroma and we need specify some parameters.
  4348. 4:50:59So the first thing is collection name.
  4349. 4:51:01A collection name is a named group of records stored in Chroma.
  4350. 4:51:05You can think of it as a way to organize the data
  4351. 4:51:08belonging to a particular application or use case.
  4352. 4:51:11For a project, let's call the collection name medical data, it's a descriptive name.
  4353. 4:51:17So here I say collection name and I set it to medical data
  4354. 4:51:23and the point is that if this collection already exists in the configured database
  4355. 4:51:28Chroma can connect to it otherwise it will create it So it is our collection name
  4356. 4:51:34and then we need to specify which embedding model should be used
  4357. 4:51:38and I'm going to use this model that we constructed here
  4358. 4:51:42So here i say embedding function
  4359. 4:51:45and I set it to the model that we defined on the top.
  4360. 4:51:49By passing this model to Chroma, we are telling
  4361. 4:51:52the vectoristore that we are going to use this
  4362. 4:51:54model and this is where things become more convenient.
  4363. 4:51:58Previously, we extracted the text from every
  4364. 4:52:01chunk and manually called embed documents.
  4365. 4:52:04This way we generated embeddings ourselves.
  4366. 4:52:07Now, Chroma can take care of these steps for us.
  4367. 4:52:12When we add our documents, it will use the configured
  4368. 4:52:16embedding function to generate their vectors.
  4369. 4:52:19And later, when we submit a text query, it can use
  4370. 4:52:22the same embedding model to embed that query.
  4371. 4:52:25And we no longer have to take up these stuff.
  4372. 4:52:29The next parameter is persist directory. Let me set it to period slash
  4373. 4:52:35We want our data to remain available after our Python program terminates.
  4374. 4:52:40So we don't want to store our data on RAM.
  4375. 4:52:43We want a folder here is created and our data is stored in that directory.
  4376. 4:52:49So here we'll specify a directory where Chroma can persist the database.
  4377. 4:52:54So this tells Chroma to use a directory named Chroma inside our program.
  4378. 4:53:00project directory and if this directory doesn't exist,
  4379. 4:53:05Chroma can create it and the database files will be stored there,
  4380. 4:53:09allowing us to reopen the persisted collection when we run the application again.
  4381. 4:53:15Now there is one more important configuration we need to discuss.
  4382. 4:53:20The distance metric.
  4383. 4:53:22When we compare two vectors, we need a way to measure how close they are.
  4384. 4:53:26There are different metrics and we talked about cosine similarity.
  4385. 4:53:31And in this project, we also want to use cosine.
  4386. 4:53:35But the question is that where should we specify this configuration?
  4387. 4:53:40There is a field called collection metadata.
  4388. 4:53:43It takes a dictionary, a key is HNSW space
  4389. 4:53:49and I set this key to cosine
  4390. 4:53:52but pay attention, this cosine you see here is cosine distance not cosine similarity
  4391. 4:53:58we discussed earlier that cosine similarity compares the angle between two vectors
  4392. 4:54:04now Chroma uses cosine distance it is calculated in this way
  4393. 4:54:101 minus cosine similarity. And another common option is a squared L2 distance,
  4394. 4:54:17which measures the sum of the squared differences
  4395. 4:54:20between corresponding vector components.
  4396. 4:54:23And by default, Chroma uses squared L 2 distance for its standard configuration.
  4397. 4:54:29And there is a very important detail here.
  4398. 4:54:32The distance metric is a collection level configuration.
  4399. 4:54:36I mean, if the collection already exists,
  4400. 4:54:39Passing different metadata doesn't automatically
  4401. 4:54:42change the metric used by the existing collection.
  4402. 4:54:45You should decide on the metric when creating a collection.
  4403. 4:54:48So changing it later doesn't work.
  4404. 4:54:52And it's a decision that should be made when you create a collection
  4405. 4:54:56and I store the result inside a variable called vectorStore.
  4406. 4:55:02It's a typical name when creating an instance from Chroma. Okay.
  4407. 4:55:07Now let's go back to our main function.
  4408. 4:55:10Remember these lines we had in the previous session.
  4409. 4:55:13And I told you that it's not the approach that we will take and I remove them.
  4410. 4:55:18And as I mentioned earlier, we are going to use Chroma and
  4411. 4:55:22Chroma is going to take this responsibility to manage vectors,
  4412. 4:55:27storing vectors, using the embedding model and other stuff.
  4413. 4:55:31So I say vectorStore and it provides a method
  4414. 4:55:34called addDocuments and I'm passing my chunks.
  4415. 4:55:38Okay, that's it.
  4416. 4:55:40We are passing our list of LangChain document
  4417. 4:55:43objects and Chroma uses the configured embedding
  4418. 4:55:46model to generate the embeddings and stores the
  4419. 4:55:49corresponding text and metadata alongside them.
  4420. 4:55:53So you're not only storing the text, so you're
  4421. 4:55:57also storing the metadata in your database.
  4422. 4:56:00And as you can see here, we no longer extract text from our document objects.
  4423. 4:56:06We are using our document object.
  4424. 4:56:08We are passing our document-objects and this is the consistency I was talking about.
  4425. 4:56:13Okay, now let's verify that our documents have been stored.
  4426. 4:56:17And for that, Chroma provides a method called get.
  4427. 4:56:22which allows us to retrieve stored records from the collection. So let's do that.
  4428. 4:56:27I say vector store dot get, and this gives me all the
  4429. 4:56:32vectors and I store them in a variable called all docs.
  4430. 4:56:38So here the result is a dictionary-like structure containing
  4431. 4:56:43fields such as the record IDs, document texts, and metadata.
  4432. 4:56:48Yes, every document has an ID and we will see that in detail later.
  4433. 4:56:54And like before, let's print the number of
  4434. 4:56:57stored records and inspect the first document.
  4435. 4:57:00Here I say print, I want the length of the document we have here.
  4436. 4:57:05As I told you, it's a dictionary and one key is documents. These are my documents.
  4437. 4:57:12Now I want to print the first document.
  4438. 4:57:14So I remove the length function and here I
  4439. 4:57:17use bracket notation to access to the first
  4440. 4:57:20element in this list. And now let's run our program.
  4441. 4:57:25Fortunately, we don't have any error here and it
  4442. 4:57:29says that we have 183 vectors in our database.
  4443. 4:57:32And it is a text of the first document.
  4444. 4:57:36But there is an issue here.
  4445. 4:57:38Before we finish, I want to show this problem and
  4446. 4:57:41we will address this problem in the next session.
  4447. 4:57:45So let's run the program one more time.
  4448. 4:57:47Okay, wait a little bit. You will see that this number has twice.
  4449. 4:57:53We had 183 and now we have 366.
  4450. 4:57:59That means that we have added another 183 records. If you run.
  4451. 4:58:04program again the count can increase again okay the
  4452. 4:58:08reason is that remember we are using a persistent
  4453. 4:58:12database okay
  4454. 4:58:14and I told you that each vector has an ID.
  4455. 4:58:18The records from the previous run are still there
  4456. 4:58:21and every time we call add documents
  4457. 4:58:24and everytime we do that without supplying stable IDs
  4458. 4:58:29Langchain generates new IDs.
  4459. 4:58:32So the previous 183 vectors had different IDs
  4460. 4:58:37and these new vectors that we are storing again have new IDs, okay?
  4461. 4:58:43And Chroma, therefore, has no reason to treat those
  4462. 4:58:47new records as updates to the existing ones.
  4463. 4:58:51So, it can store them as separate entries,
  4464. 4:58:54even when their content is exactly identical.
  4465. 4:58:58So, as you can see, this creates duplicate records in
  4466. 4:59:02our collection, and that can affect retrieval quality.
  4467. 4:59:06when we eventually search for chunks similar
  4468. 4:59:09to a query, we might retrieve multiple copies
  4469. 4:59:12of the same information instead of getting a
  4470. 4:59:15more useful selection of relevant content.
  4471. 4:59:18So we have a problem here and we have to solve it.
  4472. 4:59:21And in the next lesson, I will show you how to deal with this scenario.
  4473. 4:59:35So, how do we solve the problem we discussed in the previous lesson?
  4474. 4:59:39One approach that Chroma supports is to provide
  4475. 4:59:43an ID for every document we want to store.
  4476. 4:59:46In this way, when we try to add a document using
  4477. 4:59:49an ID that already exists in the collection,
  4478. 4:59:52Chroma won't create another record with that same ID.
  4479. 4:59:56Instead, adding a document with an existing ID updates the existing record.
  4480. 5:00:02So, in other words,
  4481. 5:00:03to solve this problem, we need to generate
  4482. 5:00:06an ID for every chunk before storing it in
  4483. 5:00:09our vector database.
  4484. 5:00:11So we had 183 chunks and we'll generate an id for each one.
  4485. 5:00:17At the end, we'll have 183 IDs for each of these vectors.
  4486. 5:00:24And this gives us a consistent way to identify
  4487. 5:00:27our chunks and prevent duplicate records every
  4488. 5:00:30time we run the program.
  4489. 5:00:32And now the question is how are we going to generate these IDs?
  4490. 5:00:36For this project, I'm going to use Python's built-in hashlib module
  4491. 5:00:40and that means we don't need to install any additional packages.
  4492. 5:00:44We just need to import this module. So on the top, I say import hashlib.
  4493. 5:00:51Now let's create a function that does that for us. I mean generating IDs.
  4494. 5:00:57So I define a new function
  4495. 5:01:00called generateId
  4496. 5:01:01and it receives our documents and it can generate an ID for each one.
  4497. 5:01:06So at the end, I want to have a list of IDs. So I need to return a list.
  4498. 5:01:13Okay. And this is a situation that using a list comprehension is a great choice.
  4499. 5:01:20So I say for each doc in my docs, I want to do this with that doc.
  4500. 5:01:26First, I wanted to access its content.
  4501. 5:01:29So I said doc page content, but
  4502. 5:01:32I don't want its actual text.
  4503. 5:01:35I want its bytes.
  4504. 5:01:36I want the encode method to access it's bytes because when we want to use a hash,
  4505. 5:01:43algorithm, we don't want its actual text, we want its bytes.
  4506. 5:01:46It's related to cryptography. So here I say encode.
  4507. 5:01:51It's a piece of text and every string offers this method.
  4508. 5:01:54And I give it as input to MD5 algorithm.
  4509. 5:01:59So I say hashlib MD5 and I paste it here.
  4510. 5:02:03So I want to hash this piece of texts. And finally, I want hexadecimal. So
  4511. 5:02:10I say hex digest.
  4512. 5:02:11So at the end, each id is a hexadecimal string
  4513. 5:02:16containing characters from 0 through 9 and a through
  4514. 5:02:20f.
  4515. 5:02:21Now let's use this function in our existing pipeline.
  4516. 5:02:24We need to generate an id for each chunk.
  4517. 5:02:27So here I call this function that I just created and I pass my chunks and I get ids.
  4518. 5:02:34So I store it in a variable called
  4519. 5:02:37ids and now to test it let's print these ids to
  4520. 5:02:41make sure everything is working correctly so
  4521. 5:02:44here i print ids i only print the first item for now i'm going to comment out
  4522. 5:02:52the lines after so
  4523. 5:02:53i run the program
  4524. 5:02:56now you can see that it's working correctly
  4525. 5:02:58and here i have a hexadecimal string so it's
  4526. 5:03:01working and now i have an id for each chunk okay now
  4527. 5:03:06the next step instead of adding our documents
  4528. 5:03:09without ids we'll pass both the chunks and their
  4529. 5:03:13corresponding ids to the vector store so here i
  4530. 5:03:17return these back and here when i'm adding documents i add ids as well so here I say
  4531. 5:03:24IDs and I pass these IDs okay that's it now chroma receives an explicit ID for
  4532. 5:03:32every document we want to store and if you run the ingestion process again and
  4533. 5:03:37the same chunk produces the same ID chroma can identify the existing record
  4534. 5:03:42and it won't store it again anymore let's run our program one more time to
  4535. 5:03:48verify everything is working but before that i'm
  4536. 5:03:51going to delete my database from here so i'm
  4537. 5:03:54going to
  4538. 5:03:54delete it i
  4539. 5:03:56run my program one more time it says we have 183 correct and i will run the
  4540. 5:04:03program one
  4541. 5:04:04more time to verify this number won't increase wait again
  4542. 5:04:10As you can see, the number of stored records remains the same.
  4543. 5:04:14And that means we have successfully prevented repeated ingestion.
  4544. 5:04:18And that's the purpose of this session.
  4545. 5:04:22Oh
  4546. 5:04:27All right, we are getting closer and closer to
  4547. 5:04:30the implementation of a complete RAG system.
  4548. 5:04:34At this point, we have our vector database and we can search through our vector
  4549. 5:04:39database to find the chunks that are most relevant to a user's query.
  4550. 5:04:44So let's implement it.
  4551. 5:04:46First, I'm going to create a conversation loop similar
  4552. 5:04:50to the one we implemented in our chatbot project.
  4553. 5:04:53So let's borrow some code from that project.
  4554. 5:04:57I want this part. Okay. That's enough. Yeah.
  4555. 5:05:01So here I don't want these lines anymore.
  4556. 5:05:04And also this line, it was just for verifying that everything was working correctly.
  4557. 5:05:10And I paste my loop here and this line is irrelevant here.
  4558. 5:05:14And yes, we get
  4559. 5:05:16an input from the user and we have this guard.
  4560. 5:05:20And now we have the user's input and this is going to be our search query.
  4561. 5:05:26now we want to pass this query to our vector store to our
  4562. 5:05:31vector database and retrieve the chunks that are most similar to it so that's a
  4563. 5:05:35process and the question is uh which method
  4564. 5:05:39which method should we use lang chain provides a method called
  4565. 5:05:43similarity search we have two methods similarity search and similarity search
  4566. 5:05:49with score so if you want the score as well we need to use
  4567. 5:05:54similarity set with course and that's a method
  4568. 5:05:57we are going to use in this project so
  4569. 5:05:59I use this method, I say vectorStore similaritySearchWithScore, similaritySearchWithScore
  4570. 5:06:06and I give it
  4571. 5:06:08the user's query and I specify k, I set k to 3
  4572. 5:06:14okay, this tells the vectorStore to return up to 3 matching chunks
  4573. 5:06:19okay, each result will contain two things
  4574. 5:06:23the retrieved document and its associated distance score
  4575. 5:06:28So here we have two pieces of information and remember that because
  4576. 5:06:32later we will use both pieces of information.
  4577. 5:06:35So I store it in a variable called retrieved chunks.
  4578. 5:06:40And there we go.
  4579. 5:06:42And here's something that's really important when
  4580. 5:06:45developing a RAG system, which is observability.
  4581. 5:06:48During development, it's a good idea to inspect
  4582. 5:06:51the documents your retriever returns.
  4583. 5:06:54You can print them to the console. We can also save them to a file.
  4584. 5:06:59Ideally, when you are developing a RAG system, it is better to
  4585. 5:07:03save them somewhere so you can track the history of changes and
  4586. 5:07:07the chunks that your retrievers returns after each change.
  4587. 5:07:11But the point is that we shouldn't develop a RAG system
  4588. 5:07:15without checking what's going in and what's coming out.
  4589. 5:07:19For now, I'm just going to print the retrieve chunks.
  4590. 5:07:22But in a real project, keep in mind that it's
  4591. 5:07:25better to save them in a file or a database,
  4592. 5:07:28or ideally use a tool like a length miss that is for this task.
  4593. 5:07:33I mean, observability and keeping track of what's happening.
  4594. 5:07:37So here we want to iterate over
  4595. 5:07:40these retrieved chunks and print corresponding information.
  4596. 5:07:45So here I'm going to say for chunk.
  4597. 5:07:49in retrieved chunks or it's better to use
  4598. 5:07:53enumerate to access to index as well so i say
  4599. 5:07:59enumerate and in addition to chunk i also
  4600. 5:08:03have the index so the index is the first thing we
  4601. 5:08:07want to print so i said print and here i use a formatted string
  4602. 5:08:12and then i put some hyphens here and then
  4603. 5:08:16i say chunk
  4604. 5:08:18placeholder, my index, and then another four
  4605. 5:08:21hyphens and backslash end to go to the next line.
  4606. 5:08:24Now remember that I told you each result in this list is a tuple.
  4607. 5:08:30The first element is a retrieved document and the second element is the score.
  4608. 5:08:35Let's first print the score, here I add another print
  4609. 5:08:40statement and I say again a formatted string.
  4610. 5:08:44I say score and placeholder and I say chunk. Okay, the second item.
  4611. 5:08:50The second item is the score.
  4612. 5:08:52And then let's format it to three decimal places.
  4613. 5:08:56So I put a colon here and then I say.3f.
  4614. 5:09:01This way we'll have three decimal place. And now let's print the metadata.
  4615. 5:09:06The metadata associated with the document.
  4616. 5:09:09we stored two metadata fields source and page so
  4617. 5:09:14here i add another print statement again i use
  4618. 5:09:19a formatted string and i say metadata say source i
  4619. 5:09:23place a placeholder here and there i say chunk the
  4620. 5:09:28first item in this tuple is our document and each
  4621. 5:09:32document has a metadata and our metadata is a dictionary
  4622. 5:09:37so
  4623. 5:09:37one field was source and another field was page so
  4624. 5:09:43let's duplicate this line and i say page and i
  4625. 5:09:48change this key to page and finally let's print
  4626. 5:09:51the actual text the first 200 characters of the
  4627. 5:09:55chunks text so here i say print chunk again
  4628. 5:09:58first item is our document this time instead
  4629. 5:10:02of metadata i want to access
  4630. 5:10:05to its page content and I just want the first
  4631. 5:10:09200 characters and then I put a backslash
  4632. 5:10:13in here to go to the next line and this
  4633. 5:10:16gives us enough information to inspect the retrieved result
  4634. 5:10:20without printing the entire content of every chunk.
  4635. 5:10:23Now let's put everything together and run our
  4636. 5:10:26program to see whether we have a problem or not.
  4637. 5:10:30It asks me to enter a question. Okay, great.
  4638. 5:10:34Here, let's try a query that should be a good test for semantic search.
  4639. 5:10:39Here I ask, what are the symptoms of high blood pressure?
  4640. 5:10:45and I press enter and as you can see I received three chunks
  4641. 5:10:52here's what makes this a useful test the wording of our query doesn't necessarily
  4642. 5:10:58have to match the wording in the pdf exactly we
  4643. 5:11:01are asking about high blood pressure while the
  4644. 5:11:04document uses the term hypertension and as you
  4645. 5:11:08can see a retriever is retrieving the exact
  4646. 5:11:12document that is related to high blood pressure
  4647. 5:11:15and i guess it is the first disease here
  4648. 5:11:18Yes, as you can see, it's a first page, not the first disease.
  4649. 5:11:23Okay, often called the silent killer.
  4650. 5:11:26And this was the exact document we were looking for.
  4651. 5:11:30It's the most relevant chunk in our document.
  4652. 5:11:32Now, I guess this is a good point to talk a
  4653. 5:11:36little about how we evaluate a RAG retriever.
  4654. 5:11:40When developing a RAG system, we need to measure
  4655. 5:11:44how well it retrieves relevant information.
  4656. 5:11:47and there are different evaluation metrics and frameworks for doing that
  4657. 5:11:52but one useful starting point is to examine the highest rank result
  4658. 5:11:57in our example, the first result is especially important
  4659. 5:12:01because it's a chunk our retriever ranked as the closest match to the query
  4660. 5:12:06if that first result contains the information we are looking for
  4661. 5:12:11that's a positive sign and in formal retrieval evaluation
  4662. 5:12:16This idea is related to metrics such as MRR, which is Mean Reciprocal Rank,
  4663. 5:12:24which evaluates how highly the first relevant
  4664. 5:12:28result is ranked across a set of queries.
  4665. 5:12:32And another useful metric is HitRate, which checks whether
  4666. 5:12:37at least one relevant result appears among the top results.
  4667. 5:12:42For now, we are doing a simple manual inspection
  4668. 5:12:46and we are not dealing with these metrics
  4669. 5:12:48but they are essential in real-world projects.
  4670. 5:12:51And in this example, our retriever has returned the chunk
  4671. 5:12:54that was about hypertension.
  4672. 5:12:57and this was the best chunks that we could retrieve
  4673. 5:13:01so our retriever is working correctly
  4674. 5:13:05Ladies and gentlemen, we're now beginning our
  4675. 5:13:08climb as we prepare to leave planet RAG.
  4676. 5:13:11Our exploration here is almost complete, just a little further.
  4677. 5:13:15And believe us, the rest of this journey is even more exciting than you think.
  4678. 5:13:21Our next stop is Planet Agent.
  4679. 5:13:22There, you'll learn to build robots equipped
  4680. 5:13:25with tools that can get real work done for you.
  4681. 5:13:28Please keep your seat belts fastened. Next stop Planet Agent.
  4682. 5:13:40Okay, in this session we are going to complete the final piece of the puzzle.
  4683. 5:13:44I mean sending our retrieved chunks along with the user's query to an LLM.
  4684. 5:13:48And one of the most important parts of a RAG
  4685. 5:13:51system is the instruction we give the model.
  4686. 5:13:54If you remember, we used the system message to define how our chatbot should behave.
  4687. 5:14:00We have the same prompt here telling a model how should
  4688. 5:14:04produce, how should respond in an augmented way.
  4689. 5:14:08So here I've already prepared a system prompt to save your time.
  4690. 5:14:12And as you can see here, the most important part of this prompt is that
  4691. 5:14:17we are instructing the model to ground its answer in the retrieved context
  4692. 5:14:22instead of relying on its general knowledge.
  4693. 5:14:25So this is a fundamental principle of RAG.
  4694. 5:14:28We want the model use the information we have provided
  4695. 5:14:32rather than simply generating an answer based on what it already knows.
  4696. 5:14:37or making up unsupported assumptions.
  4697. 5:14:40So look at here, I'm saying that answer only using the sources I've provided.
  4698. 5:14:45Don't rely on your own medical knowledge. Don't make things up. Okay.
  4699. 5:14:50And we have also told the model what to do when the
  4700. 5:14:54retrieved documents don't contain enough information.
  4701. 5:14:57It should explicitly say that it doesn't have
  4702. 5:15:00enough information to answer the question.
  4703. 5:15:03And finally, we have asked it to cite its sources.
  4704. 5:15:07This helps us to trace an answer back to the relevant document,
  4705. 5:15:11making it easier to verify where the information came from.
  4706. 5:15:15And now, finally, let's add the LLM to our project. To keep our focus on RAG,
  4707. 5:15:21I'll use the same OpenAI Wrapper used in the previous project. However,
  4708. 5:15:26LangChain has its own integrations for working
  4709. 5:15:29with language models and since we are using
  4710. 5:15:32LangChains the best practice is to use these integrations but as i told you because
  4711. 5:15:38we want to keep our focus on RAG I'm going
  4712. 5:15:41to use what you are familiar with so first
  4713. 5:15:44I'm gonna add.env so on the top I say import.env and here I
  4714. 5:15:50load environment variables I say
  4715. 5:15:54dot env dot load dot env and since i'm going
  4716. 5:15:58to read keys here below dot env i import os
  4717. 5:16:03and I also import OpenAI class.
  4718. 5:16:06So I say from OpenAI import Open AI class.
  4719. 5:16:10Now we need to create
  4720. 5:16:12the client. So let me borrow some code.
  4721. 5:16:16I guess we had another name.
  4722. 5:16:18Yeah, we called it LLM in
  4723. 5:16:21our chatbot project.
  4724. 5:16:23So I paste it and I store it in a variable called client. Okay, everything is right.
  4725. 5:16:29Now, let's create a function called LLM.
  4726. 5:16:33This function is going to receive two pieces of information, query and context.
  4727. 5:16:39Okay.
  4728. 5:16:40The query is a question that the user has asked and the context contains
  4729. 5:16:45the relevant chunks that our retriever fetched from our vector database.
  4730. 5:16:50So I define a new function, I call it LLM.
  4731. 5:16:54It receives two information,
  4732. 5:16:57query and context. Here we go.
  4733. 5:17:00First, I'm going to put our system prompt here.
  4734. 5:17:04Here's our system prompt.
  4735. 5:17:06And as you can see here, we are injecting our context inside our system prompt.
  4736. 5:17:13So this is where our context goes.
  4737. 5:17:16But besides our system prompts, we also need a user message because we are going
  4738. 5:17:21to ask something to have that. We need a list of messages.
  4739. 5:17:25The first message is a system message. And the second message is the user message.
  4740. 5:17:29So I create a list called messages. It has two messages.
  4741. 5:17:34The first message is a system message. So I set role to system.
  4742. 5:17:39And the second one is content. Okay. I set it to our system prompt.
  4743. 5:17:45I duplicate this line. I change it to user.
  4744. 5:17:48And my content is going to be the query that this function receives.
  4745. 5:17:54Let's send the query to the LLM. So I go here. I search for LL M dot chat.
  4746. 5:18:00I need this part.
  4747. 5:18:02Copy it. I paste it here.
  4748. 5:18:04I change it to client and I'm going to use the
  4749. 5:18:08same model and here instead of this, I'm
  4750. 5:18:11going to the messages list that I created on the top
  4751. 5:18:16and I set temperature to 0.7.
  4752. 5:18:20And here we are sending our request and we get the response.
  4753. 5:18:25So I store it in a variable called response.
  4754. 5:18:29Now we will return the response to the user.
  4755. 5:18:32So to the main function actually, because we're going to call it from the main
  4756. 5:18:36function.
  4757. 5:18:37So I say return response.choices k0 the syntax that we had.
  4758. 5:18:42Message. It's a message and I want its content.
  4759. 5:18:46that's our llm function it takes the user's question and the retrieved context
  4760. 5:18:52sends both to the model and returns the answer now let's use this function in our
  4761. 5:18:58main function okay here inside our main function but there is an important detail
  4762. 5:19:03here the
  4763. 5:19:04context should be a simple text
  4764. 5:19:08not a list of LangChain document objects here as you can see
  4765. 5:19:12We have a list of document objects with their scores, but we need to parse them.
  4766. 5:19:16We need to combine all the information into a simple string.
  4767. 5:19:20So after retrieving these chunks, let me remove these lines.
  4768. 5:19:25It was for observability in the previous session.
  4769. 5:19:28So first I create an empty list called parts.
  4770. 5:19:31Now I'm going to iterate over these chunks.
  4771. 5:19:35take out the information that I need and import in a string.
  4772. 5:19:39So we will eventually have three strings.
  4773. 5:19:42And finally after iterating over this retrieve
  4774. 5:19:45chunks, we can concatenate these three strings
  4775. 5:19:48that we will have in this list.
  4776. 5:19:50So here, like before, we are going to iterate over
  4777. 5:19:54this retrieved chunks and I will use enumerate.
  4778. 5:19:57to have indexes as well.
  4779. 5:19:59So I use enumerate and we have an index and each chunk.
  4780. 5:20:04Okay, I use
  4781. 5:20:05pass here for now.
  4782. 5:20:08And it is a tuple here, we can unpack it instead of using indexes.
  4783. 5:20:13It's a better syntax.
  4784. 5:20:14So I unpack its two elements into two variables called doc and score.
  4785. 5:20:21Okay, for each chunk I want to extract its chunk number, page,
  4786. 5:20:26source and the content itself.
  4787. 5:20:28So I use a formatted string and I say source.
  4788. 5:20:33I place a placeholder here. I say I plus one because our indexes start from zero.
  4789. 5:20:40Then I have the page.
  4790. 5:20:43I say page and I want to have a placeholder here as well
  4791. 5:20:47so I say doc.metadata it is located in our metadata and
  4792. 5:20:52then since it is a dictionary I use the get method
  4793. 5:20:56so I save page and the second argument a question mark so if the page metadata
  4794. 5:21:04isn't available we'll use a questionmark instead okay and after another pipe
  4795. 5:21:11again and I say retrieval score and as you remember
  4796. 5:21:16it's not similarity score it's cosine distance
  4797. 5:21:20so because of that I'm saying retrieval a
  4798. 5:21:23score not cosine score so here again braces
  4799. 5:21:27and I said score and three decimal places here
  4800. 5:21:31And finally we want to have our actual content.
  4801. 5:21:34So I put backslash in here and then my actual page content.
  4802. 5:21:39So I say doc page content
  4803. 5:21:42Okay, that's pretty much it.
  4804. 5:21:44The only thing left is to concatenate all these that we have.
  4805. 5:21:48So we can use the join method that strings provide.
  4806. 5:21:52So I call join and I pass it our list here.
  4807. 5:21:57And as a separator, I use double backslash and
  4808. 5:22:00a couple of hyphens and double back slash
  4809. 5:22:03and again.
  4810. 5:22:04Now we are ready for the final step, which is calling
  4811. 5:22:08our LLM function with the original user query.
  4812. 5:22:11and the context just created. So I call it, I say,
  4813. 5:22:17LLM and I provide our user input and the contexts as well.
  4814. 5:22:22I forgot to store it in a variable called context. Okay, it's our context.
  4815. 5:22:28Okay, everything is fine and let's print the result.
  4816. 5:22:32Okay, let's run our program.
  4817. 5:22:34oh what happened oops here the problem is that we are loading environment variables
  4818. 5:22:40after we want to use them so here was my bad
  4819. 5:22:44okay let's run our program one more time
  4820. 5:22:47I ask what
  4821. 5:22:49are the symptoms of high blood pressure
  4822. 5:22:54oh
  4823. 5:22:55it says I don't have enough information so it's weird
  4824. 5:23:00let's keep track what's happening so here we have the context
  4825. 5:23:05first let's print context
  4826. 5:23:08It's better if I investigate the problem myself and I will pause the video and I
  4827. 5:23:14will get back to you.
  4828. 5:23:16Okay, hello again. I quickly found the root of the problem.
  4829. 5:23:20And the problem is that here I forgot to append this string
  4830. 5:23:24that I generated to this list that I had on the top.
  4831. 5:23:28So I say parts that append and that's pretty normal.
  4832. 5:23:35Again we try to run our program. What are the symptoms of high blood pressure?
  4833. 5:23:43okay as you can see now it answers our question
  4834. 5:23:46this is a part of our print statement and the
  4835. 5:23:50answer starts from here high blood pressure can
  4836. 5:23:53cause symptoms such as blah blah blah and
  4837. 5:23:56the interesting part is that it's mentioning our
  4838. 5:23:59sources it says i'm saying that according to
  4839. 5:24:03source one and if you check let me bring our pdf
  4840. 5:24:08As you can see, this response is augmented based on this information.
  4841. 5:24:12You can see headaches, dizziness, and you can see these keywords here.
  4842. 5:24:17You can see dizziness, headaches. So everything is working.
  4843. 5:24:22Our application retrieves relevant chunks from the
  4844. 5:24:25vector database, combines them into a context,
  4845. 5:24:28and sends that context along with the user's question to the LLM.
  4846. 5:24:33Here we have an augmented answer as a result.
  4847. 5:24:36successfully constructed all pieces of our RAG system.
  4848. 5:24:41But before finishing this session, there is one important thing I want to mention.
  4849. 5:24:46As you can see with this implementation, let me bring our main function.
  4850. 5:24:50With this implementation every time we start the program,
  4851. 5:24:55we repeat the entire document processing pipeline.
  4852. 5:24:59We read the PDF, we convert its pages into documents, generate IDs,
  4853. 5:25:04split the documents into chunks and send those chunks
  4854. 5:25:09to the vector store
  4855. 5:25:10even though we have added
  4856. 5:25:12safeguard to avoid creating duplicate records we are still repeating a lot of
  4857. 5:25:17unnecessary work and that's not a good design for a
  4858. 5:25:20real world application a better approach would be to
  4859. 5:25:24separate to separate document ingestion from querying for example
  4860. 5:25:30We could create one script responsible for processing the PDF,
  4861. 5:25:34generating chunk and IDs and these stuff,
  4862. 5:25:37and another query for our main application. Welcome back aboard Neon Learned.
  4863. 5:25:42Planet Rag has conquered four worlds down, six to go.
  4864. 5:25:48Fasten your seat belts. Next stop, Planet Agent.
  4865. 5:25:51Here, you'll build an AI agent that uses real tools,
  4866. 5:25:55checking the weather and searching the web on its own.
  4867. 5:25:59Your next world awaits. Enjoy the journey.
  4868. 5:26:09Alright, from this point on, we are going to
  4869. 5:26:11start talking about AI agents, which are
  4870. 5:26:14a very important topic in AI engineering.
  4871. 5:26:17I'm sure you have been hearing a lot about AI
  4872. 5:26:20Agents and that proves that that's a burgeoning
  4873. 5:26:23area in this field.
  4874. 5:26:25We are going explore what AI Agentes are, what
  4875. 5:26:28they do, what problems they can solve, and of
  4876. 5:26:31course just like in previous sessions, we will
  4877. 5:26:34also build practical projects to put what
  4878. 5:26:37we have learned into practice. So let's start with a fundamental question.
  4879. 5:26:42What exactly are AI agents and what problems do they solve?
  4880. 5:26:47The answer is that up to this point, we have built a chatbot.
  4881. 5:26:51At the beginning of this course, we created a simple chatbot
  4882. 5:26:55that took an input from the user and generated a response.
  4883. 5:26:59pretty straightforward, right?
  4884. 5:27:01But as we continued, we discovered an important problem.
  4885. 5:27:06We saw that for certain tasks, the model doesn't have enough information, doesn't
  4886. 5:27:11have enough relevant information more accurately to provide a good answer.
  4887. 5:27:16And how did we address that? We introduced RAG or Retrieval Augmented Generation.
  4888. 5:27:23It allowed our application to retrieve relevant information from external documents
  4889. 5:27:29and provide that information to the LLM as context
  4890. 5:27:32so LLm could generate better answers.
  4891. 5:27:35But now we have another problem or it's better to say we have another limitation.
  4892. 5:27:40Our application is still primarily designed to take a query and generate a response.
  4893. 5:27:46For many tasks, we need something more. Sometimes we need the model to take action.
  4894. 5:27:52For example, we might want our application to retrieve real-time information, to
  4895. 5:27:58perform calculations, to query a database, to interact with an external service.
  4896. 5:28:04In these situations, simply generating an answer isn't enough for us.
  4897. 5:28:09And our application needs a way to perform actions.
  4898. 5:28:13And this is where AI agents kick in.
  4899. 5:28:16At a high level, an AI agent uses an LLM to decide what action to take.
  4900. 5:28:22question is how do we make this possible?
  4901. 5:28:25We provide the LLM with a set of available tools,
  4902. 5:28:28along with descriptions of what those tools do and what inputs they require.
  4903. 5:28:34The model can then decide
  4904. 5:28:36whether one of those tools would help answer the user's question or not.
  4905. 5:28:41And when a tool is needed,
  4906. 5:28:43the model can request that the application call
  4907. 5:28:46it, or code executes the tool, gets the result,
  4908. 5:28:50and makes that result available to the model.
  4909. 5:28:53That's the basic idea behind tool use in AI agents.
  4910. 5:28:57Now let's look at a practical example because this
  4911. 5:29:01concept will become much clearer once we walk
  4912. 5:29:04through the process and see an example. Let's
  4913. 5:29:07start with the limitation we have already discussed.
  4914. 5:29:10The knowledge cutoff. A model is trained on data available up to a certain point,
  4915. 5:29:16and its built-in knowledge might not include recent events or real-time information.
  4916. 5:29:21For example, suppose we ask an LLM, what's the weather like in Paris today?
  4917. 5:29:26The model can't answer the question using its training knowledge.
  4918. 5:29:30It needs access to current weather data.
  4919. 5:29:33Now you might be asking, but ChatGPT can answer questions about today's weather.
  4920. 5:29:38That's true.
  4921. 5:29:39And the reason is that ChatGPT connects to external
  4922. 5:29:42services and it is provided with up-to-date information.
  4923. 5:29:46But if you simply run a language model locally
  4924. 5:29:50without connecting it to proper tools, it won't
  4925. 5:29:54automatically have access to real-time weather. So it cannot answer your question.
  4926. 5:29:59It might say something like, I don't have enough information.
  4927. 5:30:03I don't have up-to-date information.
  4928. 5:30:05As I told you, we can address this limitation
  4929. 5:30:08by giving our application access to a weather
  4930. 5:30:11tool. And this brings us to an important point. which is a misconception.
  4931. 5:30:17There is a common misconception about how tool calling works.
  4932. 5:30:21Some people think when we provide a tool to an LLM,
  4933. 5:30:25the model itself executes the tool. That's not what happens.
  4934. 5:30:30A language model only generates token and when it's configured to use tools,
  4935. 5:30:36it can generate a structured tool call request that specifies
  4936. 5:30:41which tool it wants and what arguments to pass to it.
  4937. 5:30:45So it is your application that is responsible for parsing that tool request
  4938. 5:30:51and it's responsible for executing the actual function or external operation.
  4939. 5:30:56Let's make this more concrete with an example.
  4940. 5:31:00Imagine we have an LLM and we have a tool or in other words a function.
  4941. 5:31:05they are the same.
  4942. 5:31:06And this function is responsible for retrieving
  4943. 5:31:09the current weather for a given city.
  4944. 5:31:12So we have a getWeather function.
  4945. 5:31:14It takes the name of a city and it returns the
  4946. 5:31:17weather conditions in that particular city.
  4947. 5:31:20And when we use the LLM, we provide the user's
  4948. 5:31:24query along with a description of this function.
  4949. 5:31:28That description tells the model that a tool called getWeather exists.
  4950. 5:31:33It accepts a city name as its input and it
  4951. 5:31:36returns weather information for that city.
  4952. 5:31:39And notice the distinction here.
  4953. 5:31:41We are providing the model with the tools name,
  4954. 5:31:44description and its input requirements.
  4955. 5:31:47And we are not providing the implementation at all. That's the whole point.
  4956. 5:31:52The LLM doesn't have access to implementation and it
  4957. 5:31:55doesn't make sense to provide it because it's useless.
  4958. 5:31:58It's our own responsibility to call that tool.
  4959. 5:32:01Now let's consider two different user queries.
  4960. 5:32:04The first query can be what is the capital of France?
  4961. 5:32:07Okay, the model already has enough information about the capital of countries.
  4962. 5:32:13It doesn't need current weather information, doesn't need this tool.
  4963. 5:32:16So it answers the capital of France is Paris and we have no tool call here.
  4964. 5:32:22So here tool call is not necessary.
  4965. 5:32:26For another query, we can say, what's the weather like in Paris today?
  4966. 5:32:31This time, the model recognizes that it needs current weather information.
  4967. 5:32:36And instead of immediately generating a final answer, it requests a
  4968. 5:32:41call to the get weather tool and supplies Paris as the argument.
  4969. 5:32:45Now, it's our application's responsibility to handle that request.
  4970. 5:32:50So, because it's really important, let's walk through it step by step.
  4971. 5:32:55So first, the user asks about the current weather in Paris.
  4972. 5:32:59We send the query to the LLM and provide the definition of Get Weather Tool.
  4973. 5:33:04Okay, the model determines that it needs current weather information in a city
  4974. 5:33:10and returns a request to call Get Weather with Paris as the input.
  4975. 5:33:16Now our code receives the tool called Request, identifies the requested function,
  4976. 5:33:23passes Paris to it and executes the function.
  4977. 5:33:27And the weather service might return something like 22 degrees Celsius.
  4978. 5:33:32Okay, now that we have the weather resolved,
  4979. 5:33:34we send it back to the LLM as part of the conversation history.
  4980. 5:33:39Now the model can finally use the returned weather information
  4981. 5:33:43to generate a response such as
  4982. 5:33:45the current temperature in Paris is 22 degrees celsius.
  4983. 5:33:49Is there anything else you'd like to note
  4984. 5:33:52And that's it.
  4985. 5:33:53We have gone from a user question to an action and then a final answer.
  4986. 5:33:58If you look closely, you will notice that this process requires two LLM calls.
  4987. 5:34:05The first call allows the model to decide whether it needs a tool.
  4988. 5:34:10If so, it requests the tool call.
  4989. 5:34:13Our application then executes that tool and obtains the result
  4990. 5:34:18and the second LLM call provides a model with that result, with that information
  4991. 5:34:23so it can generate a final response.
  4992. 5:34:27And of course, it's obvious that not every request requires exactly two calls.
  4993. 5:34:32If the model doesn't need a tool, it might answer in a single call.
  4994. 5:34:37If it needs multiple tools, it may require several
  4995. 5:34:40tool calls and additional LLM interactions.
  4996. 5:34:43And this is one reason agent-based applications
  4997. 5:34:47can be more expensive than simple chatbots.
  4998. 5:34:51They may require multiple model calls, external
  4999. 5:34:55API requests, and additional processing.
  5000. 5:34:59And at this point you might be wondering if our
  5001. 5:35:02application has to manage all these steps?
  5002. 5:35:05Doesn't the code become complicated?
  5003. 5:35:08Yes, exactly.
  5004. 5:35:09We need to handle tool call requests, identify the
  5005. 5:35:12correct function, validate and pass its arguments,
  5006. 5:35:16execute it, return the result to the model again,
  5007. 5:35:20and determine whether the model is ready to provide final answer or not.
  5008. 5:35:25Besides that we also need to consider
  5009. 5:35:28error handling, unexpected tool inputs, API failures, and other edge cases.
  5010. 5:35:34And this is exactly where frameworks such as LangChain and LangGraph become useful.
  5011. 5:35:40LangChains provides abstractions and integrations
  5012. 5:35:45that make it easier to connect LLMs with tools
  5013. 5:35:48and build tool calling applications and so on.
  5014. 5:35:51and LangGraph helps us build more explicit workflows,
  5015. 5:35:55agent systems, support more complex execution patterns.
  5016. 5:36:00Oh
  5017. 5:36:06I know the concept we discussed in the previous
  5018. 5:36:08lesson might seem a little complicated at first,
  5019. 5:36:11but don't worry.
  5020. 5:36:12With this hands-on project, we will put all those concepts into practice,
  5021. 5:36:17and you will develop a much deeper understanding of how everything works.
  5022. 5:36:22So don't worried. We will take it step by step, and you'll master it.
  5023. 5:36:26So first, I'm going to create a new file called agent.py, and...
  5024. 5:36:33In this file, as usual, first we have our main function.
  5025. 5:36:37okay and our usual entry point which is if double
  5026. 5:36:41underscore name double underscore is equal to
  5027. 5:36:45double underscore main double underscore in that
  5028. 5:36:48case i want my main function to be executed okay
  5029. 5:36:52that's it the next thing we need is an llm so far
  5030. 5:36:55we have been using the open ai python wrapper to
  5031. 5:36:59communicate with our language model now in this
  5032. 5:37:02project we want to switch to langchains wrapper
  5033. 5:37:06So we are going to use Langchain's wrapper for Groq.
  5034. 5:37:10And this gives us the opportunity to learn another widely used integration.
  5035. 5:37:14Besides, it keeps our implementation more consistent
  5036. 5:37:17because we are currently using Langchain.
  5037. 5:37:20So to get started, we need to install its package, which is Langchain Groq
  5038. 5:37:26So here, pipenv install Langchain Groq.
  5039. 5:37:31Now let's
  5040. 5:37:32load our environment variables just as we did in previous project.
  5041. 5:37:36So I import.env first and then I use it to load my environment variables.
  5042. 5:37:43Next, we need to import ChatGroq from the library that is being installed.
  5043. 5:37:49Okay, now we can import it.
  5044. 5:37:51So here I say from LangChain, from LangChainGroq import ChatGroq.
  5045. 5:37:57Okay, that's the class we need.
  5046. 5:37:59One thing to keep in mind is that when we switch wrappers,
  5047. 5:38:02the parameter names and the way responses are returned can change.
  5048. 5:38:07The underlying concepts remain the same,
  5049. 5:38:09but the interface isn't necessarily identical.
  5050. 5:38:13For example, we can still configure parameters such as temperature
  5051. 5:38:17and the maximum number of generated tokens,
  5052. 5:38:20although the exact parameter names depend on the
  5053. 5:38:23integration and the model provider and now let's
  5054. 5:38:26create an instance of this class so here i say
  5055. 5:38:30chat Groq and i store it in a variable called
  5056. 5:38:33llm it's going to be our ll okay first we need to specify
  5057. 5:38:38which model we are going to use so i set model to
  5058. 5:38:42let's see here you know in this project i copied the model name from here which was
  5059. 5:38:48Open.
  5060. 5:38:50yeah open AI GPT OSS okay I want to use this
  5061. 5:38:53model okay this is our model now let's set the
  5062. 5:38:57temperature okay I set the temperature to 0.5
  5063. 5:39:01okay and that's our basic LLM configuration
  5064. 5:39:05You might remember that when we used the OpenAI
  5065. 5:39:08Wrapper, we had to specify a custom base URL.
  5066. 5:39:11That's because we were using the Open AI client to
  5067. 5:39:14communicate with Groq's OpenAI-Compatible API.
  5068. 5:39:17However, here we are using the dedicated LangChain integration for Groq.
  5069. 5:39:22So we don't need to configure that base URL ourselves and
  5070. 5:39:27it is already configured by internal implementation.
  5071. 5:39:31However, make sure that the API key is available in your.env file.
  5072. 5:39:36You need to have this API key here with exactly this name.
  5073. 5:39:40And this integration can read this environment variable automatically.
  5074. 5:39:44So we don't have to pass the API explicitly in our Python code.
  5075. 5:39:48Now that our LLM is configured, let's see how we can use it.
  5076. 5:39:52With the OpenAI client, we previously used a
  5077. 5:39:56call like client.chat completions create.
  5078. 5:40:00Now with LangChain we can use a simpler interface
  5079. 5:40:03we only need to call the invoke method
  5080. 5:40:06so here in our main function we can call this method
  5081. 5:40:11and pass my query for example i say what is the capital of France okay
  5082. 5:40:17and i get the response and then i print it
  5083. 5:40:22Okay, this is a message, but I only want its content, not its role.
  5084. 5:40:26Now let's run our code.
  5085. 5:40:29okay as you can see it works correctly you see
  5086. 5:40:32how straightforward this is we call llm.invoke
  5087. 5:40:35passing our question and receive a response that's
  5088. 5:40:38it the return the value you see here is just
  5089. 5:40:41a langchain message object as you see langchain gives
  5090. 5:40:45us a relatively concise interface and we don't have
  5091. 5:40:49to manually construct an open ai chat completions
  5092. 5:40:53request like before okay before we move on
  5093. 5:40:56let's take a closer look at the response object
  5094. 5:40:59this response we have here like open ai response
  5095. 5:41:02contains a lot of metadata things like token usage
  5096. 5:41:06and other kinds of information for example
  5097. 5:41:10here i say response metadata
  5098. 5:41:13Okay, as you can see here, it can contain information such as
  5099. 5:41:17the model identifier, the name of the model we are using, and
  5100. 5:41:21a lot of other information that might be useful for you.
  5101. 5:41:25As another example, let's print usage metadata.
  5102. 5:41:30So if I print it out, you see that it says our
  5103. 5:41:34input tokens was 78, output tokens, total tokens,
  5104. 5:41:38and it gives you information about your usage.
  5105. 5:41:42And finally, let's look at another thing that is pretty related to our topic.
  5106. 5:41:48I mean, AI agents. So here I print
  5107. 5:41:51tool calls and if I printed
  5108. 5:41:54you see that it's an empty list because our LLM doesn't want us to execute any tool
  5109. 5:42:00and we didn't give it any tool as well.
  5110. 5:42:04Oh
  5111. 5:42:11If you remember, when we used OpenAI Wrapper, each
  5112. 5:42:14message had two main fields, role and content.
  5113. 5:42:18The role could be user, assistant or system, depending on the type of message.
  5114. 5:42:23We have the same concept here. In LangChain, we have three classes.
  5115. 5:42:27These classes are human message, AI message and system message.
  5116. 5:42:31And from their name, you can probably guess what each one corresponds to.
  5117. 5:42:35Human message is equivalent to user role.
  5118. 5:42:38AI message corresponds to Assistant role and System message corresponds
  5119. 5:42:43to the system role and to use these three classes you can import them from
  5120. 5:42:48LangChain course. So on the top I say from
  5121. 5:42:52LangChain, core, messages, okay, import,
  5122. 5:42:56system message, human message and AI message.
  5123. 5:43:00And we have also tool message here. It is related to when we have a tool call.
  5124. 5:43:05So whenever we need one of them, we can use this class to
  5125. 5:43:10create that type of message that we want.
  5126. 5:43:13So simply we create an instance, for example, for when we want to have
  5127. 5:43:18have a human message or a message that its role is user.
  5128. 5:43:21I use human message and I pass the content that I want to its constructor.
  5129. 5:43:27So here, for example, I say what is the capital of France.
  5130. 5:43:33and this represents the same kind of message we
  5131. 5:43:36had with open ai format something like this that
  5132. 5:43:39we had before we had role the role is user and
  5133. 5:43:43we have content which is this message so
  5134. 5:43:47i'm trying to say you these two are the exact
  5135. 5:43:50same concepts with only different syntaxes okay
  5136. 5:43:53and this was about these three classes okay
  5137. 5:43:59now to build an AI agent we will start by creating a
  5138. 5:44:02regular chatbot just like the one we built at the
  5139. 5:44:05beginning of this course in our first project then
  5140. 5:44:08we will extend it by equipping it with a set of
  5141. 5:44:12tools okay that's a basic idea we start with a chatbot
  5142. 5:44:15that can communicate with an LLM and then we
  5143. 5:44:18will give it the ability to request actions through
  5144. 5:44:22the tools that we provide so let's implement it in
  5145. 5:44:25And I want it to be object oriented. So first I create a class called Agent.
  5146. 5:44:31And this class has a constructor.
  5147. 5:44:36and first we will initialize our LLM using the
  5148. 5:44:39wrapper that I showed you in the previous
  5149. 5:44:42session so here I'm going to take it cut it and paste
  5150. 5:44:46it here
  5151. 5:44:48okay and i add self keyword at the beginning of
  5152. 5:44:51it then we want to have our conversation history
  5153. 5:44:55so i say self messages it is a list and as you
  5154. 5:44:58remember the first message should be a system message
  5155. 5:45:02so i use the system message class here and i
  5156. 5:45:06pass something like this give short answers
  5157. 5:45:10next we want have a method a method called chat
  5158. 5:45:13loop and it is basically our conversation loop
  5159. 5:45:17So here I add this method. I say def chat loop.
  5160. 5:45:22our conversation loop is going to sit here so
  5161. 5:45:26let me borrow some code from maybe here
  5162. 5:45:29okay i copy these lines and i paste them here
  5163. 5:45:36after receiving a message from our user we are
  5164. 5:45:38supposed to add it to our conversation history
  5165. 5:45:41so this is my conversation history and i'm going to append
  5166. 5:45:46this message i mean what the user has entered to it
  5167. 5:45:50so i say human message
  5168. 5:45:52so you see that instead of using the previous syntax for creating messages now
  5169. 5:45:57we have this syntax which is nicer in my opinion and i put users input here then
  5170. 5:46:04We are going to have an LLM call and I told
  5171. 5:46:07you that you should use the invoke method.
  5172. 5:46:10So here I say self.llm invoke and
  5173. 5:46:14here I'll pass our conversation history, which
  5174. 5:46:19is our messages list and
  5175. 5:46:22I'll get a response and I store it in a variable called response.
  5176. 5:46:27And then I'll add the response to my conversation history.
  5177. 5:46:30So I say self.messages or conversation history append this message.
  5178. 5:46:36And in the previous lesson, I showed you that this is basically an AI message. Okay.
  5179. 5:46:42And now let me print its content.
  5180. 5:46:46Say response.content.
  5181. 5:46:48Okay, let me make them more organized.
  5182. 5:46:51Okay, that's nicer and I don't need this line anymore.
  5183. 5:46:55Okay, now we have to create an instance of it. So, store it in a variable.
  5184. 5:47:02and then I'll call its ChatLoop method.
  5185. 5:47:05Now let's run our program and test it with a few questions.
  5186. 5:47:10So here I say, let's
  5187. 5:47:12start with the greeting.
  5188. 5:47:14And I asked what is the capital of France?
  5189. 5:47:19Who is its president?
  5190. 5:47:22to test whether it has memory or not. So it has a memory. Okay, great.
  5191. 5:47:26We have a chatbot that can answer questions
  5192. 5:47:29and maintain its conversation history across
  5193. 5:47:31multiple turns. At this point, we still have a regular chatbot.
  5194. 5:47:35We haven't given it any tool yet. And that's what we will do next.
  5195. 5:47:39In the next session, we will equip it with the tools that we wanted to have.
  5196. 5:47:44And yes, in the next section, I will show you that.
  5197. 5:47:48Ladies and gentlemen, this is a message from your captain.
  5198. 5:47:52You may have noticed a few meteors passing close to our ship.
  5199. 5:47:56The fear of forgetting what you've learned. The fear never becoming an AI engineer.
  5200. 5:48:01The feeling that all your effort is for nothing.
  5201. 5:48:04If they've scared you, that's completely normal.
  5202. 5:48:07Everyone who has traveled this path to becoming an AI engineer has seen them too.
  5203. 5:48:13The only real danger out here is turning off your engines.
  5204. 5:48:17So just keep going at full speed. These meteors can't catch you.
  5205. 5:48:31Okay, from this point on, we are going to start implementing our agent.
  5206. 5:48:35But before we do that, I'd like to spend a little time explaining the difference
  5207. 5:48:40between LangChain and LangGraph because it
  5208. 5:48:43is relevant to what we are about to build.
  5209. 5:48:47You've already seen a little bit of LangChains, but you might not be familiar
  5210. 5:48:51with LangGraph yet. If you search online, you might get the impression that
  5211. 5:48:57whenever you want to build an AI agent, you should use LangGraph
  5212. 5:49:01but that is not necessarily true
  5213. 5:49:04just because you are building an agent doesn't
  5214. 5:49:06mean you automatically need LangGraph
  5215. 5:49:08why? because LangGraph can introduce additional complexity
  5216. 5:49:13and if your task doesn't require that level of control, that level complexity
  5217. 5:49:20adding complexity might not make sense
  5218. 5:49:23So if all you need is an LLM call with a few tools,
  5219. 5:49:27LangChain already provides a convenient way to do that. You don't need LangGraph.
  5220. 5:49:32So in fact, LangChains has a feature called Create
  5221. 5:49:36Agent and this Create Agent uses LangGraph
  5222. 5:49:39behind the scenes.
  5223. 5:49:41So it manages all that complexity.
  5224. 5:49:43You can use this higher level interface
  5225. 5:49:46without having to work directly with LangGraph
  5226. 5:49:49and it makes your application less complex.
  5227. 5:49:53This way you can build an agent without explicitly
  5228. 5:49:56importing LangGraph or managing its lower level
  5229. 5:49:59details yourself which is really good and convenient.
  5230. 5:50:02So let me summarize. LangChains create agent gives you a convenient
  5231. 5:50:07higher level interface for building tool using agents. While LangGraph
  5232. 5:50:12gives you more direct control over the agent's control flow and execution.
  5233. 5:50:16So in this project, we will use create agent for creating our agent.
  5234. 5:50:20And later in another project, I will show you how to use LangGraph.
  5235. 5:50:25So let's get started. First, we need to import create agent from LangChainAgent.
  5236. 5:50:31So here on the top, I say from LangChainAgent.
  5237. 5:50:36Import, create agent
  5238. 5:50:39Okay, now the question is what exactly are we going to build?
  5239. 5:50:43We are going to create an agent with a weather tool.
  5240. 5:50:46I don't want to make it over complicated.
  5241. 5:50:48I want to focus on the concept of agent itself.
  5242. 5:50:51So I don't want to really connect my application to a weather service.
  5243. 5:50:56So I will create a simple function that accepts a city name
  5244. 5:51:00and it always returns the same weather information.
  5245. 5:51:03It always says the weather is sunny and the temperature is 22 degrees Celsius.
  5246. 5:51:09And later we can replace this function with a real weather API.
  5247. 5:51:13And keep in mind that whenever I say tool, I'm referring to a simple function.
  5248. 5:51:18Okay, so here I'm going to define this tool,
  5249. 5:51:21which is a function and I call it get weather. It takes a city name.
  5250. 5:51:26It is a string and then it returns a dictionary. Here I say dict.
  5251. 5:51:32Oh, S, T, R.
  5252. 5:51:34and then we'll specify a description and here I use triple quotation
  5253. 5:51:39and I say get the current weather for a city
  5254. 5:51:44use imperative sentences when you are documenting like this
  5255. 5:51:48and then I specify the arguments
  5256. 5:51:51the only argument that we have is city which is the name of the city
  5257. 5:51:57then I'll return a dictionary containing a city
  5258. 5:52:01which is the city that the user has asked for and another key called weather
  5259. 5:52:07and I set it to sunny 22 degrees Celsius
  5260. 5:52:11the most important detail here is the description you are seeing here
  5261. 5:52:15in Python it is called Dockstring and it is a standard way to document a function
  5262. 5:52:21so other users, other developers can understand
  5263. 5:52:25what it does and how they should use it.
  5264. 5:52:28But when we are working with AI agents, that
  5265. 5:52:32description has another important purpose.
  5266. 5:52:36Langchain can use this description along with the function name and its input schema
  5267. 5:52:42to provide the LLM with the information it needs.
  5268. 5:52:46and with this information, LLM can figure out when it should call a tool.
  5269. 5:52:52So you should keep in mind that the model uses this information
  5270. 5:52:56and it influences its decision,
  5271. 5:52:59its decision whether the tool is relevant to the user's request or not.
  5272. 5:53:03That's really important and it's not just for documenting your functions necessary.
  5273. 5:53:09For example, if the user asks about the weather in Paris,
  5274. 5:53:12This description helps the model understand that get weather is needed here.
  5275. 5:53:17And if the description is vague or misleading, the
  5276. 5:53:21model might fail to call the appropriate tool.
  5277. 5:53:24So you should be really careful.
  5278. 5:53:26And before we continue, there is another important point I want to mention.
  5279. 5:53:31When building agents, the Model you choose matters. A small model
  5280. 5:53:35might work well for simple questions and answers,
  5281. 5:53:38but they might struggle with more complex tool calling tasks, something like this.
  5282. 5:53:43So, in some cases, it might fail to recognize when a tool is needed,
  5283. 5:53:48or it might generate invalid arguments or request a tool that doesn't exist.
  5284. 5:53:53For example, a model with 8 billion parameters might work for simple tasks,
  5285. 5:53:59but it isn't guaranteed to follow every instruction, in these scenarios.
  5286. 5:54:04I mean, when you have some tools and you want to have an agentic behavior.
  5287. 5:54:09So use a model that is strong enough.
  5288. 5:54:12enough.
  5289. 5:54:13okay let's get back to our agent class previously
  5290. 5:54:16we initialized our message history with a system
  5291. 5:54:20message and it simply instructed the model to
  5292. 5:54:22give the short answers now we need to update it
  5293. 5:54:25and tell the model use appropriate tools whenever needed so
  5294. 5:54:30i'm going to say use the provided tools whenever need it okay that's it
  5295. 5:54:38and this instruction gives the agent some basic
  5296. 5:54:41behavioral guidance, exactly what we need.
  5297. 5:54:44Notice that we are not telling that it always should use a tool.
  5298. 5:54:48Maybe user ask something that doesn't need a tool, so that's really important.
  5299. 5:54:52Don't assume that when we provide tools, model
  5300. 5:54:56always needs to use one of those tools.
  5301. 5:54:59So that's very important.
  5302. 5:55:00It might request one of the tools, several of those tool or
  5303. 5:55:04maybe doesn't need to request any of those tools.
  5304. 5:55:07Okay, now this is a point where we need to construct our agent in this agent class.
  5305. 5:55:14And to do that, I told you that we want to use createAgent function.
  5306. 5:55:18So here in our constructor, I'm going to add another attribute
  5307. 5:55:24called self.agent using this createAgen that we just imported.
  5308. 5:55:29First, we need to specify the model that we are going to use.
  5309. 5:55:33So as a model, I'm going to this model. So here I say self.llm
  5310. 5:55:39And then we want to provide the tools that we have.
  5311. 5:55:43So here I'm going to set this tools argument and
  5312. 5:55:48I set it to a list and I add my tools here.
  5313. 5:55:51Currently I only have one tool. So here, I say get weather.
  5314. 5:55:56That's all we need to do at first.
  5315. 5:55:59LangChain takes care of much of the orchestration. Yeah, it does everything we need.
  5316. 5:56:04And let's use it here when we are calling our LLM here.
  5317. 5:56:09Instead of using my LL, I'm going to use this agent that I created.
  5318. 5:56:13And this agent has also this invoke method, so I don't have to change it.
  5319. 5:56:18But as an input, it needs a dictionary.
  5320. 5:56:21A dictionary that has a key called messages.
  5321. 5:56:25okay and this way I provide my messages list for it and this
  5322. 5:56:30is an important difference that you need to understand
  5323. 5:56:34and another important difference is response is no longer an
  5324. 5:56:38AI message it's a dictionary it's dictionary with this format
  5325. 5:56:42and the messages entry contains a conversation history but here when we want to
  5326. 5:56:47append that message to our conversation history we only need its final message
  5327. 5:56:53So here I say response, access to its messages
  5328. 5:56:57value, and then I only want the last item.
  5329. 5:57:01Okay. And the same for printing the response.
  5330. 5:57:04Here I say, I want messages entry, and
  5331. 5:57:08then I want the, last message and I only want its content.
  5332. 5:57:14And to clearly figure out that this tool is actually used,
  5333. 5:57:19let me put a print statement here and I say called
  5334. 5:57:24This way I can realize that this line in my own code is executed.
  5335. 5:57:28So let's run our program.
  5336. 5:57:31First, I enter a simple greeting.
  5337. 5:57:34Okay, we don't have tool calling here.
  5338. 5:57:36And then I ask what is the capital of France?
  5339. 5:57:42You see that again, we do not have toolcalling and the exciting point is here.
  5340. 5:57:47When I ask something about the weather, I ask, what is weather like in
  5341. 5:57:52Paris. Paris today.
  5342. 5:57:56Okay?
  5343. 5:57:57And it says I don't have real time data.
  5344. 5:58:02Probably we have something
  5345. 5:58:04wrong here.
  5346. 5:58:06Okay, it's saying that- Oh, I'm sorry.
  5347. 5:58:09I put the print statement before my tag string.
  5348. 5:58:14Okay.
  5349. 5:58:16let's run it one more time and here I say hello
  5350. 5:58:20and say what is the weather like in Paris
  5351. 5:58:24today.
  5352. 5:58:26okay this time you're seeing that we have
  5353. 5:58:30we have tool called and this line that you have you see here has been executed
  5354. 5:58:35and it says in paris today it's sunny with a temperature of about 22 degrees celsius
  5355. 5:58:42and as you can see this response is based on this
  5356. 5:58:45tool the information that this tool provided
  5357. 5:58:47okay let's terminate it and great we have now transformed our simple chatbot into a
  5358. 5:58:55tool using agent with just a few lines of code which is really fantastic and it's
  5359. 5:59:00the power of this interface create agent and here before finishing the session
  5360. 5:59:06let's take a moment to appreciate what LangChain is doing for us here
  5361. 5:59:10it's really important to understand what's going on behind the scenes
  5362. 5:59:15tool calling involves more than just asking an LLM to use a function
  5363. 5:59:19in a manual implementation
  5364. 5:59:22we need to check whether the model has requested
  5365. 5:59:25the tool, identify which tool it wants to call,
  5366. 5:59:27extract and validate the arguments, execute the
  5367. 5:59:31function and send the result back to the model.
  5368. 5:59:34Then we need check whether model is ready to produce
  5369. 5:59:38its final answer or not, or it still wants to
  5370. 5:59:41call another tool. And there are several edge cases here to consider. For example,
  5371. 5:59:47the model might not call any tool at all. It might request multiple tools.
  5372. 5:59:52Or it might request one tool, receive the
  5373. 5:59:56result and then decide it needs another tool.
  5374. 6:00:00So it has complexity and it is called a tool calling loop and it
  5375. 6:00:05is a loop that we need to implement ourselves.
  5376. 6:00:08Besides, we also need to think about invalid arguments.
  5377. 6:00:12Maybe our LLM provides invalid arguments or maybe we
  5378. 6:00:16have execution errors and some other unexpected cases.
  5379. 6:00:20So here you can say that Create Agent has managed much of this complexity.
  5380. 6:00:35Okay, in the last session, we use Create Agent to implement our agent.
  5381. 6:00:39But Create Agent isn't the only way to do that.
  5382. 6:00:42It isn't the the only to implement tool calling in Langchain.
  5383. 6:00:45There are three broad approaches worth understanding.
  5384. 6:00:49The first approach is binding tools directly to the model.
  5385. 6:00:53And this is a lower level approach. First, we import a decorator.
  5386. 6:00:58This way I say from Langchain dot tools import tool.
  5387. 6:01:04this is a decorator and we put this decorator before the function we are defining.
  5388. 6:01:10So here I say
  5389. 6:01:12tool and this turns this function into a tool okay
  5390. 6:01:17this way i mean this decorator helps lang chain
  5391. 6:01:21turns the function's name description and turns the function's name description and input schema into a tool definition
  5392. 6:01:26Next, we bind the tool to our model.
  5393. 6:01:29So for example, inside our constructor, we could write something like this.
  5394. 6:01:34We could say self.llm.
  5395. 6:01:38This LLM object has a method called bindTools.
  5396. 6:01:43So here I say bindTools and then I provide my tools inside a list.
  5397. 6:01:49So I say getWeather.
  5398. 6:01:52and to do this we should put this decorator here to use this approach i mean to use
  5399. 6:01:58this bind tools method
  5400. 6:02:01Okay, now the model knows about the available
  5401. 6:02:04tool definitions and can request a tool call
  5402. 6:02:07when appropriate.
  5403. 6:02:08For example, we can store it in an attribute called LLM with tools.
  5404. 6:02:14But there is an important distinction.
  5405. 6:02:16Binding a tool to a model doesn't automatically
  5406. 6:02:19execute the tool or manage the entire agent
  5407. 6:02:22loop. create agent approach that could do that.
  5408. 6:02:25In this approach, we need to invoke the model, inspect the response,
  5409. 6:02:29determine whether it contains tool calls, execute the requested functions,
  5410. 6:02:35and send the results back to the model.
  5411. 6:02:37So in this approach there is a lot to do and we have the full responsibility.
  5412. 6:02:42And as I told you, it is a lower level approach. So it was the first approach.
  5413. 6:02:48And the second approach is using create agent that we did in the previous lesson.
  5414. 6:02:54We create an agent, we provide tools, and then we
  5415. 6:02:58simply invoke the agent with the conversation state.
  5416. 6:03:02And LangChain manages the tool calling loop for us entirely.
  5417. 6:03:07And the third approach is, using LangGraph.
  5418. 6:03:10This becomes useful when we need more explicit control over the workflow.
  5419. 6:03:16things like state transitions, branching, persistence,
  5420. 6:03:20or other complex execution requirements.
  5421. 6:03:23So we want to use it when we actually need it.
  5422. 6:03:27For example, we might want an agent to follow a particular sequence of steps,
  5423. 6:03:33or pause for human approval before performing an action,
  5424. 6:03:37or route different types of requests through different processing paths.
  5425. 6:03:42However, that additional control
  5426. 6:03:45also comes with additional concepts and
  5427. 6:03:47implementation complexity so the right choice
  5428. 6:03:51depends on our requirements the requirements of our
  5429. 6:03:54projects or application not simply on whether you
  5430. 6:03:57are building an agent or not so these are the three
  5431. 6:04:01main approaches but before finishing this lesson
  5432. 6:04:04i should tell you that you might encounter some
  5433. 6:04:08other methods in other tutorials for example you
  5434. 6:04:12You might see something like this.
  5435. 6:04:15Create tool calling agent
  5436. 6:04:18or something like this agent executor these were commonly used in earlier lang chain
  5437. 6:04:24implementations but in newer versions of lang
  5438. 6:04:27chain create agent is recommended approach so
  5439. 6:04:30these are kind of outdated okay that's it
  5440. 6:04:43Okay, in this session, we are going to do something really exciting
  5441. 6:04:48I'm sure you have seen this feature many times when working with LLMs
  5442. 6:04:52Sometimes when you ask ChatChipity a question, you will
  5443. 6:04:55see a message indicating that it's searching the web
  5444. 6:04:59It's searching for the web to find an answer for you
  5445. 6:05:01And you might notice this, especially when you asked about specific
  5446. 6:05:06people, recent events or information that may have changed recently
  5447. 6:05:12Well, it relates to some of the limitations we have already discussed
  5448. 6:05:16including hallucinations and knowledge cut-offs
  5449. 6:05:20and LLM's knowledge comes from the data it was trained on
  5450. 6:05:24and that knowledge doesn't automatically
  5451. 6:05:26include everything that's happening right now
  5452. 6:05:28for example, if you ask whether a particular person has passed away
  5453. 6:05:33that person might have died today so the model doesn't know about that
  5454. 6:05:38and in situations like this
  5455. 6:05:40Searching the web can help the model access more up-to-date information.
  5456. 6:05:45And there is another scenario.
  5457. 6:05:47Imagine you are asking about a person, a company, or a
  5458. 6:05:51website that the model simply doesn't know much about.
  5459. 6:05:54Maybe that information wasn't included in its training data at all.
  5460. 6:05:59So, the model
  5461. 6:06:00might not have a reliable answer on its own, but it
  5462. 6:06:04could find relevant information by searching the web.
  5463. 6:06:07to see a tangible example I run the program
  5464. 6:06:11and I ask a question about myself I say
  5465. 6:06:15Who is Mohammad SeyedAghaei?
  5466. 6:06:21okay this is not a good answer as you can see this
  5467. 6:06:24information is not accurate and it's not relevant
  5468. 6:06:27to me let's ask about my academy the academy that
  5469. 6:06:31i run so i say what is the neon learn website
  5470. 6:06:38Again, the model doesn't necessarily have enough
  5471. 6:06:41relevant information to give me a good answer.
  5472. 6:06:44So, what we are going to do in this session is to connect our application to a tool.
  5473. 6:06:50A tool that can search the net to find the relevant information.
  5474. 6:06:54And the question is, how are we going to implement that? We are going use Tavily.
  5475. 6:07:01It's a service that makes web search easy to integrate into AI applications.
  5476. 6:07:05and Tavily provides a search API designed for AI-powered applications
  5477. 6:07:10and it offers a free plan that you can use to experiment with it
  5478. 6:07:14so you can create an account and you can generate an API key
  5479. 6:07:18and you copy and have this API key with you like me
  5480. 6:07:25keep it in your environment variables because we want to use it
  5481. 6:07:30Okay, here I store it in a variable called tavily API key.
  5482. 6:07:36okay pay attention that you need to use this name
  5483. 6:07:40because the integration will look for this name
  5484. 6:07:43now that we have our api key configured let's install
  5485. 6:07:47its package its integration for lang chain so
  5486. 6:07:50here i type q to quit and then i say pipenv install lang chain tavily
  5487. 6:08:00okay once the package is installed we can
  5488. 6:08:02import the search tool into our python code
  5489. 6:08:05so
  5490. 6:08:05on the top i say from lang chain tavily import tavily search
  5491. 6:08:12Okay, and now creating the tool is very straightforward.
  5492. 6:08:15Below our existing tool, which was for getting weather.
  5493. 6:08:19I add this following line to add this new search tool. I say tavily search.
  5494. 6:08:24It is a class and I only specify max results.
  5495. 6:08:28And I set it to one and I store it in a variable called search tool
  5496. 6:08:35Okay, that's it. We now have a web search tool that we can provide to our agent.
  5497. 6:08:40And you can think of this tool as being similar to using a search engine.
  5498. 6:08:45When you search for something on Google, for example, you search
  5499. 6:08:49for an animal, you typically get a list of results.
  5500. 6:08:53And these results come from different web pages.
  5501. 6:08:57The max results argument is exactly for that.
  5502. 6:08:59This parameter controls how many search results Tavily returns.
  5503. 6:09:03and here we have set it to one which means we are asking for just one result
  5504. 6:09:08now let's connect this tool to the agent in other
  5505. 6:09:11words let's provide this tool for our agent
  5506. 6:09:14so here in addition to this get weather tool i can add this
  5507. 6:09:18search tool okay now our agent is equipped with two tools
  5508. 6:09:23one tool for getting weather and another tool for searching the internet
  5509. 6:09:28Now with that in place, let's run our program.
  5510. 6:09:32First I'll ask the same question we tried earlier to see the difference.
  5511. 6:09:35I say who is Mohammad SeyedAghaei?
  5512. 6:09:43This information is not correct. This information
  5513. 6:09:46is not about me, but this shows that it I mean our agent searched the internet
  5514. 6:09:53used that tool to search the internet and found the information
  5515. 6:09:57and now let's search with different keywords using
  5516. 6:10:01Persian alphabet. So I say who is Mohammad SeyedAghaei
  5517. 6:10:08I get the same result so nothing changed but
  5518. 6:10:11let's ask about my academy what is new there
  5519. 6:10:16offering okay this time it is returning information that is quite relevant
  5520. 6:10:20because my website appears in the search results when it searches and it has
  5521. 6:10:26gathered this information through its search and through my website
  5522. 6:10:32okay so we successfully implemented what we wanted and I can tell you there is an
  5523. 6:10:38interesting connection between what we have just built and the Rack system we
  5524. 6:10:43implemented in the previous session
  5525. 6:10:45In our Rack project, we extracted information from PDF documents
  5526. 6:10:49and this time we are providing information via the internet.
  5527. 6:10:54So we can say the overall idea is similar, but
  5528. 6:10:57the source of the information is different.
  5529. 6:11:00In our previous project, we retrieved information from our own document collection.
  5530. 6:11:05And in this project, Tavily searches the web and provides search
  5531. 6:11:09results that the agent can use to answer the user's question.
  5532. 6:11:13So in both cases, we are giving the model additional information.
  5533. 6:11:26Alright, there is a technology I know some of you have already heard about.
  5534. 6:11:31And if you haven't, I'm sure you will hear about it a lot soon.
  5535. 6:11:35It is called MCP, which stands for Model Context Protocol.
  5536. 6:11:40So what is MCP and what problem is it designed to solve?
  5537. 6:11:44Let's start with something we have already done,
  5538. 6:11:47with something that we are already familiar with.
  5539. 6:11:51We have built tools and made them available to our LLM.
  5540. 6:11:55When the model needs to perform an action, it can
  5541. 6:11:58request that our application call the appropriate tool
  5542. 6:12:02Our application then executes the tool and sends the result back to the model
  5543. 6:12:08But imagine that you move to a completely different project
  5544. 6:12:12Maybe you are even using a different programming language
  5545. 6:12:15And you need exactly the same tools Without a standardized way to expose those tools
  5546. 6:12:22you might have to implement the same integrations all over again.
  5547. 6:12:27and you will need to figure out how to describe the tools
  5548. 6:12:31how to communicate with them and how to handle their inputs and outputs
  5549. 6:12:37That's where MCP comes in.
  5550. 6:12:39MCP is an open protocol introduced by Anthropic in 2024.
  5551. 6:12:45And its goal is to standardize how AI applications connect
  5552. 6:12:49to an external tool, to external tools in general.
  5553. 6:12:53Not only external tools but also data sources and other capabilities.
  5554. 6:12:58Instead of building a completely different integration for every application,
  5555. 6:13:03we can expose capabilities through an MCP server
  5556. 6:13:07and let compatible applications connect to them through this protocol.
  5557. 6:13:13And this makes integrations more reusable
  5558. 6:13:16and helps different AI applications work with the same services.
  5559. 6:13:21Okay, for example, imagine that Gmail wants to make
  5560. 6:13:24its capabilities available to AI applications.
  5561. 6:13:28It could expose operations such as searching emails, reading messages,
  5562. 6:13:33and creating drafts, and all these things through an MCP server.
  5563. 6:13:37And an AI application that supports this protocol
  5564. 6:13:41could then connect to those capabilities
  5565. 6:13:44without having to implement a completely custom integration for every operation.
  5566. 6:13:49And to understand how MCP works, let's start with two important components.
  5567. 6:13:55MCP server and the MCP client. First, MCP Server.
  5568. 6:14:00An MCP-server is a program that exposes capabilities
  5569. 6:14:03to an AI applications through this protocol.
  5570. 6:14:07For example, it can make tools available and describe what each tool does,
  5571. 6:14:12what inputs it accepts, and what kind of output it returns.
  5572. 6:14:17And this should sound familiar because we have
  5573. 6:14:20already done something similar ourselves.
  5574. 6:14:23Remember when we created our Get Weather tool.
  5575. 6:14:26We defined a function, gave it a name,
  5576. 6:14:28described its purpose, and specified the input it needed.
  5577. 6:14:33MCP server provides a standardized way to
  5578. 6:14:36expose tools and their descriptions so that
  5579. 6:14:39compatible clients can discover and use them.
  5580. 6:14:43Now let's elaborate on MCP client.
  5581. 6:14:46The MCP Client is a component that connects an AI application to an MCP server
  5582. 6:14:51and communicate with it using the MCP protocol.
  5583. 6:14:55Imagine you are building an AI Application.
  5584. 6:14:57Your application connects to an MCP server and discovers which tools are available.
  5585. 6:15:03The client receives their descriptions and makes
  5586. 6:15:06that information available to the agent or LLM.
  5587. 6:15:10And when the model decides that it needs a particular tool, the application
  5588. 6:15:16handles the request and invokes that tool through the MCP client.
  5589. 6:15:20The result is then returned to the model so it
  5590. 6:15:23can continue processing the user's request.
  5591. 6:15:26So the important idea is that as the MCP client
  5592. 6:15:29handles communication with the MCPServer,
  5593. 6:15:32while the LLM decides which tool it needs based on the
  5594. 6:15:36information and tool descriptions provided to it.
  5595. 6:15:40And the model itself doesn't need to implement the
  5596. 6:15:42MCP protocol or directly connect to the server.
  5597. 6:15:45Okay?
  5598. 6:15:46I'm sure many of you have already connected your Gmail account to an AI assistant
  5599. 6:15:51or you have seen AI agents that can interact with email services.
  5600. 6:15:54To implement this concept, MCP is the primary way.
  5601. 6:15:58And let's walk through a simplified example to better
  5602. 6:16:02understand how the different components work together.
  5603. 6:16:06Imagine that we have an AI application and we want to work with Gmail.
  5604. 6:16:11First, our AI application connects to a Gmail MCP server
  5605. 6:16:15and the server advertises the capabilities it supports.
  5606. 6:16:19For the sake of this example, let's imagine it exposes 10 different tools.
  5607. 6:16:24These might include a tool for reading emails, a tool, for searching, a tools for
  5608. 6:16:29reading spam messages, a tool for creating an email
  5609. 6:16:32draft, and several other tools that might appear.
  5610. 6:16:36The MCP client discovers those tools and makes
  5611. 6:16:39their descriptions available to the agent.
  5612. 6:16:41Now, let's say the user sends a message like this.
  5613. 6:16:45Write an email to Alex asking to schedule a meeting for next week.
  5614. 6:16:50Okay, and the LLM receives the user's request along with the available tools.
  5615. 6:16:55and based on that information, it determines that it needs to create an email draft.
  5616. 6:17:00It then generates a tool call requesting the appropriate
  5617. 6:17:04operation, such as write draft with the necessary arguments.
  5618. 6:17:08And our AI application receives that tool call request and
  5619. 6:17:12invokes the corresponding tool through the MCP client.
  5620. 6:17:16And the MCPServer responses the request and returns the result,
  5621. 6:17:20such as confirmation that the draft was created successfully.
  5622. 6:17:24The application then provides that result to the LLM.
  5623. 6:17:27Finally, the LLM can respond to the user like this. I've created the email draft.
  5624. 6:17:32You can review it in your Gmail before sending it.
  5625. 6:17:35So, let's summarize the flow that I explained.
  5626. 6:17:38The user sends a request to the AI application.
  5627. 6:17:41The application makes the available tool descriptions accessible to the llm.
  5628. 6:17:46The llm decides which tool to call and generates a tool call request.
  5629. 6:17:51and the application invokes the tool through the protocol
  5630. 6:17:54and the MCP server executes the requested operation and returns a result
  5631. 6:17:59and the applications sends that result back to the LLM
  5632. 6:18:03and the LLM generates a final response for the user
  5633. 6:18:06and that's the basic idea behind MCP.
  5634. 6:18:09It provides a standardized way to connect AI
  5635. 6:18:12applications with external capabilities,
  5636. 6:18:14making integrations easier to reuse across different applications and
  5637. 6:18:19development environments. Welcome
  5638. 6:18:21back aboard Neon Learned.
  5639. 6:18:24Planet Agent is conquered. You're halfway there, five worlds to go.
  5640. 6:18:29Fasten your seat belt. Next stop, Planet Structured.
  5641. 6:18:32Here, you'll master prompt templates and make any LLM return clean structured data
  5642. 6:18:37exactly how your app needs it.
  5643. 6:18:40Your next world awaits. Enjoy the journey.
  5644. 6:18:50Okay, starting from this session and over the
  5645. 6:18:52next few sessions, we are going to explore
  5646. 6:18:55a group of classes in LangChain called Prompt Templates.
  5647. 6:18:58So far, we have been providing prompts to our LLM as simple text inputs.
  5648. 6:19:03And depending on the role we needed, we create
  5649. 6:19:06an instance of system message, human message,
  5650. 6:19:08or AI message and pass a plain text string to it.
  5651. 6:19:12And honestly, that's perfectly normal.
  5652. 6:19:14I mean, for many projects, this approach works just fine.
  5653. 6:19:18As our projects grow behind a certain size,
  5654. 6:19:21having lots of plain prompt strings scattered
  5655. 6:19:24throughout our code can make things messy.
  5656. 6:19:27And it becomes harder to maintain our prompts.
  5657. 6:19:30It makes it harder to reuse them across different parts
  5658. 6:19:33of our application and keep our code organized.
  5659. 6:19:36And for this reason, there are some libraries specialized for maintaining prompts.
  5660. 6:19:42One example you might have heard of is DSPy.
  5661. 6:19:46which provides a more programmatic approach to
  5662. 6:19:49building and organizing LLM powered systems.
  5663. 6:19:52And with approaches like this, instead of
  5664. 6:19:54treating every prompt as an isolated string,
  5665. 6:19:57you can organize your logic into reusable components.
  5666. 6:20:01And of course, whether you need a dedicated framework
  5667. 6:20:05like DSPy depends on the complexity of your project.
  5668. 6:20:08For a small application, introducing an additional
  5669. 6:20:12framework like this might be unnecessary.
  5670. 6:20:14Besides DSPY, fortunately, LangChain also
  5671. 6:20:17provides its own classes for prompt management.
  5672. 6:20:21And these classes let us create dynamic reusable prompts
  5673. 6:20:25without adding too much complexity to our application. The idea is pretty simple.
  5674. 6:20:30Instead of hard coding every value directly into a string,
  5675. 6:20:34we can define placeholders and inject different values into them
  5676. 6:20:38whenever we need to use the prompt.
  5677. 6:20:40If you have worked with Python formatted strings, This idea should feel familiar.
  5678. 6:20:45We define a template with placeholders and then provide
  5679. 6:20:49the values that should go into those placeholders.
  5680. 6:20:52And the difference is that LangChain gives us dedicated
  5681. 6:20:57object for managing these prompts and these templates.
  5682. 6:21:00And because prompt templates can be used as runnable components, we can also
  5683. 6:21:06connect them to other components using LangChains syntax, which is called
  5684. 6:21:11L-C-E-L or Langchain Expression Language
  5685. 6:21:14and that's where the name Langchain starts to make even more sense.
  5686. 6:21:18We can chain different runnable components together to build more complex workflows.
  5687. 6:21:25Now to save some time, I've already created a file called prompt template.py
  5688. 6:21:30and placed some of the code we have written in previous session inside it.
  5689. 6:21:35And there is nothing new. As you can see, it has a
  5690. 6:21:38constructor here it has a chat method and it has an
  5691. 6:21:41LLM call so if I run the program and as you can
  5692. 6:21:45see I get a response so this is a code I've written
  5693. 6:21:48and just write this code to follow along with me
  5694. 6:21:53Oh
  5695. 6:21:58now let's move on to our first prompt template class
  5696. 6:22:01the first class we are going to look at is called
  5697. 6:22:04prompt template it's one of the simplest classes in
  5698. 6:22:07this family prompt templates are available through
  5699. 6:22:10langchain core so if you want to import them we need to use langchain
  5700. 6:22:14core so i say from langchaincore.prompts
  5701. 6:22:20import i want to import prompt template okay this class
  5702. 6:22:24and the purpose of this class is straightforward
  5703. 6:22:27it lets us turn a regular string into a reusable template with dynamic placeholders
  5704. 6:22:33let's see how that works with a practical example
  5705. 6:22:36imagine that we want to generate an email whose
  5706. 6:22:39content changes depending on the user's name and the
  5707. 6:22:43company's brand name and we want the email to start with
  5708. 6:22:47the user's name and end with the company's name.
  5709. 6:22:50So instead of creating a completely new string
  5710. 6:22:53every time, we can define a template and we
  5711. 6:22:55can fill in the values whenever we need it. So first let's create a regular string.
  5712. 6:23:02So I define a templat. I use triple quotes to write in multiple lines.
  5713. 6:23:07I say here and here I have a placeholder and I name it a username.
  5714. 6:23:14And then in the next line I say
  5715. 6:23:17apologizing for the bad experience we will set an
  5716. 6:23:21appointment to further discuss your problem and
  5717. 6:23:26at the end sincerely and at the ends we have another placeholder called brand name
  5718. 6:23:34okay so pay attention to these two username and
  5719. 6:23:37brand name whenever we want a value to be dynamic
  5720. 6:23:41we can put a placeholder just like when we have a
  5721. 6:23:44formatted string and then inside these pair of
  5722. 6:23:48curly braces we can have a variable name and in this case we have username and
  5723. 6:23:53brand name that represents the company's name
  5724. 6:23:56And at this point, this is still a regular string.
  5725. 6:24:00We haven't converted it into a LangChain prompt template yet.
  5726. 6:24:04So let's do that.
  5727. 6:24:05To convert our string into a prompt template, we'll
  5728. 6:24:09create an instance of the class we just imported.
  5729. 6:24:12And then its constructor accepts the template and the variable names that we have.
  5730. 6:24:17So let' s do it. I store the string inside a variable called template.
  5731. 6:24:23and then we can have our prompt
  5732. 6:24:27and our prompt is constructed using this class that I told you prompt template
  5733. 6:24:33and then as I told first we have our input variables this is a dictionary
  5734. 6:24:39this is the list sorry
  5735. 6:24:40this is list and it contains our variables and we have a template
  5736. 6:24:45So this is my template.
  5737. 6:24:48and the variables that we have are username and
  5738. 6:24:52here we have brand name okay we need to put them
  5739. 6:24:56inside strings
  5740. 6:24:58now this prompt that we have here gives us a method
  5741. 6:25:01called format which we can use to insert the
  5742. 6:25:04actual values into our template so let's say the
  5743. 6:25:07user's name is Alex and the company's name
  5744. 6:25:10is NeonLearn so we can generate a dynamic prompt like
  5745. 6:25:14this i say prompt i use this method as i told you format
  5746. 6:25:19and here i need to specify my variables the variables that i have in this prompt so
  5747. 6:25:24the first one is username i want it to be alex and the latter one is brand name
  5748. 6:25:32which is neonLearn
  5749. 6:25:34okay so the method takes the values we provided and
  5750. 6:25:38insert them into the corresponding placeholders
  5751. 6:25:41and at the end let's print the result so i say print result
  5752. 6:25:46Okay, I donate these lines and, oh, result.
  5753. 6:25:49I forgot to store it in a variable called result.
  5754. 6:25:52Let's run our program.
  5755. 6:25:55Okay, that's it, dear Alex.
  5756. 6:25:57And at the end, NeonLearn added was the whole purpose of this session.
  5757. 6:26:01We have created a dynamic prompt template that
  5758. 6:26:03we can reuse whenever we need to generate this
  5759. 6:26:06kind of email.
  5760. 6:26:07And if you want to generate an email for a
  5761. 6:26:10different user or a different brand name, we
  5762. 6:26:12don't need to rewrite the entire string.
  5763. 6:26:14We simply call this method, I mean format, and we only change the values.
  5764. 6:26:19And that's the main benefit of using prompt template.
  5765. 6:26:22we separate the structure of our prompt from
  5766. 6:26:25the specific values we want to insert into it
  5767. 6:26:29you
  5768. 6:26:35Okay, the next class we are going to explore is Chat Prompt Template.
  5769. 6:26:39So what's the purpose of this one?
  5770. 6:26:42In previous lesson, we used Prompt template to turn a simple
  5771. 6:26:45string into a reusable template with dynamic placeholders.
  5772. 6:26:49But in many applications, especially when we are building
  5773. 6:26:53chatbots, we are not working with just one plain text.
  5774. 6:26:57Instead, we're working with a sequence of conversation messages.
  5775. 6:27:01and some messages come from the user some are generated by the AI and others provide
  5776. 6:27:07system level instructions things that you saw before so if
  5777. 6:27:11you want to create a reusable
  5778. 6:27:13template we need a way to represent these
  5779. 6:27:16different messages with different roles and
  5780. 6:27:19that's exactly what chat prompt template is designed for
  5781. 6:27:23so let's see how it works first
  5782. 6:27:26I import it instead of prompt template I import
  5783. 6:27:29chat prompt template.
  5784. 6:27:31and i don't want these quotes here okay it was for our previous session
  5785. 6:27:37okay this class that i imported provides a
  5786. 6:27:40convenient class method called from messages
  5787. 6:27:44and we use this method to create a chat prompt
  5788. 6:27:47template from a list of messages and each message
  5789. 6:27:50can be represented as a tuple something that we
  5790. 6:27:54saw in open ai wrapper format it has two elements
  5791. 6:27:57the first element specifies a role and the second
  5792. 6:28:01element contains the message context so let's have it
  5793. 6:28:04I told you that it is a class method.
  5794. 6:28:07So I say chat from template, from messages, and it takes
  5795. 6:28:11a list, a list of tuples, and each tuple is a message.
  5796. 6:28:15So I want to have two messages, a system message and an AI message.
  5797. 6:28:20So I specify its role like this.
  5798. 6:28:23I say system and then I have its content and for user message, I say human.
  5799. 6:28:30Let's create something practical with it.
  5800. 6:28:32Imagine we are creating a translation application.
  5801. 6:28:35We want the model to translate text from one language into another language.
  5802. 6:28:40And we want the source language and the target language to be dynamic.
  5803. 6:28:44In other words, we don't want to hard code the target
  5804. 6:28:47language and source language into the prompt.
  5805. 6:28:49We want to be able to change them whenever we use it. So let's create our prompt.
  5806. 6:28:55As a system message I say,
  5807. 6:28:57you are a translator from our source language to the target language and
  5808. 6:29:03here
  5809. 6:29:04i have another placeholder called input
  5810. 6:29:07Let's take a closer look at it.
  5811. 6:29:09Origin, as I told you, specifies the source
  5812. 6:29:11language and this one is a target language and
  5813. 6:29:14this input contains the text that we want to be translated.
  5814. 6:29:19So, the system message tells the model about the task and
  5815. 6:29:23the human message contains the actual text needs to be
  5816. 6:29:26translated.
  5817. 6:29:27Now that we have created our template, let's see how we can fill in these values.
  5818. 6:29:32We can use format message method.
  5819. 6:29:36from messages now we have a format messages so
  5820. 6:29:39let's store this in a variable called prompt
  5821. 6:29:43and then i say prompt format messages and then I'll specify these three values okay
  5822. 6:29:51I set origin to
  5823. 6:29:53English and
  5824. 6:29:55I set to to German and I want the input to be I love you
  5825. 6:30:02Okay, then I store it in a variable called
  5826. 6:30:05messages because it returns a list of messages
  5827. 6:30:09with different roles and I print it. I say print messages.
  5828. 6:30:13Now let's run our program.
  5829. 6:30:16and here you see the output as you can see
  5830. 6:30:18Langchain takes our list of message templates
  5831. 6:30:21fill in the placeholders and converts the result into message objects and
  5832. 6:30:26internally the top will be provided are represented by the corresponding
  5833. 6:30:30Langchain message classes and in this example we'll get a system message as you
  5834. 6:30:34can see and we have a human message here okay
  5835. 6:30:38that's the main idea behind chat prompt template but now
  5836. 6:30:42I want to show you something even more interesting.
  5837. 6:30:45How we can connect this prompt to our LLM using the syntax I told you about.
  5838. 6:30:51LangChains LCEL syntax. This syntax is really simple.
  5839. 6:30:57All you need is a pipe operator.
  5840. 6:30:59And this way we can connected one component to another using a pipe.
  5841. 6:31:04Then connect another component after that and continue building a chain as needed.
  5842. 6:31:09The first component receives an input and produces an output.
  5843. 6:31:13That output is passed to the next component as its input.
  5844. 6:31:17And the next components processes it and produces another output and it continues.
  5845. 6:31:22Okay, and this is how we can build a workflow.
  5846. 6:31:25A workflow by connecting multiple different components together.
  5847. 6:31:29And there is one important requirement to do that.
  5848. 6:31:33Components used in this syntax needs to be runnable.
  5849. 6:31:37needs to support the runnable interface. So otherwise you cannot do that.
  5850. 6:31:43For example, this self.llm you see here is a
  5851. 6:31:46runnble component and this class that you
  5852. 6:31:49imported for generating your prompt is also
  5853. 6:31:52a runable component or it's better to say
  5854. 6:31:54it produces runnible objects for you and this
  5855. 6:31:58means you can use these components inside
  5856. 6:32:01the syntax lcel syntax. So let's
  5857. 6:32:04do that!
  5858. 6:32:06I told you we have two pronable components here.
  5859. 6:32:09So here I can put the first component, which is our prompt.
  5860. 6:32:14And then I use a pipe and then we have our LLM. Okay.
  5861. 6:32:18And I store the result inside a variable called chain. So let's break it down.
  5862. 6:32:23The first component is prompt and it receives
  5863. 6:32:26the values needed to fill in our template.
  5864. 6:32:29And it generates a list of chat messages and the second component.
  5865. 6:32:33is our LLM and it receives those list of messages
  5866. 6:32:37and it sends them to our language model to our LLLM
  5867. 6:32:41and by connecting these two components we have created a single runnable chain
  5868. 6:32:46so this component that we have here is also runnible
  5869. 6:32:50and this means instead of manually calling the prompt
  5870. 6:32:53collecting the resulting messages and then
  5871. 6:32:56passing those messages to our lllm ourselves
  5872. 6:32:59we can let the chain handle that sequence for us.
  5873. 6:33:02And as I told you, this object that we have here is runnable
  5874. 6:33:06and one primary feature of runnables objects is
  5875. 6:33:10that you can call the invoke method on them.
  5876. 6:33:13So here we can say chain and call the invoke method.
  5877. 6:33:18And as i told you it's our first component
  5878. 6:33:21and we need to provide the values that our first components in this change needs.
  5879. 6:33:28so what values do we need we need origin to and
  5880. 6:33:32input the exact values that we have here so i
  5881. 6:33:35remove it and i move these values here and instead
  5882. 6:33:39of equal sign i use colon because i want to have
  5883. 6:33:42a dictionary here
  5884. 6:33:44okay and i put curly braces around them and now i should put these inside
  5885. 6:33:50quotations okay now everything
  5886. 6:33:52is ready and i just need to store the result
  5887. 6:33:56in a variable called response
  5888. 6:33:59and I print the response.
  5889. 6:34:02Okay, it's an AI message. I just want its content.
  5890. 6:34:06okay let's run the application and it should return the german translation of
  5891. 6:34:12uh the sentence so it says it uh so that's totally correct and that's a whole
  5892. 6:34:18process instead of
  5893. 6:34:20manually passing data from one component to another
  5894. 6:34:23we have created a chain that handles the the flow
  5895. 6:34:26for us and the best part is that we can extend
  5896. 6:34:29this we can expand this chain whenever we need to
  5897. 6:34:33but we can add more components after the LLM or build more complex workflows.
  5898. 6:34:37We can add components before it and whenever we want,
  5899. 6:34:40as long as the components are runnable objects.
  5900. 6:34:44Ladies and gentlemen, we're passing through a little turbulence.
  5901. 6:34:49If things feel harder right now, that's completely normal. It means you're climbing.
  5902. 6:34:54Every AI engineer you admire flew through this exact same part of the journey.
  5903. 6:35:00Take a deep breath, stretch your wings, maybe grab a glass of water.
  5904. 6:35:04Now look out the window, see how far you've already come.
  5905. 6:35:08The view only gets better from here, let's keep climbing.
  5906. 6:35:21Alright, next let's imagine that we want to extend our previous example
  5907. 6:35:26by including the conversation history between the user and the AI.
  5908. 6:35:31And we want the system message to stay at the beginning, the latest user message to
  5909. 6:35:36stay at end, and the conversation story to be inserted in between.
  5910. 6:35:41And the question is why might we want do this? There are several reasons.
  5911. 6:35:46We might want the LLM to take previous messages into account so
  5912. 6:35:50that its responses are more relevant to the ongoing conversation.
  5913. 6:35:54Or we might want to use few-shot prompting, where we
  5914. 6:35:58provide a few examples of the kind of output we expect.
  5915. 6:36:01And this way, we kind of control the output,
  5916. 6:36:05something that we saw in previous projects.
  5917. 6:36:08And for this example, our focus is on few-shots prompting.
  5918. 6:36:13and imagine that we are building a translation application and
  5919. 6:36:17we want the model to wrap its translations in three angle brackets on each side
  5920. 6:36:22and we can provide a couple of examples
  5921. 6:36:25a couple examples in the conversation history
  5922. 6:36:28to show the model the exact format we expect
  5923. 6:36:31The point is that currently this class, I mean Chat Prompt template by itself
  5924. 6:36:36doesn't automatically insert a separate list of conversation messages.
  5925. 6:36:40So we need another class and by using this new class we can create placeholder here.
  5926. 6:36:47So
  5927. 6:36:47at first I import this class that I told you which is called
  5928. 6:36:51message placeholder, messages placeholder more accurately.
  5929. 6:36:56and now before we use it we use the conversation history that we have
  5930. 6:37:00so on the top I create a list called history
  5931. 6:37:04and inside this list I have four messages
  5932. 6:37:07two human messages and two responses by the LLM
  5933. 6:37:10so I say human message and my message is I am tired
  5934. 6:37:15okay and the answer is Ich bin mude with the translation more accurately
  5935. 6:37:19so it is an AI message and as I told you I want three angle brackets
  5936. 6:37:25three before and three after Ich
  5937. 6:37:27bin müde let's duplicate these two lines, okay?
  5938. 6:37:31I hate you Ich hasse dich
  5939. 6:37:34These examples demonstrate the output format we want the model to follow
  5940. 6:37:38and the translated text should be surrounded by three angle brackets on each side.
  5941. 6:37:44Okay, now let's update our chat prompt template.
  5942. 6:37:47Previously, our template contained a system message and a human message.
  5943. 6:37:51This time we are going to insert a messages placeholder between them.
  5944. 6:37:54So here I'm going to insert this messages placeholders that I told you.
  5945. 6:37:59And here I need to specify a variable.
  5946. 6:38:02like these variables that we had so here i set
  5947. 6:38:06variable name and i set it to conversation history
  5948. 6:38:10so now this is all we need and let's take a closer
  5949. 6:38:13look the first message here is a system message
  5950. 6:38:16it tells the model which languages to translate
  5951. 6:38:19between and next we have this place folder it tells
  5952. 6:38:22lang chain where the conversation history should be
  5953. 6:38:25inserted the whole point of this session is that we cannot
  5954. 6:38:29put this conversation history directly here.
  5955. 6:38:32First, we need to create a placeholder and
  5956. 6:38:35this placeholder contains a variable name.
  5957. 6:38:38And when we want to invoke our chain, we
  5958. 6:38:40need to specify this variable and our value would be this history.
  5959. 6:38:45at the end that we updated our prompt we need to pass the conversation history
  5960. 6:38:49and invoke our change so we only need to specify this new variable that we had here
  5961. 6:38:55so i add conversation history and here the order
  5962. 6:38:58doesn't matter so we have conversation history
  5963. 6:39:02and i set it to my history the history that i
  5964. 6:39:05stored in the list called history and that's it
  5965. 6:39:09Let's run our program.
  5966. 6:39:11It says messages placeholder object is not callable. What's wrong with it?
  5967. 6:39:17Oh, here we forgot a comma.
  5968. 6:39:20Okay, again, one more time.
  5969. 6:39:23Okay, congratulations.
  5970. 6:39:25And this time, as you can see, the model receives the
  5971. 6:39:28system instruction and it also receives the examples.
  5972. 6:39:32And this way it realizes the format that we expect.
  5973. 6:39:36So the previous examples demonstrate the output format, the output format we want.
  5974. 6:39:41And so the model has a reference for how to format its new translation.
  5975. 6:39:45As you can, there are three angle brackets at the beginning and at the end.
  5976. 6:39:50So exactly what we wanted.
  5977. 6:39:51and we used few shot prompting to achieve this goal.
  5978. 6:40:01Okay, the next topic in this course is something called Structured Output.
  5979. 6:40:06This is a really useful and important concept
  5980. 6:40:08because in many applications, we need our LLM
  5981. 6:40:11to return its response in a special format.
  5982. 6:40:14For example, imagine that we provide the model
  5983. 6:40:17with several paragraphs describing a product,
  5984. 6:40:20and we wanted to extract some specific information
  5985. 6:40:24such as product's name, price, seller, and
  5986. 6:40:27the product's weight. We don't just want a paragraph describing the product.
  5987. 6:40:31We want a structured response containing exactly the fields we are interested in,
  5988. 6:40:36such as product name, price, seller, and weight. And why? Why is that important?
  5989. 6:40:41Because we are going to use these values in the rest of our application.
  5990. 6:40:45We want to be able to access the product's price through a special field,
  5991. 6:40:49retrieve the seller's name through another field, and so on.
  5992. 6:40:53Depending on which library, framework, or wrapper we are using,
  5993. 6:40:57the implementation can be different.
  5994. 6:40:59So, for example, if we were working directly with the OpenAI SDK,
  5995. 6:41:04we could use its supported structured output capabilities.
  5996. 6:41:07And that way we could specify the expected output format and schema.
  5997. 6:41:12But since we are using LangChain, I want to
  5998. 6:41:15show you the best practice in LangChains.
  5999. 6:41:18I mean, I wanna show you how we can approach
  6000. 6:41:21this problem within the LangChai ecosystem.
  6001. 6:41:24And remember something I've emphasized throughout this course.
  6002. 6:41:28Understanding the concept is more important
  6003. 6:41:31than memorizing every implementation detail.
  6004. 6:41:34You don't necessarily need to remember every method or every library function.
  6005. 6:41:38What's important is knowing that a capability exists.
  6006. 6:41:42Understanding what it is used for and being able to
  6007. 6:41:45find the relevant documentation when you need it.
  6008. 6:41:49That's especially true in software development today. tools and APIs change rapidly.
  6009. 6:41:55But something that is permanent is understanding the underlying concept.
  6010. 6:41:59These concepts are permanent, but the tools, frameworks and other stuff change.
  6011. 6:42:05So the question is, how are we going to implement structured output in LangChain?
  6012. 6:42:11One of the best ways is using Pydantic.
  6013. 6:42:14So first let's install it and I will elaborate on that. So here I say pipenv
  6014. 6:42:21Install Pydantic
  6015. 6:42:25Okay.
  6016. 6:42:27But before we use Pydantic with Langchain, I want to spend
  6017. 6:42:31a little time understanding how Pydantic itself works.
  6018. 6:42:36And I need to say its main purpose is data validation.
  6019. 6:42:39It tells us whether a piece of data follows a schema or not.
  6020. 6:42:44Imagine you are building a web application, a web
  6021. 6:42:47application with a form where users enter information such as their email address,
  6022. 6:42:52username and age for example.
  6023. 6:42:56and the user might accidentally enter invalid
  6024. 6:42:59information or they might deliberately do that
  6025. 6:43:02so that's really important to validate the data we receive
  6026. 6:43:06we need some rules and Pydantic is a library that
  6027. 6:43:09helps us define and enforce these kinds of rules
  6028. 6:43:13before we look at Pydantic itself let's briefly review a python feature
  6029. 6:43:17Pydantic builds on type hints suppose we define a variable like
  6030. 6:43:22name and its value is for example John okay now if we want to indicate this name
  6031. 6:43:29that we have here is expected to be a string we can write something like this
  6032. 6:43:33we can put a colon after and we say it is a string what you see here is a type
  6033. 6:43:38hint it tells tools that analyze our code that this variable is expected
  6034. 6:43:43to contain a string and type hints make code easier to understand and
  6035. 6:43:48and can help development tools provide better autocomplete,
  6036. 6:43:52auto-intelligence, and static error checking.
  6037. 6:43:55So, that's the purpose of type hint.
  6038. 6:43:57However, there is an important detail to understand.
  6039. 6:44:00Python type hints don't automatically enforce these types at runtime.
  6040. 6:44:05For example, writing something like this doesn't
  6041. 6:44:08prevent someone from assigning a different type.
  6042. 6:44:11For example putting a number inside this variable.
  6043. 6:44:14And the point is that Pydantic takes this idea further.
  6044. 6:44:19I mean, when we define a Pydantic model, it uses the declared types.
  6045. 6:44:23We also specify some validation rules and Pydantic uses them.
  6046. 6:44:27Uses to process and validate the data at runtime.
  6047. 6:44:31So you can think of type-ins as a way to describe the expected structure.
  6048. 6:44:36While Pydantic provides a mechanism for checking that data against the structure.
  6049. 6:44:42So that's the point.
  6050. 6:44:44So, to get started, to learn about PyDantic, I
  6051. 6:44:48create a new Python file called PyDantikValidation.
  6052. 6:44:53okay that's it and now we will use this file to explore
  6053. 6:44:57pydantic before integrating it with lang chain
  6054. 6:45:01so the first step is to import base model from pydantic
  6055. 6:45:06so i say from pydantic import baseModel okay what is baseModel?
  6056. 6:45:12it's a base class that we inherit from when defining our pydantic data models
  6057. 6:45:18let's have an example we want to define a user with three fields
  6058. 6:45:22id name and age so first i create a new class called user
  6059. 6:45:29it inherits from base model, as I told you.
  6060. 6:45:32and i want to have three fields id name and age i
  6061. 6:45:37want id to be an integer name to be a string str
  6062. 6:45:43i need to say int sir and int and this way with these
  6063. 6:45:47type hints i am defining the shape of my user
  6064. 6:45:51my user is supposed to have these three fields okay now
  6065. 6:45:56let's create an instance of this model and here's
  6066. 6:46:00Here's where things get interesting. Let's create a user.
  6067. 6:46:04I say user and I specify the value of these fields.
  6068. 6:46:08I say ID is 1, but I put 1 inside quotation.
  6069. 6:46:14and I say name,
  6070. 6:46:15name is Alice
  6071. 6:46:18and then age 25 okay and then i print it first
  6072. 6:46:22i store it in a variable called user and then
  6073. 6:46:26i print print user
  6074. 6:46:29and you have probably noticed something unusual here
  6075. 6:46:32I have 25 in start quotations technically a string and the same for id
  6076. 6:46:37so these are now strings not integers
  6077. 6:46:41but when I run my code you can see now that these are actually integers
  6078. 6:46:47so as you can pidentic has automatically converted the type
  6079. 6:46:52this kind of conversion is especially useful when working with web applications
  6080. 6:46:57because data submitted through HTML forms often arrives as strings
  6081. 6:47:02even when the user enters a number.
  6082. 6:47:04So Pydantic can convert compatible values into the types declared by our model,
  6083. 6:47:10saving us from writing some of that conversion logic ourselves.
  6084. 6:47:15However, this doesn't mean Pydantic can convert every value into every type.
  6085. 6:47:20Conversion depends on the input and the field's validation rules.
  6086. 6:47:25if a value can't be processed successfully, Pydantic raises a validation error.
  6087. 6:47:30For example, if we had something like this, ABC. Let me run our code.
  6088. 6:47:35As you can see, it raises an error, so it's not possible.
  6089. 6:47:39It is only possible when we have number.
  6090. 6:47:42Okay, now that we understand the basics, let's look
  6091. 6:47:45at another useful feature, the field function.
  6092. 6:47:48So far, we have only specified the expected type of each field. For example.
  6093. 6:47:54we have set that name must be a string.
  6094. 6:47:57But what if we also want to specify a minimum and maximum length?
  6095. 6:48:01Or what if you want to restrict the user's age to a particular range?
  6096. 6:48:07This is where field becomes useful. So first I import it.
  6097. 6:48:12I say field here
  6098. 6:48:13and we can
  6099. 6:48:15use field to attach additional validation constraints.
  6100. 6:48:19And this way we can have more control.
  6101. 6:48:22Here I want to have mean length 2, max length 50.
  6102. 6:48:25So let's say field, mean
  6103. 6:48:28length, 2 and max length, 50.
  6104. 6:48:32And then for age, I'm going to say
  6105. 6:48:36g e 0 and
  6106. 6:48:37le 120 okay let's break this down for the name field
  6107. 6:48:43we have specified
  6108. 6:48:45mean length to max length 50 this means the
  6109. 6:48:48string must contain at least two characters
  6110. 6:48:52and no more than 50 characters and for the age we are specifying that i mean
  6111. 6:48:58this g you see here means greater than or equal to and le means
  6112. 6:49:03less than or equal to. So the age must be between 0 and 120 inclusive.
  6113. 6:49:10Okay.
  6114. 6:49:11And if we try to create a user with a name that
  6115. 6:49:14violets the length or an age outside this range,
  6116. 6:49:18Pydantic raises a validation error.
  6117. 6:49:21And this allows us to define our validation rules
  6118. 6:49:24directly alongside the field they apply to.
  6119. 6:49:28Okay.
  6120. 6:49:29There is one more feature of field that will be particularly important
  6121. 6:49:33when we get to structure out within LangChain.
  6122. 6:49:36Besides validation constraints, field also lets us attach metadata,
  6123. 6:49:41including a description and examples. Why is that useful?
  6124. 6:49:46When we ask an LLM to generate structured data,
  6125. 6:49:49we want to clearly explain what each field represents.
  6126. 6:49:52The name of the field that we have might not always be enough and even constraint.
  6127. 6:49:58A description and examples can help the model a lot.
  6128. 6:50:02So here we can have a description.
  6129. 6:50:04For example, for this field, I set its description to
  6130. 6:50:09the users.
  6131. 6:50:10Full name.
  6132. 6:50:12and I can provide some examples for example Alice and
  6133. 6:50:19Okay?
  6134. 6:50:20Now, in addition to specifying the type and length constraints,
  6135. 6:50:24we have documented what the field represents and provided two example values. Okay.
  6136. 6:50:31So now you have a good understanding of what Pydantic does.
  6137. 6:50:37you
  6138. 6:50:43Alright, let's put what we learned about Pydantic in previous lesson into practice.
  6139. 6:50:48We are going to use Pydantic to control the structure of our LLM's output,
  6140. 6:50:52so we can get the exact type of information we need in a predictable format.
  6141. 6:50:57For this project, I'm going to copy a paragraph from Elon Musk's Wikipedia
  6142. 6:51:02and pass it to my LL as input.
  6143. 6:51:04Then I'll ask the model to extract four pieces of information.
  6144. 6:51:09his name, his age, associated companies, and his net worth.
  6145. 6:51:14So let's go to our code and do this.
  6146. 6:51:16For this example, I'm going to use the project that we had prompt template. Okay.
  6147. 6:51:21That's perfectly fine.
  6148. 6:51:23So first I'm gonna import base model and field, which we saw in the last session.
  6149. 6:51:28So I say from Pydantic import base, model and filled.
  6150. 6:51:34Then I'll create a class called user that inherits
  6151. 6:51:37from base model with the field that I want.
  6152. 6:51:40So here I'm going to create a a class call user.
  6153. 6:51:44It inherits it from
  6154. 6:51:46baseModel.
  6155. 6:51:48and the field that I have are name
  6156. 6:51:52H
  6157. 6:51:54Companies and Net Worth.
  6158. 6:51:58now it's turn for typins i want it to be
  6159. 6:52:02stir this one integer and companies are a list of strings
  6160. 6:52:08and net worth can be a simple string okay
  6161. 6:52:12and we need four fields for each one containing a description and this
  6162. 6:52:17description is going to guide our element so here i add a field with the
  6163. 6:52:22description i say the name of the user
  6164. 6:52:26Let's just copy it and paste it here.
  6165. 6:52:30Here I say the age of the user.
  6166. 6:52:33And here say the
  6167. 6:52:36companies associated with the user and finally
  6168. 6:52:40for net worth I say the net worth of the user.
  6169. 6:52:43Okay, that's it.
  6170. 6:52:45And as you can see, we are specifying the
  6171. 6:52:48expected data type for each field, along with
  6172. 6:52:51a description explaining what information it should contain.
  6173. 6:52:54Next, for this particular task, it's a good idea to use a lower temperature.
  6174. 6:52:59For example, it is okay to say 0.1 because we don't want any creativity at all.
  6175. 6:53:05We are dealing with facts and a lower temperature is more suitable here.
  6176. 6:53:10So it's for temperature. Now I remove what we have inside our chat method
  6177. 6:53:16because we are going to replace it with the implementation
  6178. 6:53:20that we have for our structured output project.
  6179. 6:53:23Okay, so the question is how do we get structured output in LangChain?
  6180. 6:53:28And the answer is that our LLM provides a method called withStructuredOutput
  6181. 6:53:34and this method allows us to specify the schemas that the models should follow.
  6182. 6:53:40So I'm going to call this method. I say self.llm with structured output.
  6183. 6:53:46and here i need to specify two things the first
  6184. 6:53:50thing is the schema that we defined here and the
  6185. 6:53:53other one is a method okay i say method to be json
  6186. 6:53:57schema okay it's suggesting really great okay
  6187. 6:54:00so
  6188. 6:54:01we are passing two arguments here the first argument
  6189. 6:54:04is our user schema which defines the fields
  6190. 6:54:06and their expected types the second argument is json-eschema
  6191. 6:54:11this way we tell Langchain to use the json schema
  6192. 6:54:14approach for structured output now let's pass our
  6193. 6:54:18input and as i told you our input is going to be
  6194. 6:54:21a paragraph from Elon Musk's wikipedia page so
  6195. 6:54:25i'm going to store this llm with structured
  6196. 6:54:28output in a variable for example structured llm
  6197. 6:54:32then I'm going to invoke it. So I say structured LLM, invoke,
  6198. 6:54:37and I paste this paragraph here. I save the result in a variable called response,
  6199. 6:54:43and then I print the response. Print response.
  6200. 6:54:47So let's run our program and see what happens.
  6201. 6:54:51Okay, as you can see here, we have name,
  6202. 6:54:53We have age and it's really nice because the age wasn't provided here.
  6203. 6:54:57It was his birthday.
  6204. 6:54:59And as you can see, it is calculated and companies are Tesla and SpaceX. Net worth.
  6205. 6:55:06Uh, okay. Good for him.
  6206. 6:55:08And great. Our model returns extracted information in the structure we defined.
  6207. 6:55:13And instead of asking the model to return an arbitrary
  6208. 6:55:17paragraph and then to try to parse it ourselves.
  6209. 6:55:20We have specified a schema with four fields, name, age, companies, and net worth.
  6210. 6:55:25Plankchain uses the schema to request a structured output,
  6211. 6:55:28and Pydantic helps validate the resulting data against our expected types.
  6212. 6:55:33Welcome back aboard Neon Learned. Planet
  6213. 6:55:36Structured is conquered.
  6214. 6:55:37Six worlds down, four to go.
  6215. 6:55:41Fasten your seat belts. Next stop, Planet Memory.
  6216. 6:55:44Here, you'll save every conversation to a database
  6217. 6:55:47so users can start new chats and pick up old ones, just like ChatGPT.
  6218. 6:55:52Your next world awaits. Enjoy the journey.
  6219. 6:56:01Now let's move on to our next topic, next important topic, which is check pointers.
  6220. 6:56:07This is a more advanced topic, but it's also extremely
  6221. 6:56:10practical and useful in many real world applications.
  6222. 6:56:14Let's gradually build an understanding of what check pointers
  6223. 6:56:18are, why we need them and what we can use them for.
  6224. 6:56:22CheckPointers can have different use cases, although in this course their central
  6225. 6:56:27role is to save and restore an application's state, including conversation history.
  6226. 6:56:34To save time, I've already created a file called
  6227. 6:56:37checkpointers.py and I've put this code that you see
  6228. 6:56:41here.
  6229. 6:56:41There is nothing fundamentally new here.
  6230. 6:56:43Here I created an agent using create agent function that I taught you.
  6231. 6:56:49And I've created an Agent that doesn't have any tool yet.
  6232. 6:56:53So we can think of this agent as a basic conversational chatbot.
  6233. 6:56:57Let's test our program. Let's run it and see the result. So I say hello. Okay.
  6234. 6:57:03What is the capital of France?
  6235. 6:57:06What about president?
  6236. 6:57:08As you can see, it's a normal chatbot that we implemented
  6237. 6:57:12and it has history and there is really nothing new.
  6238. 6:57:15I just wanted to save your time. Okay, let's get back to our question.
  6239. 6:57:19Why do we need a check pointer?
  6240. 6:57:21When we first started building chatbots, you quickly learned
  6241. 6:57:24that LLMs don't automatically remember previous conversations.
  6242. 6:57:28If you want a model to have access to conversation history, we need to store
  6243. 6:57:33that history ourselves somewhere and send it back to the model each time.
  6244. 6:57:37And this way the model has history. Even
  6245. 6:57:40here you can see this approach.
  6246. 6:57:42You can see that we are recording this conversation history.
  6247. 6:57:46Here we have a list for messages. Here we a system message or first message.
  6248. 6:57:51And then each time the user asks something, we append that message.
  6249. 6:57:56And whenever we get the response, we append that message as well.
  6250. 6:57:59This way we are recoding messages and when we want to
  6251. 6:58:03send a new request, we pass this list of messages.
  6252. 6:58:06And this way, the model receives the previous messages as context
  6253. 6:58:10and can respond based on the conversation we have had so far.
  6254. 6:58:15But there is a problem. There is a problems with this approach.
  6255. 6:58:18Our code has become a little more complicated.
  6256. 6:58:21We have to remember to append the user's message
  6257. 6:58:24and we have to do the same for assistance response.
  6258. 6:58:27We also need to think about different edge cases.
  6259. 6:58:29For example, what happens if we encounter a connection error?
  6260. 6:58:34What if the response is empty? What if we had providers token limit?
  6261. 6:58:37So we have these edge cases.
  6262. 6:58:39And another important limitation is that our conversation
  6263. 6:58:43history is stored in a regular Python list.
  6264. 6:58:46And where does that list live? It lives in memory or RAM.
  6265. 6:58:50The thing is that as soon as the program terminates, that list disappears.
  6266. 6:58:55And if we restart the application, the previous conversation is gone.
  6267. 6:58:59So what if we want to preserve conversation history?
  6268. 6:59:02In that case, we need to store it on database,
  6269. 6:59:05instead of keeping it in a simple Python list.
  6270. 6:59:08However, that introduces additional complexity.
  6271. 6:59:11We need to save records, retrieve them from database, organize conversations,
  6272. 6:59:15handle database errors, and test all those operations.
  6273. 6:59:19We also need to distinguish between different users.
  6274. 6:59:22So, you can see that there are many things that we need handle.
  6275. 6:59:26Think about an application like ChatGPT.
  6276. 6:59:29user might have separate conversations,
  6277. 6:59:31conversations about hundreds of different topics,
  6278. 6:59:34and the user wants to switch between them.
  6279. 6:59:37And each conversation needs its own history,
  6280. 6:59:39and that history needs to be associated with the correct conversation.
  6281. 6:59:43So as you can see,
  6282. 6:59:45building all of that complexity ourselves would
  6283. 6:59:48require quite a bit of additional code.
  6284. 6:59:50and this is where check pointers in LangChain become useful.
  6285. 6:59:54Sorry, check pointers are in LangGraph. It's a feature in Lang Graph.
  6286. 6:59:58LangGraph provides a mechanism for saving and restoring graph state.
  6287. 7:00:03And we will talk about states in more detail in future.
  6288. 7:00:06And when we use an agent built with LangGraph, a check pointer can manage its state,
  6289. 7:00:12and it can do it across multiple invocations. That means,
  6290. 7:00:16we don't have to manually maintain the conversation history in a list.
  6291. 7:00:20So that's the whole idea.
  6292. 7:00:22This feature, I mean check pointer, is going to handle the state of an agent for us.
  6293. 7:00:27you
  6294. 7:00:33When working with check pointers, the first thing
  6295. 7:00:36that we need is a saver in the first place.
  6296. 7:00:38And in this session, I'm going to use In-memory saver,
  6297. 7:00:42which saves the information on memory, on your RAM.
  6298. 7:00:45And later we will switch to something else, something
  6299. 7:00:49that can persist your data in a database.
  6300. 7:00:51So for learning how check pointers work, first
  6301. 7:00:55we will use In memory saver which is simpler.
  6302. 7:00:58And later when you want to store your data on database,
  6303. 7:01:02it is as simple as changing your saver. So here I say from LangGraph
  6304. 7:01:07as I told you it's a feature in LangGraph not Langchain
  6305. 7:01:10and you don't need to install LangGraph
  6306. 7:01:12because it is automatically installed when you install Langchain
  6307. 7:01:16and then from checkpoint and from memory if I'm not wrong import in memory saver
  6308. 7:01:22as a result whenever we want to save state in memory
  6309. 7:01:26we use this option InMemorySaver and as a rule of thumb
  6310. 7:01:30When developing, always use this option.
  6311. 7:01:33And when everything works and when you want to
  6312. 7:01:36switch to production, you can use another saver.
  6313. 7:01:38And as I told you in the previous session, we no longer
  6314. 7:01:42have to manage the conversation history ourselves.
  6315. 7:01:46So I want to remove all the code we had for managing history in our code.
  6316. 7:01:51First, we had this list for recording the messages.
  6317. 7:01:54and every time we had to append the user's message and the AI's message.
  6318. 7:01:59So I don't want these two lines as well. And here we don't have this list anymore.
  6319. 7:02:05So I put three dots here and later I will show you what else we should put here.
  6320. 7:02:11But what I did raises a question.
  6321. 7:02:13If we remove the message list, what happens to our system message?
  6322. 7:02:17Previously, we were storing the system message inside that list.
  6323. 7:02:21Now we need another way to provide those instructions to agent.
  6324. 7:02:24And yes, fortunately create agent has an argument
  6325. 7:02:28for explicitly introducing these instructions.
  6326. 7:02:31In other words, it gives you, I mean, this create agent gives
  6327. 7:02:35you an option to specify a system prompt and it's an argument.
  6328. 7:02:38So I say system prompt, and I say give short answers, just like before.
  6329. 7:02:44And we need to pass our check pointer here. So I specify check pointer.
  6330. 7:02:49and I create an object from this class, okay? That's all we need.
  6331. 7:02:53And this tells the agent to use in-memory check pointer
  6332. 7:02:56to save and restore its state between invocations.
  6333. 7:03:00Now the check pointer can manage the conversation history for us
  6334. 7:03:04and it can track our conversation. It tracks the conversations through a thread ID.
  6335. 7:03:11So another thing you need to pass is thread ID,
  6336. 7:03:15The thread ID is an identifier, an identifier
  6337. 7:03:18that LangGraph uses to associate checkpointed
  6338. 7:03:21state with a particular conversation thread.
  6339. 7:03:24In other words, it tells LangGraph which conversation
  6340. 7:03:27we want to continue, and without this identifier,
  6341. 7:03:30the system wouldn't know which conversation
  6342. 7:03:33history to receive when we make another request.
  6343. 7:03:36The useful part of it is that we are no longer
  6344. 7:03:39limited to a single conversation, we can have
  6345. 7:03:42multiple conversation, multiple independent conversation histories,
  6346. 7:03:46each identified by its own thread ID. So that is the reason we need a thread ID,
  6347. 7:03:51previously we didn't need that because we only had one conversation,
  6348. 7:03:55but now we can have multiple and we need a thread id for each conversation.
  6349. 7:04:00For example, we could have one conversation in
  6350. 7:04:02which we are asking questions about friends
  6351. 7:04:05and another in which we are discussing cooking.
  6352. 7:04:09So the two conversations have separate histories.
  6353. 7:04:12And when we return to the conversation about friends,
  6354. 7:04:14we can continue from its previous context.
  6355. 7:04:18So the question is, how do we specify the thread ID?
  6356. 7:04:22The answer is that we need a configuration dictionary.
  6357. 7:04:25So first we need such a configuration
  6358. 7:04:28dictionary and then we will pass it when we are invoking the agent.
  6359. 7:04:32Before we invoke our agent,
  6360. 7:04:35I create this dictionary, I store it in a variable called config.
  6361. 7:04:40And in this dictionary we need to have a key called configurable.
  6362. 7:04:44So I say configurable, this
  6363. 7:04:47is itself is a dictionary.
  6364. 7:04:49And this dictionary has a key, called thread ID.
  6365. 7:04:53The thread ID we were talking about. And here you specify an arbitrary name.
  6366. 7:04:58So I call it session one. Okay, that's it. And then.
  6367. 7:05:03when we invoke our agent, we specify a second argument. That is config.
  6368. 7:05:09Let's say config and I set it to the config I created about. Okay,
  6369. 7:05:13that's it.
  6370. 7:05:14And what about this part? Here we don't have our messages list anymore.
  6371. 7:05:19Here I still need to have a list, but this list has only one message
  6372. 7:05:24because we are tracking the conversation history somehow else.
  6373. 7:05:27So here I add a human message.
  6374. 7:05:30this message is basically the last message that the user has sent
  6375. 7:05:35so i say user input okay
  6376. 7:05:37now i guess everything is in order
  6377. 7:05:40and the only step left is to test our program to make sure everything is
  6378. 7:05:45working okay so let's run it and let's ask a few questions
  6379. 7:05:49hello okay
  6380. 7:05:50what is the capital of france who is its president
  6381. 7:05:57So at this point, if I get the right answer, it
  6382. 7:05:59means our code works correctly because here
  6383. 7:06:01we didn't do anything about managing our
  6384. 7:06:04conversation history and it is done only through
  6385. 7:06:07the check pointer that we added here. Okay, so I enter...
  6386. 7:06:11and it answers correctly so we achieved what we
  6387. 7:06:15wanted as you can see the agent can use the context
  6388. 7:06:18from the previous question to understand what we
  6389. 7:06:21are referring to and this was the purpose of this
  6390. 7:06:24session and in the next session we will do something really
  6391. 7:06:29interesting
  6392. 7:06:34all right now let's get to the exciting part as i
  6393. 7:06:37mentioned earlier we can have multiple sessions each
  6394. 7:06:41with its own conversation history and now we
  6395. 7:06:43are going to implement that feature we are
  6396. 7:06:46going to Implement this feature in our command
  6397. 7:06:48line application and fortunately this is really
  6398. 7:06:51straightforward in this example we are gonna have two
  6399. 7:06:55separate sessions and whenever the user types ch
  6400. 7:06:58the application will switch to the other session so let's implement this first a
  6401. 7:07:03at the beginning of this method, I mean chat loop method,
  6402. 7:07:07we're going to have a flag and I call it active session.
  6403. 7:07:11and I set it to session1
  6404. 7:07:14now let's add a new condition after this if condition that we have here
  6405. 7:07:19when we check if the user has entered Q or not
  6406. 7:07:22after this condition I add another condition and I check if the user had entered CH
  6407. 7:07:29if the users has entered CH in that case we want to switch the session
  6408. 7:07:33so in that I want to update this active session so I say active session is equal to
  6409. 7:07:39and if the current session is session1
  6410. 7:07:42i say if the correct active session is equal to session1
  6411. 7:07:47in that case i want to switch to session2 otherwise in the case that active session
  6412. 7:07:54is equal session2 i want update our active session to session 1
  6413. 7:08:01and then i print a message i print done
  6414. 7:08:04so this way we can realize that it is switched successfully
  6415. 7:08:08And then I use the continue keyword, the continue statement.
  6416. 7:08:12And it skips the rest of our current loop and prompts the user for another input.
  6417. 7:08:19next let's update the label that we have here
  6418. 7:08:22i want to display the current active session
  6419. 7:08:25so here i use a formatted string i place a placeholder here
  6420. 7:08:29i say active session so the user can see which session they are currently in
  6421. 7:08:35and then i add here or type c let's change it to double quotations
  6422. 7:08:41it's nicer yes okay or type ch to change sessions and
  6423. 7:08:48I don't want to hard code this part. I want to use our flag or active session flag.
  6424. 7:08:53So I replace it with active session. Let's run our program.
  6425. 7:08:58Okay. I say.
  6426. 7:09:00hello we are in the session one okay now i ask what is the capital of france
  6427. 7:09:07now i want to switch now i am in session two i say what is capital of spain
  6428. 7:09:15okay, moderate. now I want to switch. now, I ask who is its president.
  6429. 7:09:21in this session, we were talking about France.
  6430. 7:09:25it says Emmanuel Macron and I switch again and
  6431. 7:09:29this time I again ask who is its president
  6432. 7:09:34and we were talking about Spain in this session
  6433. 7:09:37so it says yes Spain doesn't have president
  6434. 7:09:41prime minister okay yes we successfully implemented
  6435. 7:09:44what we wanted we wanted two separated sessions
  6436. 7:09:47with their own conversation histories and we
  6437. 7:09:50successfully implement this concept this
  6438. 7:09:54feature.
  6439. 7:09:54Ladies and gentlemen, the captain has asked me to pass along a personal message
  6440. 7:10:00from the flight deck.
  6441. 7:10:01We've now reached cruising altitude and every like you leave helps
  6442. 7:10:05us climb even higher.
  6443. 7:10:08To keep your seat reserved for every future flight, simply subscribe.
  6444. 7:10:13And the cockpit door is always open.
  6445. 7:10:15Leave a comment and the captain will read every single one.
  6446. 7:10:20Cabin crew prepare for the next lesson.
  6447. 7:10:22Nooo!
  6448. 7:10:31All right, the next step is to store our conversation
  6449. 7:10:35history in a database instead of keeping it only in RAM.
  6450. 7:10:38And as I mentioned earlier, we can use different databases for this purpose.
  6451. 7:10:43But the choice for this project is PostgreSQL.
  6452. 7:10:47And honestly, the choice depends on our application and
  6453. 7:10:50requirements, your experience and many other factors.
  6454. 7:10:53But for the sake of this project, we are going to use Postgre SQL.
  6455. 7:10:56And I've already installed PostgreSql.
  6456. 7:10:59it's running on port 5432 and installing Postgres SQL isn't part of this course.
  6457. 7:11:05So if you want to follow along, you need to install Postgres on your machine.
  6458. 7:11:10And that's not difficult at all. You can download it, install it.
  6459. 7:11:14And if you face any problem, you can ask ChatGPT or any other AI assistant for help.
  6460. 7:11:19So that's really easy. And after installation, I can test it.
  6461. 7:11:24So I say PSQL...version.
  6462. 7:11:28it says the version of PostgreSQL on my machine
  6463. 7:11:31and then I say PG is ready
  6464. 7:11:35and it says it is running on this board 5432 you can connect to it
  6465. 7:11:41you can to our database server
  6466. 7:11:43so I say psql-u postgres
  6467. 7:11:48here dash u specifies the username in my case the username is postgres
  6468. 7:11:54and then when I press enter
  6469. 7:11:56it asks for my password the password that I set during installation so I say admin
  6470. 7:12:03and I can connect to our database so everything is
  6471. 7:12:06working correctly and you can also inspect your
  6472. 7:12:09database visually using a database management tool
  6473. 7:12:12something like data grip and this way you can
  6474. 7:12:15visually observe what's going on in your database
  6475. 7:12:19that's it now that our database is ready we need
  6476. 7:12:22to install the package that allows Langchain to
  6477. 7:12:25save checkpoints in our Postgresql database so
  6478. 7:12:29So I open up my internal terminal and I say
  6479. 7:12:33pipenv install langchain checkpoint Postgres.
  6480. 7:12:38let's check if it's okay i don't have any typo no i guess i have a typo here
  6481. 7:12:44oops the problem is that here i'm saying lang chenit is lang graph
  6482. 7:12:49lang graph checkpoint postgres so lang graph now it should be okay
  6483. 7:12:56okay it is installing this package and i don't have
  6484. 7:12:59any typo and it is actually resolving this package
  6485. 7:13:02now let's take a look at the database connection string i've
  6486. 7:13:07already added to my.n file and you should have one like me
  6487. 7:13:13So this is my connection string and as you can see it has different components.
  6488. 7:13:19This Postgresql you are seeing here specifies the connection scheme.
  6489. 7:13:25This PostgreSQL admin specifies
  6490. 7:13:28the username and password and this localhost
  6491. 7:13:31states that the database service is running on
  6492. 7:13:35the same machine, the same as our application and this is a port.
  6493. 7:13:40and this is a database that we need to have on our database service
  6494. 7:13:45and as you can see i've disabled ssl mode since you are working in a local
  6495. 7:13:50development environment so we don't need this setting again
  6496. 7:13:55one point is that it's better to have another
  6497. 7:13:57database because i used this database before
  6498. 7:14:00recording this video to test things that i
  6499. 7:14:04wanted to teach you so here it's best to add
  6500. 7:14:09a new database and I call it chat history 2
  6501. 7:14:14Okay.
  6502. 7:14:15Okay, chat history 2. That's great. And I change it to chat history2.
  6503. 7:14:19Now, everything is in order.
  6504. 7:14:21Now, let's get back to our application and connect our agent to Postgres.
  6505. 7:14:26First, inside my main function, I want to read the
  6506. 7:14:30connection string from our environment variable.
  6507. 7:14:33So, I say OS environment and the name of my
  6508. 7:14:37variable is, let us copy it to avoid typo.
  6509. 7:14:42okay that's it and i store it in a variable
  6510. 7:14:46called db url db URL next instead of importing
  6511. 7:14:50in memory saver we'll import postgres saver so here i say postgres and i import
  6512. 7:14:57postgressaver that's exactly what i need
  6513. 7:15:01this Postgres saver that we imported provides a method called from connection string
  6514. 7:15:07so we're going to use it here I say PostgresSaver from connectionString
  6515. 7:15:13and I need to provide this URL
  6516. 7:15:16and then I can use it with the with statement
  6517. 7:15:20I say with and I get a check pointer
  6518. 7:15:25Now this check pointer object provides a method called setup.
  6519. 7:15:30This method creates the database tables and other structures
  6520. 7:15:34that Postgres CheckPointer needs to store checkpoint data.
  6521. 7:15:38And the point is that we need to execute it just for the first time.
  6522. 7:15:43Once necessary tables exist, we generally don't need to run it manually every time.
  6523. 7:15:48So for the next executions, you can comment it out or have a script for it for the
  6524. 7:15:53first time, although if you run it again, it's
  6525. 7:15:56normally fine and it doesn't lead to duplication.
  6526. 7:15:59now inside this with statement i want to create
  6527. 7:16:02my agent object or in other words i want
  6528. 7:16:06to move these lines here and yes that's it now the
  6529. 7:16:10only step left is to make a small change to our
  6530. 7:16:14agent class previously we created an in memory
  6531. 7:16:17saver now we no longer want to do that
  6532. 7:16:20because we have another check pointer so what i need to do is to pass
  6533. 7:16:26this check pointer that I have here to the constructor of this class.
  6534. 7:16:31And then I add it to my constructor here. Okay. Check pointer.
  6535. 7:16:36And then we can use it here.
  6536. 7:16:38So instead of in memory saver, we can have this checkpointer. Okay. That's it.
  6537. 7:16:44Everything else about conversation logic can remain
  6538. 7:16:47the same, including our use of thread ID to keep,
  6539. 7:16:50to keep different sessions separate and that's it,
  6540. 7:16:54We have now switched from storing conversation state in memory to storing
  6541. 7:16:58checkpoints in PostgreSQL. Now we need to test our program.
  6542. 7:17:03So here you can
  6543. 7:17:05find an object, which line
  6544. 7:17:11Oops, here I'm using apprentices now.
  6545. 7:17:14It's a dictionary. Okay. Okay. We run our program one more time.
  6546. 7:17:20My first question is hello.
  6547. 7:17:22What is the capital of France?
  6548. 7:17:26Okay, now I want to terminate the program.
  6549. 7:17:29I run it one more time.
  6550. 7:17:32And now if I ask about its president and it answers correctly, it
  6551. 7:17:37means that we have successfully saved the data in our database.
  6552. 7:17:42So I ask who is
  6553. 7:17:44its president?
  6554. 7:17:48Okay, it answers correctly and it means we simply
  6555. 7:17:51and successfully implemented this feature.
  6556. 7:17:55And if you want to visually see the tables, you can come here.
  6557. 7:18:00Yes, I want to see tables in public. Public schema.
  6558. 7:18:05Okay.
  6559. 7:18:07yes public and you can see that we have these
  6560. 7:18:10tables these tables are automatically made for
  6561. 7:18:14us by LangGraph itself by this packet that we installed
  6562. 7:18:19Oh
  6563. 7:18:25Okay, there is one more thing I'd like to show you.
  6564. 7:18:28What if we want to access the conversation messages ourselves?
  6565. 7:18:31So far, LangGraph has been retrieving the conversation
  6566. 7:18:34history and making it available to the model automatically.
  6567. 7:18:38But in many applications, we may need direct access to that history.
  6568. 7:18:42For example, we might want to display previous
  6569. 7:18:45messages to the user in a chat application.
  6570. 7:18:49So we need that and the good news is we don't
  6571. 7:18:52need even query the database and we can use
  6572. 7:18:55the feature that LangGraph offers to do that.
  6573. 7:18:58I mean the agent we have already created provides a
  6574. 7:19:02method called getState and if we pass it the thread ID,
  6575. 7:19:07we can get access to the conversation.
  6576. 7:19:10So let's
  6577. 7:19:11do that after this while loop when our program terminates.
  6578. 7:19:17I want to print the messages. So first I need config.
  6579. 7:19:21So let's copy.
  6580. 7:19:25config from here. Oh, we are in the middle of our while loop.
  6581. 7:19:30OK, here I copy it.
  6582. 7:19:32and after the while loop
  6583. 7:19:36I'm going to have this line
  6584. 7:19:38and then I want the messages in session one. Okay.
  6585. 7:19:42We specify the thread ID.
  6586. 7:19:44And then as I told you, the agent we created offers a method called getState.
  6587. 7:19:51and we pass our config here let's call it config history
  6588. 7:19:57and I store it in a variable called state because it is a state.
  6589. 7:20:02And then I say state.values and I want to access to messages.
  6590. 7:20:08It's a dictionary.
  6591. 7:20:11I want to print it.
  6592. 7:20:14Now let's run our program.
  6593. 7:20:17I just say hello and
  6594. 7:20:19I quit okay and as you can see all the messages from this
  6595. 7:20:24session which was hello and the last session are here we asked about friends
  6596. 7:20:29and its president and now we said hello and you can
  6597. 7:20:32see this message here okay that's it
  6598. 7:20:36Before we finish this session and this project,
  6599. 7:20:38there is a useful thing that I want to tell you.
  6600. 7:20:42In our project, we use two conversation IDs, session 1 and session 2.
  6601. 7:20:47That's perfectly fine for this educational example, for this
  6602. 7:20:51demonstration, but you know that in real world applications,
  6603. 7:20:55we often see something else.
  6604. 7:20:57For example, when you are working with an application like
  6605. 7:21:01chatgpt when you look at the url you will see
  6606. 7:21:05a long string of numbers and characters
  6607. 7:21:08this is the best practice that is used in real
  6608. 7:21:11world applications we need a longer identifier
  6609. 7:21:14and one common option to have something like
  6610. 7:21:18this is using uuid which stands for universally
  6611. 7:21:22unique identifier and generating a uid in python is pretty straightforward
  6612. 7:21:28We don't even need to install an additional package
  6613. 7:21:32because Python already provides a module called UUID.
  6614. 7:21:36Let's see that. I say import UUid.
  6615. 7:21:39Now, in order to show you at the beginning of my
  6616. 7:21:42main function, I'm going to generate a UU ID.
  6617. 7:21:45I print, I say uuid, uuid4, and I need to convert it into a string.
  6618. 7:21:53I say...
  6619. 7:21:54str built-in function and then i return here i don't
  6620. 7:21:57want the rest of my function to be executed so
  6621. 7:22:01i run my program and as you can see a uid is
  6622. 7:22:04generated so in your real world applications
  6623. 7:22:08use something like this welcome back aboard neon
  6624. 7:22:11learned planet memory is conquered seven worlds down
  6625. 7:22:15three to go
  6626. 7:22:17Fasten your seat belts. Next stop, Planet Middleware.
  6627. 7:22:21Here, you'll use middleware to keep a human in the loop
  6628. 7:22:24so your agent asks for permission before it acts.
  6629. 7:22:29Your next world awaits. Enjoy the journey.
  6630. 7:22:38Now let's move on to another very important
  6631. 7:22:41component in Langchain which is middleware.
  6632. 7:22:44We can either create our own middleware or we can use some of the
  6633. 7:22:48built-in middleware implementations that Langchain provides.
  6634. 7:22:52But before we start writing code, before we go any
  6635. 7:22:56further, let's understand why middleware exists.
  6636. 7:23:00Remember when I explained that
  6637. 7:23:02we can use create agent to build an AI agent
  6638. 7:23:05without having to deal directly with all the
  6639. 7:23:08complexity of lang graph.
  6640. 7:23:10The idea is that create agent already handles many of the complexity, many of
  6641. 7:23:15the things we need to handle when building an AI Agent.
  6642. 7:23:19But what happens when it covers most of our
  6643. 7:23:22requirements, but we still need a few additional
  6644. 7:23:25customization.
  6645. 7:23:27Middleware gives us exactly that customization.
  6646. 7:23:30It gives us a way to customize an agent's behavior
  6647. 7:23:34without having to implement everything from scratch.
  6648. 7:23:37The main question is that where does middleware fit into the agent life cycle?
  6649. 7:23:43To answer this question, we first need to understand
  6650. 7:23:46what is the life cycle of an agent?
  6651. 7:23:49Let me try to explain it by saying that so far we
  6652. 7:23:52have been using CreateAgent to build our agents.
  6653. 7:23:55As we discussed earlier, an agent typically sends messages to an LLM.
  6654. 7:23:59If the model requests a tool call, the application executes
  6655. 7:24:04that tool and sends the result back to the model.
  6656. 7:24:07And the model can then continue processing and eventually produce a final response.
  6657. 7:24:12And there are different stages in this life cycle.
  6658. 7:24:16there are different points, more accurate to say.
  6659. 7:24:19Middleware allows us to intervene at specific points
  6660. 7:24:23in this lifecycle and customize what happens.
  6661. 7:24:26For example, we can define a middleware that runs
  6662. 7:24:30before the agent starts processing a request.
  6663. 7:24:33We can also define middleware, that runs, before the model is called.
  6664. 7:24:37We can go even further by customizing the model calling process itself.
  6665. 7:24:41For example, imagine that a connection to our model fails.
  6666. 7:24:46We could implement custom logic to retry the
  6667. 7:24:48request up to five times before giving up.
  6668. 7:24:51This gives us more control over how model calls are handled.
  6669. 7:24:55We can also run middleware after the model response and
  6670. 7:25:00at any stage of this life cycle that I explained.
  6671. 7:25:04Okay, so... The key idea is that middleware gives us some hooks
  6672. 7:25:09and we can use these hooks to customize the behavior of this life cycle.
  6673. 7:25:16Let's see where middleware lives and how we can import it.
  6674. 7:25:20For example, let's say we want to take an action
  6675. 7:25:24before the life cycle of an agent starts
  6676. 7:25:27and before when the model is going to be called.
  6677. 7:25:31For these two points, we can use two middlewares, before agent and before model.
  6678. 7:25:37so to access these things we can use i mean we can
  6679. 7:25:41go to lang chain agents and middleware and when i
  6680. 7:25:45say import there are lots of middlewares that you
  6681. 7:25:48can import for example before agent before model
  6682. 7:25:51and these are two middlewares that i was going to show you and these
  6683. 7:25:55are basically decorators and decorators let us define some functions
  6684. 7:26:01It's better to say that they allow us to turn functions to something else.
  6685. 7:26:05For example, we had a decorator for converting a function into a tool.
  6686. 7:26:10And now we have these decorators to turn a function into a middleware.
  6687. 7:26:15So after this, we can define a function.
  6688. 7:26:18For example we can find a function called log before model.
  6689. 7:26:22And it takes state and runtime. Don't worry about these.
  6690. 7:26:26These are related to blank graph.
  6691. 7:26:29And the only thing that I want to tell you is that by using these decorators,
  6692. 7:26:34you can turn a regular function into a middleware.
  6693. 7:26:37Now, the next question is that how do we register it with our agent?
  6694. 7:26:42It's really simple. There is an argument here called middleware,
  6695. 7:26:46and it's a list like the list that we had for tools.
  6696. 7:26:50And you can put your middlewares here.
  6697. 7:26:52And this way we can have a customized middleware
  6698. 7:26:55I'm going to show you something really exciting.
  6699. 7:26:58Langchain provides a large number of ready-made
  6700. 7:27:01middlewares that you can add them to your application.
  6701. 7:27:05And by doing that, you can add a lot of capabilities to your app.
  6702. 7:27:09The exciting part is that you don't have to write any code.
  6703. 7:27:12You just use these middlewares.
  6704. 7:27:14To number some of these useful middlewares, I
  6705. 7:27:17can refer to a summarization middleware.
  6706. 7:27:20Remember our earlier chatbot project when we
  6707. 7:27:23discussed summarizing conversation history.
  6708. 7:27:25As a conversation gets longer, its history can become very large.
  6709. 7:27:29We had this problem and that means more tokens may be sent to the model.
  6710. 7:27:33It leads to more cost and the other thing is that we may fill our context window.
  6711. 7:27:40So that's really bad and in that project I showed you
  6712. 7:27:43a strategy to summarize your conversation history
  6713. 7:27:47We kept the most recent messages, for example, 10 most recent
  6714. 7:27:51messages, and we summarized early messages using an LLM.
  6715. 7:27:56As you saw, implementing something like that introduced additional complexity,
  6716. 7:28:02and we needed to track the conversation, decide when summarization should happen.
  6717. 7:28:07And we considered lots of other things and many other edge cases
  6718. 7:28:11that we didn't take into account.
  6719. 7:28:13However, with this middleware, I mean with summarization middleware
  6720. 7:28:17that LangChain provides, things become much more simpler.
  6721. 7:28:21We just use this middlewares and we will do that in the next session.
  6722. 7:28:25Another useful option, another useful middleware
  6723. 7:28:28is human in loop, human in the loop middleware.
  6724. 7:28:31This is for scenarios that we need human approval.
  6725. 7:28:34Imagine that you're building an AI agent that can send emails.
  6726. 7:28:39and you might want the agent to prepare an email draft
  6727. 7:28:43but you don't want it to send the email without your approval
  6728. 7:28:46so you can introduce a human approval step by using this middleware
  6729. 7:28:51and you can review the email before it proceeds
  6730. 7:28:56so this is particularly useful in general in a
  6731. 7:28:59situation that actions have real-world consequences
  6732. 7:29:03and another useful middleware is PII
  6733. 7:29:07and it stands for personally identifiable information
  6734. 7:29:12and this includes information that can identify a person
  6735. 7:29:16such as certain personal identifiers or contact details
  6736. 7:29:20imagine that an employee accidentally includes sensitive information in a prompt
  6737. 7:29:25and this information is about to be sent to an external model provider
  6738. 7:29:30we might want to detect that information
  6739. 7:29:33mask it before the request reaches the model
  6740. 7:29:36PII can help us implement this kind of behavior.
  6741. 7:29:39Another useful middleware can be model fallback.
  6742. 7:29:43Imagine that we are using Groq as our primary
  6743. 7:29:45model, but we have also configured a model from
  6744. 7:29:48another provider like OpenAI.
  6745. 7:29:51And our goal is that if the primary model
  6746. 7:29:54becomes unavailable or it fails, we may want
  6747. 7:29:57our application to try the other model.
  6748. 7:30:01Model fallback middleware can help us implement this behavior.
  6749. 7:30:06If a model fails, it automatically
  6750. 7:30:08switches to another model.
  6751. 7:30:10Now, after introducing these useful middlewares, in the next session,
  6752. 7:30:15we will see a summarization middleware in action
  6753. 7:30:18and you will learn how to use a middleware
  6754. 7:30:21in your Langchain application.
  6755. 7:30:31all right as i told you in this session we are
  6756. 7:30:34going to use a built-in middleware called
  6757. 7:30:36summarization middleware first we need to import it so i say from lang chain
  6758. 7:30:42agents middleware import summarization
  6759. 7:30:45middleware
  6760. 7:30:47this is a middleware that we are going to use and this is inherently a class
  6761. 7:30:51so we need to create an instance
  6762. 7:30:55and the question is that where should we put this middleware
  6763. 7:30:58we should put it inside this middle-ware argument list that I showed you
  6764. 7:31:02so this is the list and I put it here
  6765. 7:31:06okay I create an Instance
  6766. 7:31:08i need to pass some arguments the first argument that we need to
  6767. 7:31:12pass is the model the model that we are going
  6768. 7:31:15to use to summarize our messages so here i say
  6769. 7:31:19model self.llm okay and the next two arguments are
  6770. 7:31:24max tokens before summary and messages to keep
  6771. 7:31:29and these two arguments specify how many
  6772. 7:31:33messages we want to keep unchanged and what
  6773. 7:31:37is
  6774. 7:31:37are threshold before summarization after how many tokens we want to summarize.
  6775. 7:31:43But these two arguments are deprecated. Let me show you.
  6776. 7:31:47Max tokens before summary and messages to keep.
  6777. 7:31:53These two arguments recently deprecate it and honestly before recording this session
  6778. 7:31:58I realized that.
  6779. 7:31:59And now the new way to introduce these two values are trigger and keep.
  6780. 7:32:06I say trigger and I set it to a top hole. The first element is tokens.
  6781. 7:32:12I set to 500. And honestly, 500 is a very small value
  6782. 7:32:17because in real world applications, we want a much higher threshold.
  6783. 7:32:21But to make our test quick, to trigger summarization quickly and see the result,
  6784. 7:32:27I want to set it a very, very small, value.
  6785. 7:32:31And the next one is keep the number of messages we want to keep.
  6786. 7:32:35again it is a tuple the first element is messages
  6787. 7:32:39and the next one is the number of messages we want
  6788. 7:32:43to keep okay these are the settings that we need
  6789. 7:32:46and in order to have a different session let's set
  6790. 7:32:50active session to for example three it should be
  6791. 7:32:54okay and here again i set it to three to have
  6792. 7:32:58a clean conversation history now let's run our
  6793. 7:33:02program and test it okay here i say hello
  6794. 7:33:06What is the capital of France?
  6795. 7:33:11Who is its president?
  6796. 7:33:14and tell me a story
  6797. 7:33:20till another one.
  6798. 7:33:23Another.
  6799. 7:33:25I want to make sure that I will hit this number 500 tokens
  6800. 7:33:30Tell me a longer one about its dishes.
  6801. 7:33:36okay now let's quit and if we inspect the messages
  6802. 7:33:42I think we didn't have summarization
  6803. 7:33:47Okay, let's test it one more time. I can reduce this number.
  6804. 7:33:53I set it to just 100 and I change it to 4 and this one also to 4
  6805. 7:34:01okay let's run our program one more time
  6806. 7:34:04Hello.
  6807. 7:34:06What is the capital of France?
  6808. 7:34:10Tell me a story about France...
  6809. 7:34:15Another story.
  6810. 7:34:17again and
  6811. 7:34:18let's terminate the program now i'm sure that we have summarization
  6812. 7:34:24okay this is the point that we terminated the
  6813. 7:34:27program and as you can see it says here's
  6814. 7:34:30a summary of the conversation so we had summarization
  6815. 7:34:33successfully and we could implement summarization
  6816. 7:34:36as simple as you see with this summarization middleware welcome
  6817. 7:34:41back aboard neon learn
  6818. 7:34:44Planet Middleware is conquered, eight worlds down, two to go.
  6819. 7:34:49Fasten your seat belts. Next stop, Planet LangGraph.
  6820. 7:34:52Here, you'll build an agent with LangGraph that grades its
  6821. 7:34:56own work and keeps improving it until it scores above 7.
  6822. 7:35:00Your next world awaits. Enjoy the journey.
  6823. 7:35:10Alright?
  6824. 7:35:11Throughout this course, you have heard the name LangGraph many times.
  6825. 7:35:15I've mentioned several times the create agent function we
  6826. 7:35:18have been working with uses LangGraph behind the scenes.
  6827. 7:35:22And starting from this point, we are going to take a closer look at LangGraph.
  6828. 7:35:27So the question is, what exactly is LangGraph? The name gives a pretty good hint.
  6829. 7:35:32Instead of building just a chain, we're going to build a graph.
  6830. 7:35:36And the question again is, What are the fundamental building blocks of a graph?
  6831. 7:35:41And the answer is nodes and edges We have a set of nodes
  6832. 7:35:45and these nodes are connected to each other through edges
  6833. 7:35:49Inside LangGraph nodes define actions Actions we want to perform
  6834. 7:35:54For example, a node might make an LLM call
  6835. 7:35:57run a query against a database or execute a Python function
  6836. 7:36:01And what about edges?
  6837. 7:36:03Edges define where we go next In other words
  6838. 7:36:06After an action finishes, an edge determines which node should execute next.
  6839. 7:36:11Okay, and there is another important component, state.
  6840. 7:36:16State is the data that flows through our graph and it is available to its nodes.
  6841. 7:36:21For example, if you have three nodes, all
  6842. 7:36:24three nodes can access the same graph state
  6843. 7:36:27and they can read the data in that state and they also return updates to it.
  6844. 7:36:33So in other words, nodes don't just read state. They can also modify it.
  6845. 7:36:38Modify it by returning updates to its fields.
  6846. 7:36:41Now that we understand these three important components,
  6847. 7:36:45I mean nodes, edges and state, let's see how they work in code.
  6848. 7:36:50Okay, so in our project, I add a new file called graph.py.
  6849. 7:36:56OK, and we want to write our code here.
  6850. 7:36:59As I mentioned, we have three fundamental building blocks. Nodes, edges, and state.
  6851. 7:37:04And when you are building a graph in the first
  6852. 7:37:07place, you need to define the shape of your state.
  6853. 7:37:11For that let's consider an example.
  6854. 7:37:13Suppose I want to use AI to write an essay.
  6855. 7:37:17I'll ask LLM number 1 to write the essay. Then I'll send that essay to LLM number 2.
  6856. 7:37:23And the LLM number 2 will evaluate it and give it a score. A score from 1 to 10.
  6857. 7:37:30And if this score is greater than 7, we are done.
  6858. 7:37:34otherwise we will send the essay back to the first LLM number one
  6859. 7:37:39and ask it to improve it and this way we have a graph
  6860. 7:37:42so in this example that is the workflow we are going to build
  6861. 7:37:46now the question is what information do we need to keep track of
  6862. 7:37:50first we need the essays topic the topic that we want to write about
  6863. 7:37:54next we want the content it is a string and initially this field is empty
  6864. 7:38:00and the first llm will fill it
  6865. 7:38:03Finally, we need score, which will hold the
  6866. 7:38:05valuation score and we will use a floating point
  6867. 7:38:08number and initially it is also empty.
  6868. 7:38:11So we need a data structure that can hold these fields, these key value pairs.
  6869. 7:38:17And when you hear key value pair, you can think of a Python dictionary.
  6870. 7:38:21Yes, we could use a regular Python dictionary, the point is that however,
  6871. 7:38:26a regular dictionary doesn't explicitly describe the expected types of its value
  6872. 7:38:32so it's a better option here to use type dict type
  6873. 7:38:36dict is available in python
  6874. 7:38:38and you don't need to install an additional package
  6875. 7:38:40and it lets you define the expected keys and values like a dictionary
  6876. 7:38:45and the ultimate result will be dictionary when you want to create an object of it
  6877. 7:38:50so let's do that I say from typing import typed dict.
  6878. 7:38:56OK. And now we need to create a class. I name it Essay.
  6879. 7:39:02And this class should inherit from this typed dict class.
  6880. 7:39:07something like Pydantic that we had and as I told
  6881. 7:39:11you we have three fields topic content and score
  6882. 7:39:15topic is string and we need to specify that content
  6883. 7:39:19is again string but it can also be none and at
  6884. 7:39:23first
  6885. 7:39:24it is none because when we execute our graph it
  6886. 7:39:27doesn't have any value and the same to a score
  6887. 7:39:30and as i told you it's a class and we can create an
  6888. 7:39:34instance of it okay so i say essay and i need to
  6889. 7:39:38specify these keys topic content and score for example topic can be this is a topic
  6890. 7:39:46Okay, content, it is a string. This is content.
  6891. 7:39:51And we have a score, which is a floating point number.
  6892. 7:39:53Oh, I set it to a string, I say float. Okay, 2.2 can be good.
  6893. 7:40:00Okay, and the interesting part is that, as I told you, the
  6894. 7:40:05resulting object is still an ordinary Python dictionary.
  6895. 7:40:09So let me show you. I store it in a variable called SA, and then I print it.
  6896. 7:40:15I print this essay, to make sure, I also print its type.
  6897. 7:40:23and if I run my program as you can see it's
  6898. 7:40:25a regular dictionary and this is a regular
  6899. 7:40:28dictionary that you're seeing here okay great
  6900. 7:40:31so I'll remove these print statements and our
  6901. 7:40:35first step is complete we have defined the
  6902. 7:40:38state and you need to pay attention here the
  6903. 7:40:41essay class you're saying here describes the structure of our state
  6904. 7:40:46while this essay you're saying here is a dictionary
  6905. 7:40:50instance and it contains the actual data
  6906. 7:40:53so it is what that is going to flow in our graph
  6907. 7:40:57it is only the blueprint kind of now in
  6908. 7:41:00the next step we need to create our nodes
  6909. 7:41:04remember a node represents an action something we want to do as part of our workflow
  6910. 7:41:09and what's a natural way to represent an action in python a function
  6911. 7:41:14So we'll start by defining two functions for two nodes.
  6912. 7:41:18One function to write the essay and another to score it.
  6913. 7:41:23So we are going to have two functions. The first function is write essay.
  6914. 7:41:29SA and its type is SA.
  6915. 7:41:31So it refers to its structure, its shape, and it is the name
  6916. 7:41:36of the parameter that this function is going to receive.
  6917. 7:41:40And for now, we are going to use simple dummy implementation.
  6918. 7:41:45Later, after learning the core concepts of langGraph, we
  6919. 7:41:49will replace these dummy functions with real LLM calls.
  6920. 7:41:53So here's my first function.
  6921. 7:41:55As I told you, this function takes the current essay state
  6922. 7:41:59as input and returns an update for the content field.
  6923. 7:42:02So here I say return. I want to update content.
  6924. 7:42:08and I say, for example, the essay is about the topic.
  6925. 7:42:13placeholder here to dynamically inject this field here.
  6926. 7:42:18Say SA, which is a dictionary and I want its topic. Okay,
  6927. 7:42:23pretty good.
  6928. 7:42:24Now let's define the other function, which is a scoreSA.
  6929. 7:42:30we only want to update the score of this state
  6930. 7:42:34okay so I set it to for example 10
  6931. 7:42:38Here's the important point you're seeing here.
  6932. 7:42:41A node only needs to return the fields it wants to update.
  6933. 7:42:45So here we are not returning content or topic
  6934. 7:42:48because we only want to update the score.
  6935. 7:42:52And that's the point about LangGraph.
  6936. 7:42:55Now we have our two node functions and we have already defined our state.
  6937. 7:43:01Now at this point, we are ready to start building the graph.
  6938. 7:43:04So first, let's import state graph from blank chain. Here I say from land graph.
  6939. 7:43:10Graph import.
  6940. 7:43:12state graph okay we need
  6941. 7:43:14the state graph and it allows us to define a graph around a particular
  6942. 7:43:20state schema and we'll pass our essay class to its
  6943. 7:43:24constructor and then it gives us a builder so
  6944. 7:43:27let's do that
  6945. 7:43:29here i use this state graph
  6946. 7:43:31and i pass my schema the shape of our state and it gives me a builder object
  6947. 7:43:38okay and now we can use this builder object to
  6948. 7:43:42register our nodes define the edges and configure
  6949. 7:43:45the graphs workflow and once we are finished once we
  6950. 7:43:49have finished defining the graph we can compile it
  6951. 7:43:53so first let's add our nodes so i say builder i use the add node method
  6952. 7:43:59And I add WriteSA as my first node and ScoreSA as a second node.
  6953. 7:44:06and the order doesn't matter here
  6954. 7:44:08notice that we are passing the functions themselves not calling them here
  6955. 7:44:12i don't put parentheses
  6956. 7:44:15pay attention here and when we add a function this way
  6957. 7:44:19lang graph uses the functions name as a nodes name
  6958. 7:44:22so our nodes will be called writeSA and scoreSA if you wanted a different name
  6959. 7:44:28you could pass a first argument here defining the name that you preferred okay
  6960. 7:44:34now it's time to define the edges for our initial workflow
  6961. 7:44:39we want the graph to execute these steps in order start write
  6962. 7:44:44SA score SA okay and the start node is a special
  6963. 7:44:49marker that represents where execution begins and
  6964. 7:44:53we don't need to implement it ourselves we just
  6965. 7:44:56import it so here in addition to state graph i
  6966. 7:44:59import start node
  6967. 7:45:01and it can play the role of starting out for us so here after having
  6968. 7:45:07these nodes i can define edges and we do it using the add edge
  6969. 7:45:12method so here i say builder.add edge and then you will specify the
  6970. 7:45:18nodes that you want to connect to each other so i want to
  6971. 7:45:22connect my start node to the write-say node and pay attention you should
  6972. 7:45:28put the name of your node inside quotations. So I say writeSA.
  6973. 7:45:33And then we need an edge between writeSA and scoreSA. So let me duplicate this line.
  6974. 7:45:39Here I say WriteSA and here I say scoreSA
  6975. 7:45:45okay that's it and these are my edges and as i told
  6976. 7:45:49you the final step is to compile it to compile
  6977. 7:45:53the graph that we shaped with this builder object so i say builder.compile and i
  6978. 7:46:00store the result in a variable called graph and this graph object that we have here
  6979. 7:46:06has a runnable interface it's a runable object so we can call the invoke method
  6980. 7:46:13I say graph.invoke
  6981. 7:46:16and then we pass an object and we specify the
  6982. 7:46:20values of our state so here i say topic topic is
  6983. 7:46:25a
  6984. 7:46:25field that should be specified by us so i say programming but
  6985. 7:46:30content and score are are valued by these two functions
  6986. 7:46:35So the initial value for them is none. Say content none.
  6987. 7:46:41and score none as well okay and
  6988. 7:46:44I don't need this line it was just for demonstration
  6989. 7:46:49And now it returns a state.
  6990. 7:46:52I store it in a variable called state and then I print it.
  6991. 7:46:56Okay, notice that here we are passing a state, a regular Python dictionary.
  6992. 7:47:01And I told you that we could also construct an essay using, I mean, by
  6993. 7:47:07creating an object of this class, but the result is not different.
  6994. 7:47:12So here I'm passing an s state and as this state goes through
  6995. 7:47:17this graph, these fields are updated and then I will get a
  6996. 7:47:21final state and then, I'm printing this final state.
  6997. 7:47:25OK, let's execute our program.
  6998. 7:47:29And as you can see, these fields are updated.
  6999. 7:47:32And this was the core idea of building a graph in LangGraph.
  7000. 7:47:38Ladies and gentlemen, this is a message from your captain.
  7001. 7:47:43We've now reached planet LangGraph
  7002. 7:47:45and your captain wants you to know he's truly proud of you.
  7003. 7:47:48You've stayed with us all the way here and you never gave up.
  7004. 7:47:52This journey has been made many times, but very
  7005. 7:47:55few passengers have ever made it this far.
  7006. 7:47:59Your captain is certain you'll reach your destination.
  7007. 7:48:01You'll become an AI engineer and land a great job at a great company.
  7008. 7:48:07Please have your passports ready. Your AI Engineering Visa is about to be stamped.
  7009. 7:48:21One of the most important capabilities of AI agents is decision making.
  7010. 7:48:26An AI agent needs to decide what action to take next.
  7011. 7:48:30But what exactly do we mean by decision making?
  7012. 7:48:33For example, if condition A is true, go to
  7013. 7:48:36one node, otherwise go to a different node.
  7014. 7:48:39This is essentially the same if-else logic we already have from regular programming.
  7015. 7:48:44And in LangGraph, we can implement this behavior using conditional edges.
  7016. 7:48:49for this session let's start with a very simple example
  7017. 7:48:53first here i'm going to remove the previous
  7018. 7:48:56code from this file we want to have an example
  7019. 7:48:59but before explaining the example let's import
  7020. 7:49:02things that we need for example first let's
  7021. 7:49:05important our state graph so i say we want
  7022. 7:49:08to build a graph we say lang graph graph import state graph
  7023. 7:49:13And what does the state graph expect as input? It expects the schema as you saw.
  7024. 7:49:18And in previous example, the state represented an essay
  7025. 7:49:22and it contained fields such as topic, content, and score.
  7026. 7:49:27But in this example, we are going to keep things much simpler.
  7027. 7:49:31Our state will be a simple integer number. So we are gonna have something like this.
  7028. 7:49:36I'm going to say state graph and its state is just a regular integer. That's it.
  7029. 7:49:43And I store it in a variable called builder and
  7030. 7:49:46another very common name is also workflow.
  7031. 7:49:50So here in this example, I use Workflow.
  7032. 7:49:54as i told you each node represents an action we are going to create two nodes we are
  7033. 7:49:59going to have two actions the first node will
  7034. 7:50:03multiply the current state by two the second node
  7035. 7:50:06will add one to the current the state so let's
  7036. 7:50:09implement that here i have these two functions
  7037. 7:50:13function add it receives state and we can specify type hint or not okay
  7038. 7:50:20and it adds one to the current state
  7039. 7:50:24as it's a regular integer we don't need these curly braces
  7040. 7:50:28and we have the multiply function
  7041. 7:50:30okay and it's going to multiply the current state by 2.
  7042. 7:50:35that's it now let's decide how our graph should behave
  7043. 7:50:39this is the behavior that i expect we have a starting point we have
  7044. 7:50:43the multiplication node and then the add node and we
  7045. 7:50:48have a conditional decision and our condition is that
  7046. 7:50:52If the current state is greater than 10, we are done.
  7047. 7:50:55Otherwise, we want to go back to the multiply node.
  7048. 7:50:59So when we start, we multiply the state by two,
  7049. 7:51:03we add one, and then we check if the current
  7050. 7:51:07state is, greater than ten, we terminate.
  7051. 7:51:10And if it's not greater than, ten, We want to jump back to, to the multiply node
  7052. 7:51:16and now the important question is how do we implement this conditional decision
  7053. 7:51:21just like our other nodes we can define a regular python function here
  7054. 7:51:25so let's define another function called should continue
  7055. 7:51:31receives the state and the function returns
  7056. 7:51:33the name of the destination node as a string
  7057. 7:51:36whenever we want to jump okay whenever we
  7058. 7:51:39want to specify the next node and we have two
  7059. 7:51:42destinations either we have the end node
  7060. 7:51:46or the multiply node and if the state is greater than 10 we want
  7061. 7:51:51to terminate we want return the end note otherwise we want
  7062. 7:51:54to go to the multiply note so here i say return end
  7063. 7:51:59if state is greater than 10 otherwise
  7064. 7:52:04We want to go to the multiply node
  7065. 7:52:07We don't have this end here. We
  7066. 7:52:09need to import it. So here I say.
  7067. 7:52:13it's like the start node that you saw in the last
  7068. 7:52:17session okay that's it and now we have defined our
  7069. 7:52:20node functions we have to find our conditional logic
  7070. 7:52:24now we can build our graph so here i can add
  7071. 7:52:27the two nodes i mean multiply and add so i say workflow add node
  7072. 7:52:33multiply okay
  7073. 7:52:34as well as add node
  7074. 7:52:37and here we need a conditional node we need
  7075. 7:52:40a conditional note after the add node but before that let's establish an edge
  7076. 7:52:45between multiply and add so i say
  7077. 7:52:49Edge.
  7078. 7:52:50multiply and add
  7079. 7:52:53okay and then we need a conditional edge and for
  7080. 7:52:56establishing conditional edges you shouldn't use
  7081. 7:52:59this add edge method you need to use its
  7082. 7:53:02specific method which is add conditional edges
  7083. 7:53:05so here i say workflow add conditional ages i want to connect add to shoot continue
  7084. 7:53:13In other words, it's saying that execute should continue after the add node
  7085. 7:53:18and use its return value to determine where execution should go next.
  7086. 7:53:24If the function returns end, the graph stops.
  7087. 7:53:27If it returns multiply, it goes back to the multiply node.
  7088. 7:53:32and the final step is to set the entry point
  7089. 7:53:35in the previous lesson we used the start node
  7090. 7:53:38but there is another approach and in this session we're going to use that approach
  7091. 7:53:43that approach is using the set entry point method
  7092. 7:53:47so I say set entrypoint
  7093. 7:53:51and I want this multiply node to be my first node this
  7094. 7:53:55is my first note
  7095. 7:53:57and now everything is in place we can compile it
  7096. 7:54:01So I say compile and then I store the result in a variable called
  7097. 7:54:08This is another common name when you compile your workflow or your graph.
  7098. 7:54:13And then you want to invoke it.
  7099. 7:54:15I invoke it and my initial state is 1 and then invoke.
  7100. 7:54:22forgotten book.
  7101. 7:54:23Okay, then I print it.
  7102. 7:54:25Okay, so here's
  7103. 7:54:26my initial state.
  7104. 7:54:28It goes through my graph and then I'll print the final result.
  7105. 7:54:33If I execute it, you can see 15 here.
  7106. 7:54:37Let's take a closer look to see why we see 15.
  7107. 7:54:42First we have one.
  7108. 7:54:44and it reaches the first node which is the multiply node
  7109. 7:54:48and it multiplies the state by 2 so we will have 2
  7110. 7:54:53the next node which the add node adds 1 to it so it becomes 3
  7111. 7:54:59then we have the conditional node is 3 greater than 10?
  7112. 7:55:04it's not so we go back to our multiply node so it is multiplied by 2
  7113. 7:55:10it becomes 6
  7114. 7:55:12and the next node it becomes 7 and we have this conditional node
  7115. 7:55:17is 7 greater than 10? no it's not so it returns to multiply node
  7116. 7:55:22so 7 is multiplied by 2 it becomes 14 and then 1 is added to it and it becomes 15
  7117. 7:55:31now when we go to this node to this conditional note 15 is greater than ten
  7118. 7:55:38So it is over and 15 is printed on the terminal.
  7119. 7:55:43you
  7120. 7:55:49Okay, now that you have learned all the fundamental
  7121. 7:55:52concepts we need to build the graph, we are
  7122. 7:55:54ready to move on to a more realistic project,
  7123. 7:55:57which is using LLMs to write an essay, evaluate
  7124. 7:56:01it, and revise it based on feedback.
  7125. 7:56:04And this is going to be a really useful project
  7126. 7:56:07because it brings together many of the concepts
  7127. 7:56:10we have covered throughout the course.
  7128. 7:56:12First, let's load our environment variables
  7129. 7:56:15because we are going to have an LLM call.
  7130. 7:56:18We are going contact with the provider. So import.env.
  7131. 7:56:23and then i load my environmental variables using the dot env method so i say dot env
  7132. 7:56:29load dot env and this way i have my environment
  7133. 7:56:33variables now let's import type dict so i
  7134. 7:56:37say from typing import typed dict and our
  7135. 7:56:41state is going to contain several keys we have
  7136. 7:56:46a topic we have a draft and we have score and we also have a feedback because the
  7137. 7:56:51The first LLM needs to have a feedback to improve itself.
  7138. 7:56:56Then we need our provider and we are going to use the Langchain integration.
  7139. 7:57:00So I say from Langchain Groq import ChatGroq.
  7140. 7:57:06And finally, we need to import things that are necessary to build our graph.
  7141. 7:57:10So I say from langgraph.graph import end start and stateGraph
  7142. 7:57:19We will use StateGraph to build a graph start to define the
  7143. 7:57:23starting point and end to stop the workflow when we are done.
  7144. 7:57:27And then we need define the shape of our graph, the shape our state.
  7145. 7:57:32So I create a class called content state and it inherits from type dict
  7146. 7:57:39And then we have our fields. We have topic, which is a string.
  7147. 7:57:43And we have draft, which is a String again. And we've score, it is an integer.
  7148. 7:57:50And we our feedback, it is a String again. And we have revision count.
  7149. 7:57:56Revision count.
  7150. 7:57:59which is a integer number. So first here we have topic.
  7151. 7:58:04This is a topic we want the essay to cover and we
  7152. 7:58:07will specify it ourselves when we run the graph.
  7153. 7:58:10For example, we might ask for an essay about
  7154. 7:58:13health or explain why sleep matters for learning.
  7155. 7:58:16Next we have our draft and this field contains the most
  7156. 7:58:20recent version of the essay written by our first LLM.
  7157. 7:58:24Then here we had a score.
  7158. 7:58:27integer number representing the score assigned by our second LLM.
  7159. 7:58:31And we also need a feedback. This field contains the reviewer's feedback,
  7160. 7:58:37the second LLLM's feedback explaining what needs to be improved.
  7161. 7:58:41For example, the review might say the writing is unclear, or it needs more evidence,
  7162. 7:58:48or it need more examples and things like this.
  7163. 7:58:51And the first LL will use this feedback to improve the next version of the essay.
  7164. 7:58:56And finally, we have Revision Count.
  7165. 7:58:59This tracks how many times the essay has been revised.
  7166. 7:59:03And this field is especially important because we need to
  7167. 7:59:06prevent our graph from getting stuck in an infinite loop.
  7168. 7:59:10Imagine that our goal is to achieve a score greater than, for example, 9.
  7169. 7:59:15And the essay might never reach that score or
  7170. 7:59:18it might take many iterations to get there.
  7171. 7:59:21and without having stopping condition, our graph could keep running indefinitely.
  7172. 7:59:27We will solve this problem by adding a guard and for that guard, we need this flag.
  7173. 7:59:32Then we need to initialize our LLM. So let's borrow some code from here.
  7174. 7:59:38Yeah, that's exactly what we need.
  7175. 7:59:41Okay.
  7176. 7:59:42So here I'm going to define my LLM. It is a global variable here and I'm
  7177. 7:59:47going to use this model, but a higher temperature is better. And to clarify, I'm
  7178. 7:59:53going to the same model for both writing and reviewing the essay.
  7179. 7:59:58But you could have two LLMs.
  7180. 8:00:00This way it can be your first LL and it could be your second LL.
  7181. 8:00:05And you can use another model. Okay. It is as simple as that.
  7182. 8:00:09So it's not a big deal.
  7183. 8:00:11And honestly, in a real-world workflow, using different models can be better
  7184. 8:00:15because a model might be biased toward its own writing.
  7185. 8:00:20Now, let's identify the components of our graph.
  7186. 8:00:24We need a node that writes the essay, another node that reviews it,
  7187. 8:00:28and a conditional function that decides whether
  7188. 8:00:31the essay needs another revision or not.
  7189. 8:00:34So, let us define them. Let's say dev write draft.
  7190. 8:00:38gets a state.
  7191. 8:00:41and this way with this type hint, we specify
  7192. 8:00:44its type, which is going to be content state.
  7193. 8:00:47Okay.
  7194. 8:00:48Currently, we don't have any implementation yet
  7195. 8:00:51and I need two other functions, review draft
  7196. 8:00:54and the third one should continue
  7197. 8:00:58or should revise. Okay.
  7198. 8:00:59These ones also don't happen implementation yet.
  7199. 8:01:03Before implementing these functions, first we want to set up the graph itself. So.
  7200. 8:01:08To do that, I use state graph that I imported. I give it the shape of my state,
  7201. 8:01:14and then I store it in a variable called Builder.
  7202. 8:01:18Now let's register our nodes So I say Builder Add node
  7203. 8:01:24Okay, I say Write draft
  7204. 8:01:27And again, review draft.
  7205. 8:01:30and as i told you you can specify the name of these
  7206. 8:01:33nodes yourself for example let's do that so you can
  7207. 8:01:36see and here the name our function and the name
  7208. 8:01:40of our node is the same you could use different names
  7209. 8:01:44as well and next we need to specify our edges
  7210. 8:01:47first i'm going to connect the starting node to
  7211. 8:01:51the right draft node so i say builder add edge
  7212. 8:01:55start and this node, write draft.
  7213. 8:01:58And then after this node after write draft, we need to move to the review node.
  7214. 8:02:03So let's duplicate it. I'm going to go from write draft to review draft.
  7215. 8:02:09to a word typo let's copy paste these things after
  7216. 8:02:12the review node finishes we need to decide
  7217. 8:02:15what happens next should we stop or should we
  7218. 8:02:19return to the writing node for another revision
  7219. 8:02:22and we need handle it using a conditional edge
  7220. 8:02:26so here i say builder add conditional edges
  7221. 8:02:30and here i'm going to connect my review draft
  7222. 8:02:34and then after it i need to put this function should revise
  7223. 8:02:39this function is executed after our review draft node
  7224. 8:02:43and finally let's compile it i say builder compile
  7225. 8:02:48and I store the result in a variable called graph.
  7226. 8:02:52our graph structure is now in place we just need
  7227. 8:02:55to implement the logic inside the functions but
  7228. 8:02:58again before implementing the functions let's see how
  7229. 8:03:01we'll eventually run our graph i mean i want to
  7230. 8:03:04pass an initial state containing the topic
  7231. 8:03:07and initial values for other fields so here
  7232. 8:03:11i'm going to invoke this graph because
  7233. 8:03:13it is a runnable object we can invoke it
  7234. 8:03:17And then we provide our state here. And I think we can borrow some code from here.
  7235. 8:03:23Yes, let's do that.
  7236. 8:03:26I paste it here, but these fields needs to be inside quotations.
  7237. 8:03:32and the topic is going to be why sleep matters for learning and then
  7238. 8:03:38we need some commas between
  7239. 8:03:41these fields okay now the draft is empty at first the
  7240. 8:03:46score is uh zero we don't have any feedback yet
  7241. 8:03:51and revision count is zero
  7242. 8:03:54Okay, so this is our initial state and once the nodes are implemented,
  7243. 8:03:59the graph will execute the workflow and return the final state.
  7244. 8:04:02So these values will be updated. And then when we have the state,
  7245. 8:04:08we can print the result. I want to print the final draft,
  7246. 8:04:12the number of revisions, and the final score. So here I say print.
  7247. 8:04:18I use a formatted string
  7248. 8:04:20and I say
  7249. 8:04:21final
  7250. 8:04:22draft and then I specify the number of revisions so here I say I access this final
  7251. 8:04:30state and I want this key revision count okay
  7252. 8:04:35and I say revision one revision or several
  7253. 8:04:39revisions and then i want backslash n and let's have another print statement to make
  7254. 8:04:46our code more clear and then I want the draft itself So I say result draft.
  7255. 8:04:53And finally we want the final score. So here I use a formatted string.
  7256. 8:04:58First I go to the next line. Then I say final score, and this is our final score
  7257. 8:05:04I say, result. Okay. Final score.
  7258. 8:05:08okay so the only thing left is to implement the functions that we don't have
  7259. 8:05:14any implementation for them yet let's start with the simplest one
  7260. 8:05:19here we only need a condition our target score is
  7261. 8:05:23eight so if the score is greater than eight we can
  7262. 8:05:27terminate and if it's not we go back to to the right
  7263. 8:05:31draft node we also need another guard that i told you
  7264. 8:05:34if after three revisions we didn't reach the score
  7265. 8:05:38we want to terminate this way we will also enforce
  7266. 8:05:41a maximum of three revisions and this prevents
  7267. 8:05:45an infinite loop okay so let's have it
  7268. 8:05:48First, I add an if statement and I say if our score
  7269. 8:05:53is greater than or equal to 8 we want to terminate and i return the end node
  7270. 8:06:00OK, and another if statement and I say if our revision count.
  7271. 8:06:05is greater than three, greater than or equal to three.
  7272. 8:06:10Again, I want to return.
  7273. 8:06:13And then finally, if neither of these conditions
  7274. 8:06:16is matched, I want to return to the right draft.
  7275. 8:06:20So I say return right draft
  7276. 8:06:24Okay, so that is the implementation for this function.
  7277. 8:06:28Now let's implement review draft.
  7278. 8:06:30The job of this node is to ask the LLM to evaluate the current
  7279. 8:06:35state, assign a score from 1 to 10, and provide a feedback.
  7280. 8:06:40So let's start by providing a prompt. It is going to be my prompt.
  7281. 8:06:44I say I use a formatted string and I use triple quotation.
  7282. 8:06:49And I say rate this paragraph from one to 10 and give one sentence of feedback.
  7283. 8:06:58okay then we need to provide our paragraph so i say paragraph and
  7284. 8:07:03i extract it from my state
  7285. 8:07:06It is our draft and then we specify a guideline for it.
  7286. 8:07:11I say respond in exactly this format.
  7287. 8:07:16I specified the score.
  7288. 8:07:18It is going to be a number.
  7289. 8:07:20And then we want that one sentence feedback.
  7290. 8:07:25fixed.
  7291. 8:07:26So this is going to be our prompt.
  7292. 8:07:29We are explicitly asking the model to return its
  7293. 8:07:32answer in a simple, predictable text format.
  7294. 8:07:34We could also use structured output and we covered that in an early session.
  7295. 8:07:39And that would generally be better, a more robust way to do that.
  7296. 8:07:43But in this example, I just wanted to show you how
  7297. 8:07:47to extract information from plain text as well.
  7298. 8:07:50because you may encounter this scenario and it's good to be familiar with.
  7299. 8:07:56So next we need to send our request.
  7300. 8:08:00So I say LLM invoke and I'm going to send this prompt.
  7301. 8:08:05and I get a response.
  7302. 8:08:07and I only want its text so I say response content it's an AI message and I
  7303. 8:08:14only want it's text field
  7304. 8:08:16and I store it in a text.
  7305. 8:08:19now we need to parse it we need to extract the score and feedback from that text
  7306. 8:08:24and since we asked the model to put the score
  7307. 8:08:27and feedback on separate lines we can split the response into individual lines
  7308. 8:08:32and look for the lines that start with score colon and feedback colon
  7309. 8:08:38so that's really simple and we can do it using a list comprehension
  7310. 8:08:44So first I'm going to split this text into its lines.
  7311. 8:08:48So I'm gonna use the split method.
  7312. 8:08:52and my separator is going to be backslash n to separate lines
  7313. 8:08:57and i'm going to iterate over it so i say for line in these lines
  7314. 8:09:04and then i'm gonna put a condition i say if line start with
  7315. 8:09:11with exactly this thing. In that case, I want you to return that line.
  7316. 8:09:19and I put the whole line between brackets and
  7317. 8:09:23this way I will have a list of one element so
  7318. 8:09:27here in this list I will only have one element I
  7319. 8:09:31extract it now I have a line containing the score
  7320. 8:09:34that I'm looking for so eventually I will something like this
  7321. 8:09:39score blah blah blah
  7322. 8:09:40Okay, I store it into a variable called scoreline.
  7323. 8:09:47Okay, and I do the same thing for feedback. I say feedback line.
  7324. 8:09:53Okay, uppercase F. And now I have two separate lines.
  7325. 8:09:58At the beginning of each line, we have score and feedback.
  7326. 8:10:02So the only thing left is to get rid of these two things.
  7327. 8:10:06I mean, score and feed back.
  7328. 8:10:09and to do that we can use the replace method
  7329. 8:10:13so here first i say scoreline dot replace
  7330. 8:10:17Replace what? Replace this part.
  7331. 8:10:20with empty.
  7332. 8:10:22and then I strip it to get
  7333. 8:10:24rid of any additional space at the beginning and at the end.
  7334. 8:10:29Now I want to convert it into an integer. So I use the built-in integer function.
  7335. 8:10:35And I store the final result in a variable called score.
  7336. 8:10:38And then I'm going to do the same for feedback,
  7337. 8:10:43but I don't need to convert into integer because it's a string.
  7338. 8:10:48okay feedback line here now it is ready and finally we are going to update our state
  7339. 8:10:56we return the fields we want to update score and feedback so here i say return
  7340. 8:11:02I want to update our score field to this score
  7341. 8:11:06that we extracted from the LLM's response
  7342. 8:11:10and the same for feedback.
  7343. 8:11:13okay here we go one thing that is important and you
  7344. 8:11:17need to keep in mind is that this simple parsing
  7345. 8:11:20approach assumes that the model follows the
  7346. 8:11:23requested format exactly if it changes the labels
  7347. 8:11:26or returns an invalid score the code could fail so for this reason i told you that
  7348. 8:11:32it's better to use structured output and it is more robust but you
  7349. 8:11:37Here it is perfectly fine and we are using a relatively
  7350. 8:11:41strong model so it can handle this scenario.
  7351. 8:11:44Now the final function is writeDraft.
  7352. 8:11:46This node is responsible for generating the initial
  7353. 8:11:50essay, also improving it whenever we have a feedback.
  7354. 8:11:54So let's begin with a simple prompt.
  7355. 8:11:57They say,
  7356. 8:11:59write a short paragraph about instead of saying essay
  7357. 8:12:05I'm saying paragraph because I don't want it to be long
  7358. 8:12:09and then we need to provide the topic so I say state topic
  7359. 8:12:15okay I want a paragraph about this topic and this prompt tells the LLM what topic
  7360. 8:12:21what topic to write about but what happens when we are revising an existing draft
  7361. 8:12:27in that case we need to give the model the reviewers feedback. So we need check.
  7362. 8:12:33We need to check if we have a feedback. We want to add it to our original prompt.
  7363. 8:12:37So here I add a condition and I say in case we have feedback.
  7364. 8:12:44we want to add something to our prompt so i say
  7365. 8:12:48prompt plus equal i want to concatenate this new
  7366. 8:12:51string to it i say first i put
  7367. 8:12:56I put double backslash and I say previous feedback to address.
  7368. 8:13:02and then I provided, I say state feedback.
  7369. 8:13:07okay that's all we need if feedback exists we
  7370. 8:13:10append it to the prompt and this gives the model
  7371. 8:13:13additional guideline when generating the next version now let's send the prompt
  7372. 8:13:18and get the response
  7373. 8:13:20Okay, like here.
  7374. 8:13:22And finally, we
  7375. 8:13:24need to update two things. The draft and the revision count field.
  7376. 8:13:29So I say return.
  7377. 8:13:32draft and it is LLM response
  7378. 8:13:36And then we have Revision count.
  7379. 8:13:40Okay, say...
  7380. 8:13:43Revision count plus one. I want to add one to it.
  7381. 8:13:48it was the final function that we implemented now
  7382. 8:13:51every time this node runs the counter increases by
  7383. 8:13:55one and if you have a new draft the draft will
  7384. 8:13:58be updated so let's put everything together and
  7385. 8:14:02run our graph okay let's run it
  7386. 8:14:06Okay, what's the problem? Oh, here we have a type I guess.
  7387. 8:14:11starts with
  7388. 8:14:14Okay, one more time.
  7389. 8:14:18okay everything is working correctly after just one
  7390. 8:14:22revision we have this essay and as you can see
  7391. 8:14:26it is biased and in the first try it got the score score nine so
  7392. 8:14:32let's make it harder i change this number to nine and one more time
  7393. 8:14:37Maybe the next time it gets 8.
  7394. 8:14:40Okay, again after the first time it got 9.
  7395. 8:14:44So let me change the prompt a little bit. I say be very strict.
  7396. 8:14:52Okay, one more time.
  7397. 8:14:54okay this time after two revisions we have this
  7398. 8:14:57essay and this is our final score finally made it
  7399. 8:15:01to get a score more than or equal to nine let's
  7400. 8:15:04change this condition to only show you that we
  7401. 8:15:07won't have an infinite loop i'm going to say
  7402. 8:15:11if it is greater than 11 so it will never
  7403. 8:15:14happen because we are telling model that it
  7404. 8:15:17needs to generate a number between 1 and 10. so let's run our program
  7405. 8:15:23And you will see that after three revisions, we will have this paragraph here.
  7406. 8:15:28And that happens because we intentionally changed this
  7407. 8:15:33condition and we went out of the range that was possible.
  7408. 8:15:38Welcome back aboard Neon Learn.
  7409. 8:15:40Planet Lang graph is conquered, nine worlds down, only one remains.
  7410. 8:15:46Fasten your seat belts. Final stop, Planet Multi-Agent.
  7411. 8:15:49Here you'll build a team of specialist AI agents,
  7412. 8:15:53each one an expert, all working together as one.
  7413. 8:15:58Conquer it and become an AI engineer.
  7414. 8:16:01you
  7415. 8:16:07Alright, by now you are familiar with the concept of an AI agent.
  7416. 8:16:11An AI agent is essentially an LLM that has access to a set of tools.
  7417. 8:16:16Whenever the model needs to use one of those tools, it requests a tool call.
  7418. 8:16:21We execute tool call, provide the result back to the model,
  7419. 8:16:25and the agent continues working based on that result.
  7420. 8:16:29Now in our next project, we are going to explore multi-agent systems.
  7421. 8:16:34So what exactly does that mean?
  7422. 8:16:36Instead of having just one agent, you are going to work with multiple agents.
  7423. 8:16:41For example, imagine we have agent 1 which specialized in a
  7424. 8:16:45particular task and has access to a specific set of tools.
  7425. 8:16:49Then we have Agent 2 which specializes in another
  7426. 8:16:53task and have access to different tools.
  7427. 8:16:56Finally, we have the agent 3 which coordinates the other agents.
  7428. 8:17:01We can think of this agent as our orchestrator, receives a request, decides
  7429. 8:17:06which agent should handle it, and delegates the task accordingly.
  7430. 8:17:11Let me make this concept clearer with an example.
  7431. 8:17:15Imagine that agent 1 has a tool that can reduce the file size of an image.
  7432. 8:17:21Agent 2, on the other hand, has a tools that can improve image quality.
  7433. 8:17:26And
  7434. 8:17:27then we have a decision-making agent. and this agent can coordinate the workflow
  7435. 8:17:33and this is our orchestrator and suppose we receive something like this
  7436. 8:17:38the user asks I want to improve the quality of this image
  7437. 8:17:42in that case it routes the request to agent 2
  7438. 8:17:45because that agent specializes in improving image quality
  7439. 8:17:50Now imagine the user says, I can't upload this file because its size is too large.
  7440. 8:17:56In that case, the orchestrator agent routes a request
  7441. 8:18:00to agent 1, which can reduce the file size.
  7442. 8:18:03So what have we done here?
  7443. 8:18:05Instead of having a single agent with two tools, now we have three agents.
  7444. 8:18:10Three agents working together with two tool.
  7445. 8:18:14At this point, you might be asking yourself a reasonable question.
  7446. 8:18:18why would we use three agents with only two
  7447. 8:18:21tools instead of simply having a single agent
  7448. 8:18:24with two tools it doesn't make sense and yes you are absolutely right
  7449. 8:18:28For this simple example, a single agent with only two tools would be sufficient.
  7450. 8:18:34And I used this example simply to make the concept easier to understand for you.
  7451. 8:18:39So the question is, when do multi-agent systems actually become useful?
  7452. 8:18:43They become useful when we are dealing with a large number of tools.
  7453. 8:18:48In real-world applications, we might have 30, 40, or even 50 tools.
  7454. 8:18:53And many of these tools may have very little to do with one another.
  7455. 8:18:57and if we give all those tools to a single agent
  7456. 8:19:00and also provide it with a very long complicated prompt
  7457. 8:19:04the system can become harder to manage okay and it is less reliable
  7458. 8:19:10I mean the model might struggle to determine which tool it should call
  7459. 8:19:14sometimes it might not call a tool at all
  7460. 8:19:17in other cases it might select the wrong tool
  7461. 8:19:20or fail to follow the intended workflow
  7462. 8:19:24So the idea is that instead of giving every tool to one
  7463. 8:19:28agent, we divide the workload among specialized agents.
  7464. 8:19:32Each agent focuses on specific tasks and has
  7465. 8:19:35access to a relatively small set of tools.
  7466. 8:19:38And this makes each agent's responsibilities clearer
  7467. 8:19:42and can reduce the complexity of the overall system.
  7468. 8:19:46So the goal isn't simply to increase the number of agents.
  7469. 8:19:50The goal is to organize responsibilities.
  7470. 8:19:53And of course, multi-agent systems aren't always the best choice.
  7471. 8:19:58For simple workflows, a single agent might be a better option.
  7472. 8:20:03Now, there are two major approaches that we can implement this concept using them.
  7473. 8:20:09The first one is the supervisor approach and the second one is handoffs approach.
  7474. 8:20:16The first approach, I mean supervisor, is the approach that I discussed.
  7475. 8:20:21And the second approach is called handoffs.
  7476. 8:20:24In this approach, we don't necessarily need a separate central supervisor.
  7477. 8:20:29Instead, agents can cooperate.
  7478. 8:20:32Agents can transfer control to one another when they think it is appropriate.
  7479. 8:20:37Imagine that we have agent 1 and agent 2. Agent 1 starts handling a request, but
  7480. 8:20:44At some point, it determines that Agent 2 is a
  7481. 8:20:47better option, can better handle the situation.
  7482. 8:20:50So Agent 1 can then hand off the conversation
  7483. 8:20:54and the relevant state goes to Agent 2.
  7484. 8:20:58And Agent 2 takes over and continues working from that point.
  7485. 8:21:03So rather than having a central supervisor, we can
  7486. 8:21:07give the ability of controlling the situation
  7487. 8:21:10to the agents themselves. That's the basic idea behind handoffs.
  7488. 8:21:15And in our next project, we are going to focus on the first approach,
  7489. 8:21:19the supervisor architecture,
  7490. 8:21:21because it builds on the concepts you are already familiar with.
  7491. 8:21:26you
  7492. 8:21:32all right now let's implement a multi-agent system using the supervising pattern
  7493. 8:21:38what are we going to build we are going to
  7494. 8:21:40build a very simple project with only two specialized agents
  7495. 8:21:45one agent will be an expert on fruits and the other will be
  7496. 8:21:48an expert and vegetables the fruit expert will have access to a tool that provides
  7497. 8:21:54information about fruits while the vegetable expert will have access a tool
  7498. 8:22:00that provides information about vegetables.
  7499. 8:22:02And as you can see here, I've already prepared these two tools.
  7500. 8:22:06And as we can see, there are just two simple Python functions.
  7501. 8:22:11The first function takes a fruit name and the second one takes a vegetable name.
  7502. 8:22:16And the type hints here are telling you that the
  7503. 8:22:19input parameter is expected to be a string.
  7504. 8:22:22And the function returns a string
  7505. 8:22:25and these are as you can see simple almost dummy tools
  7506. 8:22:30our fruit tool only knows about tomatoes and apples
  7507. 8:22:34while our vegetable tool only know about carrots and potatoes
  7508. 8:22:38and that's it they don't have any additional knowledge
  7509. 8:22:42if we pass a fruit or vegetable that isn't included the function
  7510. 8:22:47returns a message saying that no specific information was found so
  7511. 8:22:52let's build our multi-agent system first
  7512. 8:22:55we need to load our environment variables because
  7513. 8:22:59we have LLM calls so here I import dot-env
  7514. 8:23:03and then I load environment variables then
  7515. 8:23:08I need to import ChatGroq so I say from Langchain Groq import ChatGroq.
  7516. 8:23:15and then we need an llm so let's copy some code from for example here
  7517. 8:23:23Yeah, that's perfectly fine.
  7518. 8:23:25Okay, so it is going to be our LLM.
  7519. 8:23:29Now let's move on to creating our sub-agents.
  7520. 8:23:33So first I import create agent from LangChain.
  7521. 8:23:38Agents, import, create agent.
  7522. 8:23:42okay we'll
  7523. 8:23:43create an agent that uses our LLM and has access to the fruit info
  7524. 8:23:48fruit info tool that you are seeing here and we will do the same for VG info
  7525. 8:23:53okay so at this point we need to create two agents
  7526. 8:23:58so I say create agent our
  7527. 8:24:00model is going to be this LL that we stored in this variable and then we need
  7528. 8:24:06to specify the tools for this agent we only have this
  7529. 8:24:12tool and
  7530. 8:24:13we should define here below these functions
  7531. 8:24:16okay great and then we need to specify a system
  7532. 8:24:20prompt and in this system prompt we are going to say
  7533. 8:24:24it is a fruit expert and we will ask it to respond
  7534. 8:24:27in one sentence okay so i say system prompt
  7535. 8:24:31you are a fruit. Expert respond in one sentence.
  7536. 8:24:38Now let's create the other agent.
  7537. 8:24:42Come on.
  7538. 8:24:43I forgot this comma. Another one.
  7539. 8:24:47for the veggie expert, so here I say
  7540. 8:24:50Oh, I didn't store it in a variable. Okay. So here I say fruit agent.
  7541. 8:24:56And this one is VEG Agent.
  7542. 8:25:00okay and this agent has access to this
  7543. 8:25:04tool
  7544. 8:25:05here we
  7545. 8:25:06should say VEGExpert pay attention when you copy paste these lines
  7546. 8:25:12that's good so at this point we have two specialized agents
  7547. 8:25:16each one has its own responsibility and access
  7548. 8:25:19to a tool relevant to that responsibility
  7549. 8:25:22now it's time to create our third agent, the coordinator
  7550. 8:25:26our main agent which acts as a supervisor
  7551. 8:25:29and let's start by defining it, we need a new agent Here I say create agent.
  7552. 8:25:36and I store it in a variable called mainAgent.
  7553. 8:25:40Then, like before, we specify the model.
  7554. 8:25:44and then we also need to specify tools and system prompt
  7555. 8:25:50But for the moment, I'm leaving this list and system prompt MPT and I will explain.
  7556. 8:25:56Okay.
  7557. 8:25:57uh well the main question is that how do we give
  7558. 8:26:01our main agent access to these two sub-agents
  7559. 8:26:05and the key is that we need to make these sub-Agents
  7560. 8:26:09available to the main agent as tools in this step
  7561. 8:26:12we don't simply put the agent objects directly into the tools list instead we wrap
  7562. 8:26:19each sub agent in a regular python function that
  7563. 8:26:22invokes the corresponding agent and returns its
  7564. 8:26:26response okay so at this step we need to wrap each
  7565. 8:26:30of these functions in a function to make it a tool
  7566. 8:26:35so first for the fruit expert i define this function i call it ask
  7567. 8:26:42Fruit Expert
  7568. 8:26:44It received a question.
  7569. 8:26:46And the type is string and it also returns a string.
  7570. 8:26:51then i put a description because i'm defining
  7571. 8:26:55a tool so here i say ask the fruit expert
  7572. 8:26:59Use for all
  7573. 8:27:01fruit questions.
  7574. 8:27:04and that's it then we need to invoke the corresponding agent
  7575. 8:27:09so i say fruit agent
  7576. 8:27:12invoke
  7577. 8:27:13and as you remember we need to provide a dictionary
  7578. 8:27:17and this dictionary has a key called messages
  7579. 8:27:20and as a value of this key we need provide a list and
  7580. 8:27:24in this list we have a user message so here i say role
  7581. 8:27:29user and content
  7582. 8:27:32question the question that this function receives
  7583. 8:27:35that's fine and let's do the same thing
  7584. 8:27:38for the vegetable expert so i just copy and
  7585. 8:27:42paste this line i say ask a veggie expert
  7586. 8:27:47and they say ask the
  7587. 8:27:50ask the veggie expert used for all questions and here instead of invoking
  7588. 8:27:55This agent I'm going to invoke VEG Agent.
  7589. 8:27:59OK, there is nothing else to change.
  7590. 8:28:05But here we forgot to store the result in a response and return it.
  7591. 8:28:11here I should return response and I need to
  7592. 8:28:14extract the last message and its content only
  7593. 8:28:18so here I say messages and then the last item content
  7594. 8:28:24And the same
  7595. 8:28:26for here.
  7596. 8:28:28okay response oh we have a typo here i guess yeah response
  7597. 8:28:34okay so we have wrapped each subagent in a python function
  7598. 8:28:39these functions can now serve as tools for our main agent
  7599. 8:28:43and when the main agent calls one of these functions
  7600. 8:28:47the function invokes the corresponding specialist and returns
  7601. 8:28:51its answer now we can update our mainagent and provide
  7602. 8:28:56these tools along with a prompt
  7603. 8:28:59so here i say ask fruit expert which is a
  7604. 8:29:03tool now and ask VEG expert. Okay.
  7605. 8:29:07And then as a system prompt,
  7606. 8:29:10I say you have two experts.
  7607. 8:29:13and we specify the name of these experts it's better although we have provided
  7608. 8:29:20these here and here okay and then i say
  7609. 8:29:24always delegate questions to
  7610. 8:29:29the appropriate expert and this
  7611. 8:29:31is our system prompt
  7612. 8:29:34so this is a core idea behind our supervisor architecture
  7613. 8:29:38the main agent receives the user's request decides which specialist to call
  7614. 8:29:43and uses the specialist's response to answer the user
  7615. 8:29:47okay so let's the entire system now I invoke this main agent
  7616. 8:29:54Then as usual, we provide a dictionary containing this key and then a list.
  7617. 8:30:00And inside this list, we have our user message.
  7618. 8:30:04And as a content
  7619. 8:30:06we specify our question, we say, is a tomato a fruit?
  7620. 8:30:12for sure inside quotations and then we store the result
  7621. 8:30:17and then we print it.
  7622. 8:30:20let's say messages, its last item and all its content
  7623. 8:30:26Now let's run our program and see what happens.
  7624. 8:30:30okay as you can see our system is working correctly
  7625. 8:30:34and it says yes botanically a tomato is a fruit
  7626. 8:30:37it develops from the ovary of a flower and contains
  7627. 8:30:40and the other information and if you pay attention
  7628. 8:30:44this answer is based on this information that we have developed here botanical fruit
  7629. 8:30:51and yes this information is augmented so it was our goal and now we have a
  7630. 8:30:58multi-agent system and we have successfully implemented a simple multi-
  7631. 8:31:02agent system using the supervisor approach Ladies
  7632. 8:31:07and gentlemen, you have arrived
  7633. 8:31:10All 10 worlds are conquered.
  7634. 8:31:15Take a moment, you've earned this.
  7635. 8:31:18You've built chatbots, smart search, rag systems,
  7636. 8:31:21AI agents, and teams of agents that work together.
  7637. 8:31:26You are now an AI engineer. Thank you for flying with Neon Learn.

About this transcript

This page contains the full transcript of AI Engineering Full Course | 8+ Hours | LLMs, RAG, LangChain, LangGraph, AI Agents by neonlearn, generated from the public captions YouTube serves with the video. The transcript has 72,203 words across 7,637 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.