AI Engineering Full Course | 8+ Hours | LLMs, RAG, LangChain, LangGraph, AI Agents — Transcript
Full transcript
- 0:00Hi everyone, welcome to the NeonLearn YouTube channel.
- 0:03This is a comprehensive AI engineering course.
- 0:06My goal is simple, I want to teach you the most
- 0:09important AI engineering concepts in just
- 0:11one course. And this course is going to be fully project oriented.
- 0:16We will build 10 real world projects together and cover a lot of concepts and tools.
- 0:22We will work with LLMs, vector databases, embedding models and libraries like
- 0:28LangChain, LangGraph, Transformers and Sentence Transformers.
- 0:32And the only prerequisite is Python.
- 0:35So we will learn everything about AI engineering from scratch.
- 0:40Ladies and gentlemen, welcome aboard Neon Learn.
- 0:44Before we take off, your captain would like to walk you through the journey ahead.
- 0:4910 worlds, 10 missions, 1 destination.
- 0:52Please give your full attention to the captain.
- 0:55As I said, this course includes 10 projects.
- 0:59In our first project, we will build a chatbot from scratch.
- 1:02We will add memory to our chatbot, stream responses token by token,
- 1:06and summarize a conversation every 10 messages to save tokens and reduce costs.
- 1:12By the end of this project, you will know how to build
- 1:16an AI assistant for your own website or business.
- 1:19In project 2, we will build an AI system that can classify text.
- 1:23For example, it can detect if a customer is happy or unhappy with a product.
- 1:28This is called sentiment analysis.
- 1:30It can also classify articles into topics like health, economics, or technology.
- 1:36And we will do this without using an LLM.
- 1:39Instead, we'll use Natural Language Inference, or NLI.
- 1:44In project 3, we learn about embeddings.
- 1:48Embeddings are one of the most important building blocks of AI engineering
- 1:52and we will use them to build a semantic search system.
- 1:56In project 4 we will build a RAG system.
- 1:59RAG stands for Retrieval Augmented Generation.
- 2:02It is one of most valuable skills in AI engineering
- 2:06and I'm sure you have heard about it a lot.
- 2:09In project 5, we will build an AI agent.
- 2:12We will give our agent a set of tools and it can use them whenever it needs to.
- 2:17In project 6, we'll learn more about LangChain.
- 2:21First, we explore Prompt Template, which can help us manage our prompts.
- 2:25Then, we learn about Structured Outhood, which
- 2:28help us control the output of a model.
- 2:31We could give it a Wikipedia article about a person.
- 2:35Then, the model extracts details like
- 2:37their name, age, wealth, and other information.
- 2:41It returns everything in the exact format our application needs.
- 2:45In project 7, we will take memory management to the next level.
- 2:49We will use CheckPointer to save conversations in a database.
- 2:53And this means each user can start new conversations with a chatbot
- 2:59and return to old ones whenever they want.
- 3:02In project 8, you will learn about middleware.
- 3:05Have you ever used an AI assistant that asks for
- 3:07your permission before taking certain actions?
- 3:10In this project, you will learn how to build this kind of feature using middleware.
- 3:15It gives you more control over how your AI applications work.
- 3:21In project 9, we will explore Lang Graph.
- 3:24I will show you how to build an AI agent that improves its own work using a graph.
- 3:28Here's how it works. First, an LLM writes an article about a topic we provide.
- 3:34Then another LLm reviews article and gives it a score from 1 to 10.
- 3:40If the score is below 7, the article goes back to the first LL.
- 3:45And this process continues until the article gets a high score.
- 3:50Finally, in project 10, we will explore multi-agent systems.
- 3:54Instead of using just one agent, we'll use several agents.
- 3:58Each agent will specialize in a specific task.
- 4:01Together they can work on more complex problems.
- 4:0810 worlds of AI engineering lie ahead. Your journey begins at the launch base.
- 4:13Fasten your seat belt. Next stop, play a chatbot.
- 4:16Here you'll build your own AI chatbot from scratch
- 4:19with memory, real-time streaming, and smart summarization.
- 4:24Your first world awaits. Enjoy the journey.
- 4:32In this session, we're going to set up the Python
- 4:34environment that we need for this project.
- 4:37If you already know how to do this, I mean, how to have a virtual Python
- 4:41environment, write Python code and install third party packages.
- 4:45You can skip the session or at least fast forward this session. Okay.
- 4:50For writing Python code in this project, I'm
- 4:53going to use VS Code, Visual Studio Code.
- 4:55And if you don't have this code editor, you need to go to code.visualstudio.com
- 5:01and install the version that is compatible with your machine.
- 5:05And this is really straightforward.
- 5:08We don't need to follow this step together because it would be a waste of your time.
- 5:13And in addition, you'll need the Python interpreter as well.
- 5:17You need to get Python interpreter installed on your machine
- 5:20For doing that, head over to python.org
- 5:23Here again you can find a version that is suitable for your machine.
- 5:27Following this step to make sure that everything is right,
- 5:31I open up a terminal and I type Python 3.
- 5:35And if you are on Windows, just type Python. And I press Enter.
- 5:39And this way I can make sure that Python is correctly installed on my machine.
- 5:44The next step is to create our exciting project.
- 5:48And to do that, there are quite different ways.
- 5:52My favorite one is to create a folder first.
- 5:55I give it a name like AI engineering for the sake of this project.
- 6:01Then I open a terminal in this folder and I type code space period and this way.
- 6:08As you can see, this folder is successfully opened in my
- 6:12VS Code, which is really simple, really straightforward.
- 6:16But you could also first open your Visual Studio
- 6:20Code and go to File tab and open folder.
- 6:23And you could do that as well, but it's my preference.
- 6:26You could choose whatever you want.
- 6:28Then here I add a new file called chatbot.py.
- 6:34And this file is going to contain the code for
- 6:36our first exciting project, which is designing
- 6:39a chatbot.
- 6:40The next step is to install some extensions in this extension panel.
- 6:45And these extensions will turn our VS Code, which
- 6:49is a code editor, into a very powerful IDE.
- 6:52The first and the most important extension that we need to install is Python.
- 6:59which is created by Microsoft itself. So we need to install this extension.
- 7:05As you can see, I've already installed.
- 7:06And the point is that this extension uses PyLint as its linter.
- 7:11So you need to have PyLind as well. And to do that, we need install it in terminal.
- 7:17And it's easier for me to open up the VS Code
- 7:21internal terminal by pressing Ctrl and back tick.
- 7:26I need to execute this command, python3
- 7:30or if you are on Windows, again say python and then "-m pip
- 7:36install dash u pilot okay
- 7:38I've already
- 7:39installed pilot so I don't need to execute
- 7:43this comment but you have to if if you haven't for the next step we need to
- 7:48install a formatter which I use AutoPep8 and this formatter will format your code
- 7:54according to pep8 standard which is a popular preference in many projects and I
- 8:00After installing this formatter, whenever you want to format your Python code,
- 8:06open up your control panel by pressing Command-Shift-NP and type Format Document.
- 8:12It's tedious when you want it and it happens a lot when we want to repeat this step.
- 8:18So in order to avoid this step, I go to CodeTab,
- 8:22Preferences, Settings, and here I type Formats
- 8:26and Save.
- 8:28Okay, I ticked.
- 8:29this item and by doing that whenever I press ctrl and s the document is
- 8:35formatted for me automatically which is really simple and the next extension
- 8:42that we need is code runner which makes it
- 8:45easier for us when we want to run our code
- 8:47and after installing this extension we need to apply an important setting
- 8:53because in our project we want get an input from the user
- 8:58but the point is that Coderunner runs your code in Output tab
- 9:02which you cannot enter any input
- 9:05so to change this behavior again I go to settings
- 9:11and I search Coderaunner terminal you need to tick this item
- 9:17by doing that your program will be executed in terminal instead of Output Tab
- 9:23Now as a final step we need to create a virtual environment.
- 9:28It gives us an isolated environment
- 9:30to install third-party packages.
- 9:32In this way our projects can have their own dependencies
- 9:36which is exactly what we want in many cases
- 9:39and there are different ways to do that to
- 9:41create a Virtual environment but in this tutorial I will use pipenv. I use pipenv to
- 9:48create a virtual environment. This tool gives you an experience similar to NPM.
- 9:53NPM in JavaScript, which I really like it.
- 9:55This tool prevents you from memorizing a lot of
- 9:59other commands, which is really convenient.
- 10:02Okay, to use pipenv, first we need to install pipenv itself.
- 10:06And to do that, again, I open internal terminal
- 10:09in VS code by pressing control and backtick.
- 10:12And I type pip3, or if you're on Windows, just pip and pip.
- 10:17three install pipenv okay i've already done this step i don't need to do that
- 10:24in order to create a virtual environment with this tool
- 10:27you just need to say pipenv install it doesn't install anything it just create a
- 10:33virtual environment on your machine and as you can see
- 10:37two new files are created for us one of them
- 10:40is for tracking dependencies only and another one tracks the dependencies
- 10:47with the exact version of them okay as i told you pipenv
- 10:52create a virtual environment for you.
- 10:54This virtual environment has an address and
- 10:58we need, we do need this address because we
- 11:01need it in two places for two important settings.
- 11:05In order to get this address, you simply type pipenv dash dash venv.
- 11:12This gives you the address of that environment that is created for us.
- 11:17and the first place that we need this address is for code runner
- 11:21code runner needs to know where is this location
- 11:25this is the location of virtual environment
- 11:27otherwise it will use a wrong interpreter and to change this i open settings
- 11:34go to preferences settings
- 11:36okay i go to work space because i want to apply this setting
- 11:41only for this project not my whole system i switch to json mode
- 11:46and I type executor map
- 11:50And as you can see, many settings, many rows are
- 11:53added for different programming languages.
- 11:56We are working with Python. So we need to find Python.
- 11:59I replace this Python here with this address and I add slash bin slash Python 3.
- 12:07And if you're on Windows, again, Python, which is the Python interpreter itself.
- 12:12Now we are giving code runner the exact address of the Python.
- 12:16Python interpreter that we need for this project in this virtual environment.
- 12:21Okay, another place that we need to use its address is for VSGOD itself.
- 12:27VSGOd currently is using the wrong interpreter. I say the wrong
- 12:32interpreter because now we have a virtual environment and we need to
- 12:36use the interpreter that you just saw.
- 12:38And in order to change that, I open control panel by pressing Command-Shift-P and
- 12:46I type Select Interpreter
- 12:48Then I say, enter interpreter path, and I paste the address that I copied.
- 12:56and I press enter and now everything is ready for developing our exciting projects.
- 13:04Ladies and gentlemen, before our next flight
- 13:07begins, a quick request from your cabin crew.
- 13:10Please switch your phone to airplane mode or better yet silent.
- 13:14Then stow it away out of sight in your bag or in another room.
- 13:18Close any other tabs and let every distraction stay on the ground.
- 13:23If this journey needs your full attention,
- 13:25we'll now dim the cabin lights so you can focus. Please
- 13:28fasten your seatbelt.
- 13:30We're ready for
- 13:32takeoff.
- 13:42Okay, in this session we will start writing our first
- 13:45exciting project which is developing a chatbot.
- 13:48First, I create my main function, a function called main.
- 13:53And since this function doesn't have any
- 13:56implementation yet, I just put pass keyboard
- 13:59inside.
- 13:59Because in Python, when a block doesn't happen implementation,
- 14:04Python will start complaining about this matter.
- 14:07So for this reason, I put pass inside here. Then here I put an if statement.
- 14:13Let's write its code and I will explain it to you. Don't worry about it.
- 14:18First I say double underscore.
- 14:21name double underscore
- 14:24is equal to
- 14:25inside codes double underscore
- 14:29main double underscore in that case I want the main function to be executed
- 14:35okay what does that mean?
- 14:36let me explain it by saying that in Python there is a special variable called name
- 14:43and in the case that
- 14:46this file I mean chatbot.py is executed directly
- 14:51the value of this variable would be double underscore main double underscore
- 14:56only in that case and in that case I want the main function to be executed
- 15:02otherwise I mean in the case that this file is executed by another file I
- 15:09mean not directly the value
- 15:12of this variable will not be main this is a pattern
- 15:15that you will see a lot in different projects when
- 15:19you when we want to treat this file as a script
- 15:23okay now let's implement our main function
- 15:27At the first step, I want to get a piece of text from the user.
- 15:30So I use the input function, which is a built-in function in Python.
- 15:37And as an argument, I pass the label that I want the user to see.
- 15:42For example, ask something. Okay, that's
- 15:46great.
- 15:47And I get the value that user enters and I
- 15:50store it inside a variable called userInput.
- 15:55after that I'm gonna send this value for a function for a function called bot
- 16:00I define this function def bot
- 16:04this function is gonna contain the logic of our bot
- 16:07and this function will connect to a real LLM shortly
- 16:11I want this function to have a parameter called message
- 16:15which is the question that this bot is gonna answer in other word the query
- 16:20the query that this function supposed to answer but
- 16:24at this stage I'm not going to use this parameter because at this
- 16:27stage I want to keep things really simple and at this stage I
- 16:31want this function to send a simple piece of text for every single query
- 16:36this piece of texts is going to be this but hello how can I help you
- 16:43Okay, great. And I store it in a variable called answer.
- 16:47And I simply return this answer.
- 16:49Now I'm gonna call this function, call this
- 16:52function inside my main function and
- 16:55I send what the user has entered as its argument.
- 16:59So here I say user input.
- 17:02okay and I store it in a variable called bot answer
- 17:06then I print it print bot answer okay as simple as that now let's run our program
- 17:15OK, when you run your program for the first time,
- 17:18you see this virtual environment is activated.
- 17:22The next time you see it says ask something and it's
- 17:25better to put a colon here alongside the space.
- 17:28Okay, I say hello and it answers hello, how can I help you?
- 17:33As you can see, it's not smart. It sends this response for every single query.
- 17:37And in the following sessions, we're gonna make this
- 17:40function smarter by connecting it to a real LLM.
- 17:44Ladies and gentlemen, a quick note about your luggage.
- 17:48The full source code for every project in this course is waiting for you on GitHub.
- 17:54You'll find the link in the description of this video.
- 17:58If it helps you on your journey, please give the repository a star.
- 18:02and follow the captain on GitHub so you never miss a new flight.
- 18:07Thank you and enjoy the flight.
- 18:16Alright, now we are getting to a point where instead of using our dummy chatbot,
- 18:21we're gonna do the exciting part. I mean connecting our chatbot to an LLM.
- 18:25What does that mean exactly?
- 18:27Up to this point, you are using something like
- 18:30ChatGPT or Cloud or whatever chatbot you are
- 18:33using these days. What do you do?
- 18:35You're asking a question, you're getting an answer, and that's the process.
- 18:40Now, instead of that, instead using a chatbot directly,
- 18:44you're gonna write a program, you're going to send your question to this program
- 18:48and this program will send your questions to
- 18:51an LLM, get the response and return it to you
- 18:54that's the whole point of this project that we are doing
- 18:57now at this point to understand things better
- 19:00we need to cover a little bit of theory
- 19:02I hate theory too, but don't worry.
- 19:05We will cover, we will see, we'll see everything I explain in theory, in practice.
- 19:11On the other hand, it's necessary because in this course, there is no point.
- 19:16There is no in just writing a bunch of code
- 19:18without understanding what it actually means.
- 19:21So let's start by saying that if we take a step back and look
- 19:26at the bigger picture, what are we actually going to do?
- 19:31to see, to do.
- 19:32We are going to become an AI engineer.
- 19:35We are gonna add an AI feature to our application.
- 19:39We are goint to use an AI model and
- 19:42add it to our application.
- 19:44And this whole field is called AI engineering. Okay.
- 19:49In other words, if you follow this path all the
- 19:52way through and specialize in this field,
- 19:54you will become an A.I. engineer. which is actually a very hot and in demand field
- 20:01and you will see tons of job postings for it
- 20:05Another field you might come across is ML
- 20:07engineering or machine learning engineering.
- 20:10So
- 20:10what is the difference between an ML engineer and an AI engineer?
- 20:17An ML engineer typically focuses on building, building
- 20:21models, building and training machine learning models.
- 20:25And they do it from scratch. And it tends to be more math-heavy, research focused.
- 20:32And yes, they are working on things like designing models, training them, improving
- 20:38their performance and things like that.
- 20:41On the other hand, an AI engineer usually works with pre-trained models
- 20:47models that are already ready and they use
- 20:50these models to build smarter applications
- 20:54so instead of building the models from scratch
- 20:58they focus on using these models and integrating them into their products
- 21:04their real world applications and yes they focus
- 21:11these AI-powered features to their application.
- 21:14So, to summarize it, to put it in a very simple way,
- 21:18I can say ML engineers build and train models.
- 21:22On the other hand, AI engineers use these models.
- 21:26Another thing that I should explain it for
- 21:30you is NLP or Natural Language Processing.
- 21:34The goal of NLP is to enable computers to understand and generate human language.
- 21:42And we achieved this through algorithms, through model architectures.
- 21:47There is no magic behind and to achieve these
- 21:51goals we need algorithms and these architectures
- 21:54For example, in the past we had and still today we had an
- 22:00architecture called RNN or Recurrent Neural Network.
- 22:05And later transformers came along and I can say
- 22:09they completely revolutionized this field.
- 22:12In fact, much of the power behind these AI
- 22:16systems, these chatbots you see these days can
- 22:20be traced back to these architecture transformers and attention mechanisms.
- 22:27And the next word you hear is LM
- 22:33And the next one is LLM. Let's first focus on LM.
- 22:37An LM or language model is an AI model that
- 22:41can understand and generate human language.
- 22:45Okay, so language model.
- 22:48Now, what about an LLM?
- 22:50It stands for large language model, a language model that is large.
- 22:56It's a language that has been trained on a massive amount of data.
- 23:01Most of which comes from the internet.
- 23:04In other words, you can think of an L LL as a person,
- 23:07as a a person who has read and memorized an enormous amount of information.
- 23:13from the internet.
- 23:15Now what about LMM?
- 23:18LMM stands for Large Multi-Model Model
- 23:23The difference is that in addition to text, it can work with other types of data,
- 23:28or more accurately we say modalities, such as images, audio, and sometimes
- 23:34video.
- 23:35and after knowing these keywords we should answer
- 23:41what does an LLM actually do under the hood at a very basic level
- 23:47it just predicts the next word
- 23:50Pay attention to this, just one word at a time.
- 23:54And here, technically saying word, it's not quite accurate.
- 24:00The more precise term is token.
- 24:02But for now, don't worry, don´t worry about tokens.
- 24:05Just think of it as a word that we have in our language.
- 24:10For example, imagine we give your model something like this. Today I am very.
- 24:16the model tries to predict what comes next, the next word.
- 24:21For example, it might predict tired. Today I'm very tired.
- 24:25Okay, but the
- 24:26model, of course, doesn't stop here.
- 24:29It takes the text that you gave it, it adds the word that it just predicted,
- 24:35which becomes today I am very tired, and it tries to predice the next world.
- 24:40For example, today I am very tired because
- 24:44okay now we have this whole sentence so the
- 24:48mechanism is like this it predicts one word
- 24:52it adds it to the text and it keeps repeating these loops, this step.
- 24:58So this is the basic loop behind LLMs, behind text generation.
- 25:04So to summarize it, when you send a query to a language model,
- 25:09you give the model some text and the model predicts the next token
- 25:14then predicts next one, the next one the next and so on
- 25:17So if you ask a model a question and it gives you a 100 word answer
- 25:24It didn't generate those 100 words at once
- 25:28It generated them one token at a time in a loop Okay?
- 25:32Until it decide that the response is complete
- 25:36There is a special token called completion token
- 25:39And when this token appears, it's done and it returns the response So...
- 25:45That's a basic idea behind how an LLM generates text.
- 25:50and there are lots of different options
- 25:54lots of language models that you hear their names a lot
- 25:58for example ChatGPT, you use it, it is created by OpenAI
- 26:03Cloud is from Anthropic, Gemini, Llama, Mistral, DeepSeek, Qwen
- 26:09and instead of just listing a long list of different models
- 26:16it's better to see, to show you It's mine.
- 26:21Ladies and gentlemen, as we orbit planet chatbot,
- 26:24we're now approaching one of its moons.
- 26:27Its name is Hugging Face.
- 26:29In the world of AI engineering, this moon is a true landmark.
- 26:34A vast open harbor of free AI models, data
- 26:37sets, and tools shared by a global community.
- 26:41Almost every AI engineer docs here sooner or later,
- 26:45and on this journey, so will you. Please
- 26:47remain seated as we begin our approach.
- 26:57In the world of AI engineering, HuggingFace is
- 26:59a website whose name you're gonna hear a lot.
- 27:02And not only are you going to hear about it, you're
- 27:04actually going to be working with it quite a lot
- 27:07At first, this appearance interface might look a little
- 27:10bit intimidating, but don't worry about it at all
- 27:13In fact, it's really, really simple
- 27:16and they've done a really great job organizing everything
- 27:20And honestly, I'm really glad that something like this exists. It's a hub for AI.
- 27:25So, what exactly is HuggingFace? You can think of it as something similar to GitHub.
- 27:30You should definitely be familiar with GitHub.
- 27:33It allows people to publish their projects, collaborate on each other's code,
- 27:37work together to improve, to build something in general open source projects.
- 27:42And Hugging Face has the same goal.
- 27:44And these hubs, hubs like Huggingface and GitHub,
- 27:49has given us a huge amount of power.
- 27:51and has contributed a lot to the progress of
- 27:54software development and AI development these days.
- 27:56So I really appreciate their work and on HuggingFace, anyone can
- 28:01take a model, a model they have trained and publish it here.
- 28:05Other people can come and find these models, download them and even improve them.
- 28:10improve and publish them again so that's the
- 28:14purpose of hugging face and it's not it's
- 28:17not limited to AI models it you can find data
- 28:20sets as well the spaces what about the spaces
- 28:23the spaces allow you to to take something we have
- 28:25built publish it and basically say say to others
- 28:28here I have this try it and you don't even have to have hardware you can
- 28:34experiment with it without having hardware.
- 28:38And yeah, that's HuggingFace and let's take a quick look around.
- 28:42The first and the most important section is here, models, model section.
- 28:48Here you can find different kinds of models that have been published on Huggingface.
- 28:52You can search for a specific model by its name
- 28:56and you can apply different filters here.
- 29:00there are many many many different filters here and just keep in mind that
- 29:04hugging face changes a lot I mean this appearance filters might change for example
- 29:11you might see a particular task here today but it might not be here tomorrow or
- 29:17you might a task here that wasn't here yesterday anyway
- 29:22here there are different tasks that you can filter models based on this
- 29:27tasks.
- 29:27You also have parameters and parameters basically tell you how large a model is and
- 29:35they have a significant impact on both the size and the capacity
- 29:39the capacity of a model
- 29:42so generally speaking a model with fewer parameters
- 29:46is smaller and requires fewer resources
- 29:49but it also has a lower capacity and here we have libraries
- 29:53you can see different libraries here like TensorFlow, PyTorch, Transformers
- 29:58which is HuggingFace's own library and we have different providers here
- 30:03Honestly, I'm gonna keep this part, keep this session very brief
- 30:08because at this stage you don't need to worry about these details
- 30:11and we will cover them later, but I just don't want you to be intimidated.
- 30:16So that's about models and as I told you, you have datasets here.
- 30:21Here again, you can search through different
- 30:24datasets and you can apply different filters.
- 30:27Yeah, that's enough for now and I highly recommend that you
- 30:31you spent some time exploring Hagenface, click
- 30:35around, search for different models and
- 30:38just get familiar with the platform.
- 30:40Now let's take a quick look at one of these models, for example, Llama.
- 30:46which is released by Meta or Facebook
- 30:49you can see these different models and hugging face
- 30:52and I can say you for almost every popular model you find here
- 30:57there are different variations there are difference versions
- 31:00and in most cases the underlying model is the same but that company or maybe others
- 31:06may have changed that model for example a change in the number of parameters
- 31:12and they publish it again
- 31:13That's the reason you can see different versions for Llama here
- 31:17For example, one of them has 8 billion parameters Another one, 70 billion parameters
- 31:24And let's take a look at this one One important section is this part
- 31:28You can find the model's identifier here
- 31:31You're gonna need this name when you want to use this model
- 31:35When you want download it from HuggingFace
- 31:37And with this icon you can copy this identifier
- 31:41And here what you are looking at right now is
- 31:44called model card and you can see that here.
- 31:48You can think of the model card as a documentation for this model.
- 31:52It contains all the information that you need, that
- 31:56the publisher has decided to share with others.
- 31:59And you can read about what the model is, who created
- 32:03it, how it was trained, what data sets were used and
- 32:07benchmarks it was evaluated on and many many other details that you might need.
- 32:13Basically, if you want to understand the model in
- 32:17more depth and gather some information about it,
- 32:21decide whether it is a good model for you or not, this
- 32:24is where you can find the necessary information.
- 32:28and for some models you can even test the model here and you
- 32:33can see you can enter a prompt and
- 32:36and see the model's response right here without downloading
- 32:41this model on your system or on your server so
- 32:45there is a lot more information on these pages and i really recommend that you
- 32:51spend some time
- 32:53don't be afraid don't afraid to explore and
- 32:56Just click around, read things and just see where it takes you. Okay.
- 33:01In addition, before ending the session, there is a cool thing here.
- 33:08There is an icon here and it says use this model, different
- 33:12libraries and different providers and we have a notebook.
- 33:17For example, if I click on Hugging Face, you
- 33:21can see it guides you and that's really cool.
- 33:25Even if you are completely beginner and new to
- 33:28Hugging face, it tells you how to use this model.
- 33:32So here it has suggested that you can use a pipeline or you can
- 33:37auto classes and these auto classes gives you more control.
- 33:42That's really great and we will cover this, don't be worried.
- 33:45I'm just getting you familiar with these stuff.
- 33:49Yes, the whole point of this session is to know that Hugging Face exists,
- 33:54and it's basically one of the most important hubs,
- 33:58and you're going to work with it a lot. So just explore it.
- 34:06Now let's talk about how to choose and use a model.
- 34:09If the model you want to use is commercial, for
- 34:13example, a model like Sonnet by Anthropic,
- 34:16then you basically need to go through its provider and you need to purchase a plan.
- 34:21Yes, that totally depends on the provider because that model
- 34:25is commercial and is only provided by that company itself.
- 34:29But what if the model is open source?
- 34:31What if we have found a model on Hugging Face and
- 34:35we're going to use it in that case you have two options
- 34:38the first option is to run the model on your own hardware
- 34:43or use it directly this way
- 34:47This is the first option, the other option, the second option is to use a provider
- 34:50that has already deployed that model on their own hardware
- 34:54and they give you the access through an API key.
- 34:59So, and what is our approach in this course?
- 35:03In this course, since I know many of you don't have enough hardware
- 35:09that is enough for running these models locally, I'm going to leave
- 35:15the on-premise option aside and we are going
- 35:17to use a provider and the question is which
- 35:20provider? The answer is Groq.
- 35:23Personally, when I'm experimenting with different
- 35:26models, I use Groq, but the most important part
- 35:28is that it gives you a free tier so you don't
- 35:32have to spend money to follow up this course,
- 35:36which was really important for me.
- 35:42Ladies and gentlemen, a brief disclosure before we continue.
- 35:46Along this journey, we'll make stops at several third-party
- 35:50services like Groq, Tavily, Hugging Face, and Lang Chain.
- 35:54Neon Learn is not affiliated with, sponsored
- 35:57by, or endorsed by any of these companies.
- 36:00We chose each tool purely because it's great for learning.
- 36:04All product names and trademarks belong to their respective owners.
- 36:09Thank you for flying with NeonLearn.
- 36:19Okay, as I mentioned for this course we're gonna use Groq and to avoid any confusion
- 36:24this Groq is different from that Grok, that API chatbot from Elon Musk
- 36:29there are different products, the names may sound similar but they're not the same
- 36:33Okay, the first question, why have I chosen this product, this provider?
- 36:37The answer is pretty simple, it's very generous, it
- 36:40is very generous when it comes to its free tier
- 36:42Personally, I use it for a lot of my experiments
- 36:46Honestly, it happens rarely when I hit limits.
- 36:50It's also not a bad option for production.
- 36:52I mean, if you move to the paid plan and if you
- 36:56want to use Groq in your real-world projects,
- 36:59the pricing is still very reasonable.
- 37:02And the good point is that it offers a good
- 37:04range of different models which are suitable
- 37:06for different kinds of tasks.
- 37:08And it has models that are capable enough for your tasks.
- 37:12Another benefit is pricing.
- 37:14In terms of pricing is really good in comparison with many other popular providers.
- 37:20So for production is also good.
- 37:23And about pricing later on in this course, I
- 37:26will show you how to calculate pricing, how
- 37:30to calculate the cost of using these models,
- 37:32which is really important in the world of AI
- 37:35engineering.
- 37:36But for now, the most important thing that I
- 37:38want to say is that we're going to use Groq
- 37:41and it gives us a free tier and it's enough. It's enough for us.
- 37:48So in the first step you need to create an account and it's really simple.
- 37:53I've already done this step.
- 37:55So if I click on start building, it takes me to my dashboard.
- 38:00Here we have something very important API key.
- 38:03You are going to need that.
- 38:04You're going to create and API key and please, please do not try to copy my API key,
- 38:10I'm intentionally not hiding mine because I don't want to waste my time to edit.
- 38:16these videos hiding my api keys and it's also good
- 38:20for you because you can uh clearly follow me along
- 38:23this course and understand things better so
- 38:27i'm telling you again don't copy my ap i key
- 38:30and i i will delete i will the api key that i use
- 38:34during this course after uh after recording
- 38:38and here uh let's take a quick look at our playground playground is where
- 38:43we can directly interact with the models that Groq offers.
- 38:47And here we can see a list of different models that are currently available.
- 38:53In the future, maybe other models will be provided,
- 38:57and maybe some of these models are not provided anymore.
- 39:01And in that case, Groq will inform you if you are currently using these models.
- 39:07and for example look at this model Qwen this model
- 39:12is by Alibaba and you can choose this model
- 39:15send some prompts to this model and get the response
- 39:18currently I'm saying that you can experiment
- 39:21with this model here and later on we will connect
- 39:25these models to your own application so we can
- 39:28send queries through our own application which is really interesting and
- 39:33Here another point is that another interesting thing,
- 39:36another thing that you might be wondering is that
- 39:39earlier I told you that if you want to use OpenAI's model,
- 39:44you normally need to go to OpenAI and purchase some credit.
- 39:49And now you might say, why do I see OpenAI models here in Groq?
- 39:53Well, the important distinction is that the famous
- 39:58OpenAI model, yes, they are enterprises are paid.
- 40:02to go to OpenAI itself. But the point is that OpenAI has also released some
- 40:08open-weight models and Groq supports those models exactly.
- 40:14So these models are open-weighted.
- 40:17You can go to Hugging Face, download these
- 40:19models and open when you're harder yourself.
- 40:21So anyone can do that.
- 40:23And the final thing, if you look at these models, if
- 40:27hover them, you can see the limits on your free tier.
- 40:31For example, how many liquids you can make per minute, per day.
- 40:36and how many tokens you can process during the day.
- 40:40So at this step, the only thing that I want you to do is to create a Groq account
- 40:47and create your own API key. We need that API key in the following sessions.
- 40:55Now, the really interesting thing about Groq is that
- 40:59they have built specialized hardware and infrastructure
- 41:03that is extremely well optimized for running these kinds of models.
- 41:08And because of that, the responses are incredibly fast.
- 41:12In fact, even if you download and run these models on your
- 41:16hardware, you cannot get the same result only if you provide
- 41:21that kind of hardware which would be really pricey for you.
- 41:25So for these purposes, using a provider really makes sense.
- 41:29And I always say that only focus on your product, your AI
- 41:33features, and use a provider because they know how to do that.
- 41:37They use the proper type of hardware, which is really important.
- 41:49Okay, here is my API key.
- 41:52Hopefully by the time of watching this video, you
- 41:55have already created your account and created
- 41:58your API key because as I told you, we are
- 42:00going to need it in the following sessions.
- 42:02But now I'm going to talk about the connection between you and Groq.
- 42:07How exactly do we establish a connection between
- 42:10our application and the provider, which in
- 42:12this case is Groq, okay? Okay.
- 42:15Imagine we have a query. A query, for example, what is the capital of France?
- 42:19The question is that how do we send this query
- 42:22to the provider and get a response back?
- 42:25We do it through an API. If you are already familiar with web development,
- 42:30this concept should be familiar to you. If not, I can say at the simplest level,
- 42:36you can think of it as a URL that you are seeing at your browser.
- 42:40You are watching this video on YouTube and there is a URL.
- 42:45When you are working with API, we have a similar thing,
- 42:48but instead of opening your browser and entering that URL,
- 42:52we are using that API in our application. So that is what API means.
- 42:59Now there is something important that you should know about
- 43:04you should about it when it comes to AI engineering and working with APIs
- 43:08In practice, you are really going to interact with these APIs yourself
- 43:12For example, you're not going to use Python's request package
- 43:17constructing an HTTP request, adding your API
- 43:20key to it, sending it and parsing the response
- 43:24No, no, you don't do that
- 43:26Technically you can do, but in AI engineering, we usually
- 43:30have a better approach, more standard approach.
- 43:32Instead, we use something called a wrapper. So why exactly is a wrapper?
- 43:37Think about the word itself, okay?
- 43:39When you wrap something, you put something around
- 43:42it, like a sandwich, okay, like the shower one.
- 43:45So a wrapper is basically a piece of software.
- 43:50that sits around another piece of software and
- 43:53makes it easier for you to interact with.
- 43:56and you can see an example of it here here it suggests that you use Groq
- 44:02the wrapper of Groq itself it says instead of interacting with my api directly
- 44:07just use my python package
- 44:09the point is that they have already developed this package
- 44:12and this package handles a lot of complexity for you
- 44:16and you don't have to create the request yourself
- 44:19you just import this package and create an instance and client
- 44:22and you start using it
- 44:24and that's really simple and it's just a wrapper around that API
- 44:29and Groq isn't the only provider that does that
- 44:33if you work with OpenAI models, you have the same concept there
- 44:37and even if you use other libraries and other providers, you'll have the thing there
- 44:44for example, Langchain is an AI engineering library, they use it a lot
- 44:49To elaborate on that, to explain why don't we
- 44:53just communicate with the API directly again.
- 44:56One simple, as I told you, is simplicity.
- 44:59And if we did everything yourself, you would have to handle a lot of edge cases.
- 45:04For example, you had to deal with authentication, constructing requests
- 45:09correctly, handle errors, parse responses, and many, many other things.
- 45:14It's clear that.
- 45:16The more code you write yourself, the more opportunities there are for mistakes.
- 45:20So when there is a better approach, that doesn't make any sense.
- 45:24The second advantage is standardization.
- 45:28It gives you a standard way to interact with the model.
- 45:31We can communicate with our provider in a more consistent way.
- 45:36And this becomes especially important when you join a team.
- 45:40Imagine you're joining a team and you have to ask many questions.
- 45:43For example, what parameter do I use for API keys?
- 45:47And how do I tell the model how many tokens it should generate?
- 45:51and what do I call this parameter, that parameter, the
- 45:55parameter for maximum number of tokens and many other things.
- 45:59With these wrappers, things are much easier and we have a
- 46:04standardized interface and we all agree that standard.
- 46:10and many many details are already taken care of.
- 46:14So this way instead of everyone inventing their own way of communicating,
- 46:20we have a standard and consistent way and that's important
- 46:25when you are collaborating on a project.
- 46:28Ladies and gentlemen, a special message from the captain.
- 46:32This journey was originally planned as a premium first-class flight.
- 46:37But the captain made a decision to open the doors
- 46:40and bring as many passengers on board as possible.
- 46:44So this entire course is now completely free right here on YouTube.
- 46:49If it's helping you, please share it with friends who'd love this journey.
- 46:53Every passenger you bring makes this ship stronger.
- 46:57The captain is truly grateful to every one of you.
- 47:00Thank you for flying with Neon Learn.
- 47:04Okay, in the previous session we talked about wrappers.
- 47:07I told you that instead of using an API, we're going to use a wrapper.
- 47:11And as an example, we saw this Groq wrapper, which is published by Groq itself.
- 47:16It suggests us to use this wrapper.
- 47:19However, in this tutorial, we are not going to use Groq's wrapper.
- 47:22We're going use OpenAI wrapper.
- 47:23The reason is that honestly, I think OpenAI Wrapper is more widely used.
- 47:27So it's more useful to you.
- 47:30In addition, Langchain is also extremely popular and you
- 47:33will see these too often in job hostings as a requirement.
- 47:36And you should be familiar with these two,
- 47:38OpenAI and Langchain, if you want to get a job.
- 47:41You should understand how to work with these
- 47:44wrappers, what parameters they have and what
- 47:46is the format of output.
- 47:48That's really important and one of my goals here
- 47:51is to get you closer to the actual job market.
- 47:54So in this tutorial, I've decided to use OpenAI Wrapper.
- 47:57something that is really important and you will see a lot.
- 48:00But you might be wondering, you might say,
- 48:03wait a moment, earlier we decided to use Groq as our provider.
- 48:07So what exactly is happening here?
- 48:09Because obviously the OpenAI wrapper is associated with OpenAI.
- 48:14while Groq is a completely different provider and what's happening here?
- 48:18How can we use OpenAI Wrapper with something else like Groq?
- 48:21Well, the interesting thing is that even if we don't want to use
- 48:26Open AI's models, there are still cases we can use OpenAi Wrappr.
- 48:31In other words, you can use the OpenAI wrapper, but
- 48:35instead of sending your request to OpenAI servers,
- 48:38you can configure the wrapper and send the request to another provider such as Groq
- 48:44of course this only works if the provider supports the OpenAI compatible API format
- 48:51so that's important and fortunately Groq supports this
- 48:55so that is exactly what we are going to use we are gonna use OpenAI Wrapper
- 49:00but instead of sending request to OpenAI we are goint to send our request to Groq
- 49:06and honestly I'm getting a little bit tired of theory myself.
- 49:10So let's stop talking about theory and let's get into the code.
- 49:16Ladies and gentlemen, we are now beginning our descent onto planet Chatbot.
- 49:22Please fasten your seat belts and prepare for landing.
- 49:25Touchdown, welcome to the surface.
- 49:28At the heart of this planet lives a large language model,
- 49:31and it's time for our first contact.
- 49:34We'll send our very first message and wait for its reply.
- 49:48After having many theory-based sessions, now
- 49:51we want to come back to our code and continue
- 49:53developing our project, which is a chatbot.
- 49:57So the first step is to install the wrapper I told you, OpenAI.
- 50:01And this is the first time, I guess, we want to install a third-party package.
- 50:04I open up internal terminal by pressing Ctrl and back tick, and I type pipenv.
- 50:13install.
- 50:15whenever you want to use this tool, I mean pipenv to install a package
- 50:18you simply say pipenv install and then you list the name of the packages you want
- 50:23to install now we have two packages that we want to install
- 50:27the first one is OpenAI, as I told you, the wrapper itself
- 50:30and I also want to instal.env at the same time
- 50:35I will explain what this package is for just wait a moment
- 50:40I just press enter to install these two packages We need to wait a bit.
- 50:46Okay, installation is done and everything is ready.
- 50:49I mean, these two packages are installed and we can start using them.
- 50:53As I told you, when
- 50:55you want to use Groq, you need an API key.
- 50:58And this is my API key and now you have your own API key
- 51:02and the point is that there is absolutely nothing wrong
- 51:06with putting your API key directly into your code.
- 51:09Just for learning and experimenting purposes.
- 51:12but this is not okay at all for production. Why?
- 51:17Because eventually you are going to push this code to a repository
- 51:21and you definitely don't want your API key sitting directly inside your source code
- 51:27where other people can see and access it.
- 51:30So in situations like this, when in situations we have a secret key, an API key,
- 51:37the best practice is to use environment variables.
- 51:41Here in this project, I create a file called.inv
- 51:49Okay.
- 51:50Then I take this API key.
- 51:54And I go here
- 51:57I paste it, I cut and paste it here. Now this API key is inside my.n file.
- 52:04Now I'm going to give it a name.
- 52:06I call it Groq API key. Okay.
- 52:12And as a convention, we use uppercase letters. Okay. That's it. Everything is okay.
- 52:20with our.n file. So what happens after this?
- 52:24Now this API key becomes an environment variable and wherever I need this API
- 52:31key, I inject it into my application.
- 52:35instead of hard coding this piece of information here
- 52:39I use it as an environment variable I store it as environment variable
- 52:44and whenever I need it, I load it
- 52:46and this is a very common pattern you will see in real world applications
- 52:51now the question is that how can we actually access it in Python?
- 52:57in Python there are different ways we can handle environment variables
- 53:01but one of the most common and popular approaches is
- 53:04is using the python.env package that we installed earlier.
- 53:09So the first step for using this package is importing it.
- 53:15I say import.envs.
- 53:18and there is also one more thing that we need which is OS module so I also import OS
- 53:27Now, how
- 53:28can we use it?
- 53:29We simply call load.env, the helper function that this.env provides.
- 53:35So I say.envs.load.envs as simple as that.
- 53:40So this loads the variables from.enV into the environment.
- 53:46And whenever we need it somewhere in our application, we can access it.
- 53:51And how can be accessed? Using OS.
- 53:56Here I say os.getEnv
- 54:00and then you pass the name of the variable you have
- 54:04which is Groq API key for us in this case.
- 54:08and that's it and to see it in terminal to see everything is working correctly
- 54:14I printed.
- 54:16Okay, now if I run my program,
- 54:19Again, after activating environment variable, as
- 54:22you can see, the API key is printed here for us.
- 54:26Now, everything is ready to use OpenAI Wrapper. I'm going to import it first.
- 54:32I say from...
- 54:35OpenAI Import
- 54:38OpenAI is a class, it's a class that we can create an instance of it. So down here,
- 54:46I say
- 54:48OpenAI
- 54:49and this way by putting parentheses I can create an instance of it
- 54:54after that we need to pass some arguments
- 54:57and the first argument is definitely API key
- 55:01so here I say API key and you can see that it is suggesting that to us
- 55:06so I put the API key that I have here
- 55:11and I don't need this anymore I remove it and this is my API key
- 55:17and there is also one more important thing this is OpenAI Wrapper
- 55:23but we want to connect it to Groq so we need to change something
- 55:28and we apply this change using an argument and that argument is baseline
- 55:35so base URL I'm sorry so here I say base URL
- 55:41and this is a URL that is provided by Groq and when you set it
- 55:45your requests are going to be sent to Groq instead of OpenAI
- 55:51and the API is api.groq.com OpenAI v1 so here I say HTTPS
- 55:59API.
- 56:01groq.com
- 56:03and open AI
- 56:06v1 and finally i'm going to store the client
- 56:09object it is a client object inside a variable
- 56:12and i call it llm okay
- 56:15Usually you will see this variable named something like client, but here I
- 56:20prefer to call it LLM because this object is going to be our inference for
- 56:25communicating with the language model and I want you to feel it closer.
- 56:31now we have our LLM object and we have configured it correctly
- 56:36now the next step is to use it to send a request to it
- 56:41for this purpose here inside my bot function instead of having this fixed text
- 56:47I say I use this LL I say
- 56:49LLm chat.completions.create
- 56:55okay don't worry I will explain each part of it one by one for now
- 57:00Let's just write our code, provide the arguments that we need,
- 57:04and we will come back to it shortly.
- 57:07So the first thing that we needed is model name.
- 57:10and the question is that what model are we
- 57:13going to use and where do we choose it from
- 57:15where do you find it and the answer is here in your
- 57:19dashboard you can see the list of models that are provided
- 57:24you can choose the model that you want you can copy
- 57:28its name pay attention when you want to use Groq
- 57:31you should copy the name of the model here not on hugging face hugging face
- 57:37is for when you want use hugging face to use models but here we are dealing with
- 57:43Groq so we we need this identifier okay i'm going to use this model and open a model
- 57:51and it is gpt-oss 20 billion parameter and i copy its
- 57:57name and here i say model and i paste it here inside
- 58:05okay so far so good and let's move on to the next argument which is messages okay
- 58:14This messages array is a more accurate list in Python we call it list.
- 58:19It's going to be a list of messages and each message has a special format.
- 58:25It needs two pieces of information, role and content.
- 58:30So to put a message here, I say role.
- 58:35It should be set and
- 58:37Alongside, you should specify content.
- 58:41Don't worry about it for now, I will explain to you. For now,
- 58:43I just set the role to user.
- 58:47and for content I'm going to put the message that this function receives here
- 58:54okay that's it basically we have three roles user, system and assistant
- 59:00for now just understand that by using user role
- 59:04you're telling the LLM that this message is coming from the user
- 59:09in other words a human is asking a query
- 59:13and you're supposed to enter this query
- 59:16Yeah, that's enough for now. We will talk about that in detail later.
- 59:21So here I'm going to send this request and whatever comes back,
- 59:25I'm gonna store it in a variable called response.
- 59:30Now, response itself is an object, it is nested and it contains several fields.
- 59:38And when we want to extract the LLM's response, the
- 59:41first item that we need to access is choices.
- 59:45So I say response.choices
- 59:48and I need the first
- 59:51item. Again, I will explain it. Please be patient.
- 59:55And then inside I have message.
- 1:00:00Okay, and finally we have
- 1:00:03content.
- 1:00:06And finally it gives me the answer that the LLM
- 1:00:09has generated and I just return it instead of
- 1:00:14the previous thing. I say return and that's it.
- 1:00:19I agree.
- 1:00:19The syntax is a little bit weird at first, but I can make it more clear to you.
- 1:00:26So let's start here when we are creating our request.
- 1:00:30What does that mean? What does this chat refer to?
- 1:00:33It means that we want to interact with the model through a chat interface.
- 1:00:39there are also other types of interactions
- 1:00:41like image generation, embeddings and so on
- 1:00:44but in this case we are building a chatbot
- 1:00:46so we want to communicate with this model using the chat interface
- 1:00:51okay and here what does this completions mean?
- 1:00:55It refers to the idea of text completion that I talked about.
- 1:00:59You mean, I told you that an LLM generates text one token at a time.
- 1:01:04And that's essentially the meaning behind these completions here.
- 1:01:08And create is just the trigger.
- 1:01:11When we call create, we are essentially saying create a completion for me.
- 1:01:17And yeah, you are sending your request. And this is a point where
- 1:01:23where you are building your request, you're sending your request.
- 1:01:27And you send your request and get a response back. So
- 1:01:30what do we have here? Response.choices.
- 1:01:33Firstindex.Message.Content. What does that mean? Why do we
- 1:01:37have a list here? That's because
- 1:01:40in some
- 1:01:41cases you might have more than one response. But
- 1:01:45in 99% of the cases, you are gonna have just one item. And
- 1:01:50This array will contain just one item. So we need this single item
- 1:01:55and because of that, I'm extracting it. Here we have a list that contains one item
- 1:02:01and I'm taking out that one item,
- 1:02:05And after this we have message.content.
- 1:02:08Be aware that the format of the response is also based on a message.
- 1:02:13Remember this message that we constructed here.
- 1:02:16We had a role and we had content and we have the same here.
- 1:02:21We have a role, and we a content.
- 1:02:23Although here we have role assistant instead of role user.
- 1:02:28And content is the answer that the LLM has generated.
- 1:02:34So again, the user sends a message with the role user
- 1:02:38and the response that comes from LLM has the role assistant.
- 1:02:43I know this might feel a little confusing right now, but don't worry.
- 1:02:48I'm going to repeat these concepts several times
- 1:02:51in different contexts and at different speeds.
- 1:02:54So we will have plenty of examples, explanations and you don't need to be worried.
- 1:03:01So up to this point, we have managed to get input from the user, send it to
- 1:03:08a real LLM, and now we have the response.
- 1:03:12And the only
- 1:03:13thing left is to test our program and make sure everything is working.
- 1:03:19correctly.
- 1:03:20One more time, okay, it says ask something and I say hello.
- 1:03:27Okay, again, I run my program one more time and this time I say
- 1:03:34What is the capital?
- 1:03:37of
- 1:03:38France
- 1:03:40And as you can see, it responds and here we go.
- 1:03:44You can see that we are successfully communicating with the LLM
- 1:03:48and getting a real response back.
- 1:03:51So we have officially connected our application to a real language model.
- 1:03:57Ladies and gentlemen, please direct your attention
- 1:04:00to the screen for a brief safety demonstration.
- 1:04:03In the unlikely event that you enjoy this lesson,
- 1:04:06a like button will drop down in front of you.
- 1:04:09Simply pull it toward you and press.
- 1:04:12Please take a moment to locate your nearest subscribe button.
- 1:04:16It may be located just below this video.
- 1:04:19Comments can be found under your seat.
- 1:04:22Please use them freely at any time during the flight.
- 1:04:26This concludes our safety demonstration. Enjoy the flight.
- 1:04:37In the previous session you saw that whenever
- 1:04:39you want to send a request to the provider
- 1:04:42you need to specify a number of parameters and in this session
- 1:04:46I want to explain what those parameters actually mean and are
- 1:04:50The first one is Model and you saw that.
- 1:04:52This is very straightforward and we need to
- 1:04:55tell the provider which model we want to use.
- 1:04:58The provider provides us with a number of different models and we
- 1:05:03need to specify exactly which one we want work with.
- 1:05:07And the next one is Messages.
- 1:05:09As you saw, Messages is a list of messages and
- 1:05:12you might be wondering why is this a list.
- 1:05:15why isn't just a single message we are only sending one request
- 1:05:19for now let me leave it aside I'll make that clear for you in just a little bit
- 1:05:26okay so for now, let's ignore it and instead let's focus on this one role
- 1:05:34each message should have a role and we have three different roles okay
- 1:05:41And the point is that these roles are not specific to OpenAI.
- 1:05:47They are a general concept in AI engineering and you will see them in
- 1:05:52different places, maybe with different names, but the concept is the same.
- 1:05:57So first, what are these roles? These are System, User and Assistant.
- 1:06:04So far, we have already seen the User role.
- 1:06:06whenever we want to send a message as a user
- 1:06:10we specify its role as the user role
- 1:06:14the user-role represents what the user is asking okay
- 1:06:17but what about the assistant role? I've already said that
- 1:06:21that is the response that is generated by the LLM
- 1:06:26But I think it is better if I show you
- 1:06:30if I show you this in practice so here instead of printing
- 1:06:34the content the content of response I just print
- 1:06:38the role of the response and
- 1:06:40here if I
- 1:06:42run my program and
- 1:06:45say for example hello as you can see it says this message
- 1:06:48that we are printing has this role assistant role
- 1:06:52okay we have another role which is system role what
- 1:06:55is it used for what is the system role used for
- 1:06:59system messages are like instructions are
- 1:07:02like instructions that we give to the model
- 1:07:05for example when you are talking to chat gpt
- 1:07:09You might tell it, I mean during the day when
- 1:07:12you have conversation with AI chat, but you
- 1:07:15might tell, answer this question in a very natural way, in a
- 1:07:21very concise way, explain it using the simplest
- 1:07:25language that you can keep the answer
- 1:07:29really short. Summarize this for me.
- 1:07:31And these are examples of instructions that you specify for the model.
- 1:07:36And here in our program when we... want to specify these kinds of instructions,
- 1:07:41we need to use this kind of role,
- 1:07:45system role.
- 1:07:47Now let's move on to another parameter which is temperature
- 1:07:52What does temperature mean?
- 1:07:54Temperature is related to the model's creativity and randomness. In
- 1:07:59other words,
- 1:08:00how deterministic do we want the model to be?
- 1:08:03there are two well known parameters that can affect this behavior, I mean randomness
- 1:08:09one of them is temperature, another one is top P
- 1:08:14They influence how likely the model is to
- 1:08:17produce different responses to the same input.
- 1:08:22In other words, they give us some control over how predictable or creative the model
- 1:08:28is.
- 1:08:29and as a general intuition most of the time you
- 1:08:32will see developers working with temperature
- 1:08:35you will use temperature most of time and we rarely change top peak
- 1:08:40they behave differently under the hood but just as a generally intuition
- 1:08:45and about temperature I should say it is
- 1:08:49typically specified as a value between 0 and
- 1:08:532
- 1:08:53If you bring the temperature closer to zero, the model becomes more deterministic
- 1:08:59and that means if you give it the same prompt multiple times
- 1:09:04you are more likely to get very similar responses
- 1:09:07for example, imagine we say tell me a story
- 1:09:11With a very low temperature, the model will
- 1:09:14tend to produce a more predictable response
- 1:09:17But as you increase the temperature, the model becomes more creative
- 1:09:21So you're more likely to get different responses each time to the same input
- 1:09:26So you can think about it like this
- 1:09:28Lower temperature, more
- 1:09:30deterministic and predictable Higher temperature,
- 1:09:33more varied and creative
- 1:09:35The question is that, when do we want to have a deterministic model?
- 1:09:39when we are dealing with factual tasks like coding tasks
- 1:09:43in these cases we don't need creativity we don' need much creativity
- 1:09:48we care about getting a reliable response a consistent response
- 1:09:52so in these case we can lower the temperature
- 1:09:55on the other hand if you are asking the model to write
- 1:09:58something creative for us, brainstorm ideas to come up with different possibilities
- 1:10:03in that case we can increase the temperature and in most cases
- 1:10:07if we want something in between a value that you can keep in mind is 0.7
- 1:10:14and especially for something like this we are developing a chatbot
- 1:10:18and we want something in between not too creative, not too factual
- 1:10:23okay now another parameter that you should know about is max tokens
- 1:10:27so what does max tokens mean?
- 1:10:29it sets a limit on how many tokens the model is allowed to generate
- 1:10:33as I told you until now I've been using the word word
- 1:10:37to make things easier to understand but in the world of LLMs
- 1:10:42precise term is token and I will definitely talk about what tokens mean
- 1:10:48what tokens are later for now let's just imagine that tokens are same as
- 1:10:54words in our language anyway what does
- 1:10:57this parameter actually mean and do
- 1:11:00what it does is that it puts a ceiling on the amount of text that the model is
- 1:11:07allowed to generate for example Imagine you ask the model to write a story.
- 1:11:12The model might start generating a thousand word story,
- 1:11:15but maybe you only want 100 words.
- 1:11:19and one reason can be cost generated tokens have a
- 1:11:22cost so you might want to control this process to
- 1:11:25control how much the model is allowed to to generate
- 1:11:28tokens for example you might set max tokens
- 1:11:31to 100 but uh there is one important thing that
- 1:11:34you need to know about this this parameter it's
- 1:11:38not a graceful way to stop the model because
- 1:11:41it's more like putting a hard limit let me give
- 1:11:43you an example imagine the model it's gonna generate a story
- 1:11:47this starts with something like this
- 1:11:50once upon a time there was a bird that lived in the forest
- 1:11:54every day would go out to find food and in order to
- 1:11:59and it stops.
- 1:12:01And here the model suddenly hits that limit that we gave it.
- 1:12:07So the model gets stopped
- 1:12:09here.
- 1:12:10It was actually going to continue, but we forced it to stop.
- 1:12:15So when you are setting this
- 1:12:17token, keep in mind that it is a hard limit.
- 1:12:19It doesn't guarantee that the ending is meaningful.
- 1:12:21It may not naturally finish its thought within that limit. And another problem
- 1:12:27parameter is streaming. What about the streaming?
- 1:12:30Normally, when we send a request to the provider
- 1:12:33like what we did in the last session,
- 1:12:35the model generates a response, a complete response, and it is returned to us.
- 1:12:41So we have to wait, we have wait, but this is not similar to this
- 1:12:45experience that we have with ChatGVT, Groq and these kinds of chatbots.
- 1:12:50Instead, the response is generated piece by piece.
- 1:12:53Why? Because users are used to it.
- 1:12:55Users hate staring at a screen, waiting several seconds without saying anything.
- 1:13:02So instead, it feels much better, more natural if
- 1:13:05the response starts appearing piece by peace.
- 1:13:09OK, and we see what is generated immediately by now.
- 1:13:13This way the model generates some text and we
- 1:13:17display it and it generates more text and
- 1:13:20we continue displaying it and that is what streaming is about and allows us to do.
- 1:13:26We will cover that, we will cover streaming in our chatbot project.
- 1:13:30I will show you how to do that.
- 1:13:32And these were the important parameters that you need to know.
- 1:13:37you need to be familiar with. And in the next session, we will add the conversation.
- 1:13:51Now let's make our application more interesting.
- 1:13:54Right now our program works like this.
- 1:13:56Every time I run the program, I type something,
- 1:13:59I get a response and the program terminates.
- 1:14:01But that's not the experience similar to what you have in everyday life.
- 1:14:08You open a chatbot, you say something and then you
- 1:14:11again have an opportunity to ask something else.
- 1:14:15And here we
- 1:14:15need to create a loop, we need add a loop.
- 1:14:18to have the similar experience
- 1:14:22and that is exactly the kind of scenario in programming that we need a loop
- 1:14:29and the question is what kind of loop do we need a while loop here because
- 1:14:33we want this process to continue for as long as the user wants
- 1:14:39the user want to terminate and we terminate the program so here in my
- 1:14:46main function, I'm going to add a while loop
- 1:14:50and as its condition I put true because I want this while loop to run indefinitely
- 1:14:57and
- 1:14:57whenever the user decides they don't want to continue
- 1:15:01we will use the break statement to jump out of the loop
- 1:15:05now here I'm going to move this line of code that gets an input from the user
- 1:15:12and I just want to change its label a little
- 1:15:16bit I say type
- 1:15:19Q
- 1:15:21to
- 1:15:22terminate.
- 1:15:24Okay, here I add a colon and now everything is perfect for the label.
- 1:15:28Okay, the next step I want to clean up.
- 1:15:30I want clean up what the user enters by using two methods.
- 1:15:37The first is strip
- 1:15:40and the other one is lower what is a strip for a strip
- 1:15:44for removing unnecessary spaces from the
- 1:15:47beginning or from the end the user might enter
- 1:15:51something like this space space space hello
- 1:15:55space space and by adding this method these spaces
- 1:15:59are removed okay so this one for a street and the another one a lower it converts
- 1:16:06all letters to lowercase for example if user instead
- 1:16:10of q types q uppercase q it is converted to
- 1:16:14lowercase q and this means that our program doesn't
- 1:16:18have to care about capitalization anymore
- 1:16:21and these were these two methods next we need an if statement because
- 1:16:26if we don't put any if statement here our y loop is going to
- 1:16:31continue forever but we need an if statement and check if what the user has
- 1:16:36entered this queue we want to terminate the program so I say if
- 1:16:41user input
- 1:16:43is equal to
- 1:16:44Q, in that case, print.
- 1:16:46Good luck.
- 1:16:51and then jump out of this loop I use break statement for this
- 1:16:56okay and then I want to keep the rest of my code
- 1:17:00but I move it inside this while loop okay now we have a conversation loop
- 1:17:06so what happens
- 1:17:08now on every iteration
- 1:17:11we get an input and then we check we check if it's Q
- 1:17:18we print good luck and
- 1:17:19exit the loop and finally exit the program and
- 1:17:24if it's not Q we send that input for our LLM.
- 1:17:29And
- 1:17:30and a response arrives we display it on the screen on the
- 1:17:33screen and we give the user another opportunity
- 1:17:36another opportunity to to enter something else and
- 1:17:40this way we have created a continuous continuous
- 1:17:43loop and now let's run the program let's test it
- 1:17:47so i run the programme
- 1:17:50it says something for the first
- 1:17:52scenario i type
- 1:17:54cue as you can see it works correctly
- 1:17:58And now I type hello.
- 1:18:01okay it says hello how can I assist you today and
- 1:18:05you can see I still can continue asking something else okay for example
- 1:18:13What is a
- 1:18:15capital of
- 1:18:17France
- 1:18:19and as you can see it answers and we successfully
- 1:18:23implemented what we wanted to implement
- 1:18:27you
- 1:18:33okay in this session first i'm going to do a
- 1:18:35little bit of refactoring and the reason is that
- 1:18:39if we keep putting all new features in our main
- 1:18:42function it's going to get messy and that is not
- 1:18:45a good approach it it makes our application less
- 1:18:49maintainable and less extensible so let's do that
- 1:18:54first i take this while loop here i cut it
- 1:18:57and i move it into its own its own function
- 1:19:00So I define a new function called chat
- 1:19:04and I paste this piece of code here
- 1:19:07okay, now I go to the main function
- 1:19:10and I call this chat function that I just defined
- 1:19:14so let's run our program to make sure it is working like before
- 1:19:18okay, my visual environment is activated one more time and now I type hello
- 1:19:23as you can see my application is working correctly
- 1:19:27now I'm going to talk about something very very important
- 1:19:32and that is conversation history. Let me show you an example. Here I ask what is
- 1:19:39the capital of France
- 1:19:41and it answers the capital
- 1:19:44of France is Paris now I ask what is
- 1:19:48It's population.
- 1:19:52As you can see, it says I'm not sure what you're talking about
- 1:19:57and it doesn't have any information about the subject but why is that?
- 1:20:03Here I'm asking what is the capital of France
- 1:20:05so it's obvious that this it that you see here is referring to France
- 1:20:10but the model doesn't know what's happening here.
- 1:20:14Ladies and gentlemen, we've just experienced a small communication glitch.
- 1:20:19You may be feeling a little anxious right now. That's completely understandable.
- 1:20:25As you've just seen, large language models have no memory.
- 1:20:28Every message reaches them as if it were the very first.
- 1:20:32To keep a conversation going, we have to remember it for them
- 1:20:36and send the history along with every new message.
- 1:20:39But don't worry, your captain will show you exactly how it's done.
- 1:20:45At this point you should know that LLMs
- 1:20:48don't automatically have memory.
- 1:20:50And the way an LLM works is basically like
- 1:20:54this. A query comes in, the model generates a response,
- 1:20:58and it's over. Okay? It doesn't save any information.
- 1:21:03and when the next query comes in, it's totally new.
- 1:21:07The model doesn't have any access to the previous
- 1:21:10messages, previous conversation, and it remembers
- 1:21:13nothing. So, if I ask, like this scenario, what is the capital of France, and
- 1:21:19it answers Paris, and then I ask a new question, what
- 1:21:23is its population, the model doesn t know, doesn t inherently know what that
- 1:21:30it refers to that's a whole point of what i'm
- 1:21:33going to talk about and you should know that
- 1:21:36conversation history doesn't happen at model level
- 1:21:39it happens at application level if you want
- 1:21:42history you need to use your software knowledge
- 1:21:46to add this feature to your application and here
- 1:21:49you're looking at an actual AI engineering task that means
- 1:21:53we are going to use our software engineering knowledge
- 1:21:58we are going to use the software engineering concepts
- 1:22:01to solve a problem that is related to AI models and what do we do?
- 1:22:05Our LLM doesn't have any memory and we are gonna solve that how can we do that?
- 1:22:12Remember
- 1:22:13The LLM receives an input and produces an output.
- 1:22:20what we can do is to store the conversation history and send it with our query
- 1:22:26and when we do that we have history okay so what
- 1:22:30happened we restore conversation history somewhere
- 1:22:34and now whenever we want to send a new query we send
- 1:22:37that history along with our along with that query
- 1:22:42and now you can understand you can understand it better
- 1:22:45why we have this list here instead of just one message here instead
- 1:22:50of just passing one query we can we can pass previous messages
- 1:22:55previous questions and answers and when we do that
- 1:22:59the model will have enough information about that topic
- 1:23:03you
- 1:23:09Okay, in this session we are going to see how
- 1:23:13we can add conversation history to our code
- 1:23:16and honestly this is very, very easy to implement and logic is actually quite simple
- 1:23:22and the process is at first we need to create
- 1:23:25an empty list and on each iteration we need
- 1:23:29to add the user's query to it and after receiving the response we need
- 1:23:34to add response as well.
- 1:23:36and whenever we want to send a new query we
- 1:23:39need to send that new query plus the history
- 1:23:42so let's do it first up here i mean
- 1:23:46below this llm global variable i'm going to
- 1:23:50define a new
- 1:23:53a variable called messages and I set it to
- 1:23:56and empty list now
- 1:23:58here we should add what the user has asked so here i say messages that append
- 1:24:06It's going to be
- 1:24:08a new message
- 1:24:10Let's copy it from here.
- 1:24:12paste it here okay that is exactly what we need but
- 1:24:17here instead of passing this list i'm going to pass
- 1:24:22this list that I defined up here
- 1:24:25okay now after
- 1:24:26receiving response before returning it to the main function
- 1:24:31do the
- 1:24:32chat function more accurately
- 1:24:36I'm going to store it in a variable called
- 1:24:39response.
- 1:24:41Text.
- 1:24:44Okay, and
- 1:24:46Finally, I'm going to return it.
- 1:24:49Return response text.
- 1:24:51and before that I'm going to add it to this
- 1:24:55list that we have here
- 1:24:57so I say messages.pend
- 1:25:01and here I'm going to have a message
- 1:25:05we need to follow this structure. So
- 1:25:08let me copy it from here and paste it here.
- 1:25:11However, I need to change the role to assistant.
- 1:25:16And instead of message, I should say response text.
- 1:25:21This is what is happening here. First, I have an empty array, an empty list.
- 1:25:28This is my history.
- 1:25:30and on every iteration first i add the query to it
- 1:25:33here i receive the response i extract its text
- 1:25:37and then i'm going to add it to to that history
- 1:25:41and every time that i'm sending the query i'm
- 1:25:45going to pass the whole history okay that is a
- 1:25:49point and that's it let's test our program okay
- 1:25:53Virtual environment is activated. One more time, I ask hello.
- 1:25:57And now I ask what is the capital of France.
- 1:26:03And finally, I ask, what is its population?
- 1:26:09okay we are getting the correct answer and now it's talking about Paris so
- 1:26:15we have successfully implemented a conversation history for our program
- 1:26:21you
- 1:26:27okay in this session i'm going to talk about the
- 1:26:29system role so far we have talked about the user
- 1:26:32role the assistant role but we haven't talked about
- 1:26:35this system role yet the system rule is used
- 1:26:38to define a set of instructions for the model for
- 1:26:41example before we send the rest of messages we can
- 1:26:44tell the model something like this we can say
- 1:26:47you are an assistant respond in this way
- 1:26:50use this format follow these instructions and this
- 1:26:54role is basically for giving instructions and
- 1:26:56usually we put this type of messages at the
- 1:27:00beginning of this list uh your sync here before
- 1:27:03before the rest of the conversation so
- 1:27:06to add a system message to this conversation
- 1:27:10here instead of having an empty list i can add a
- 1:27:14system message so i add a new message the format should be the same
- 1:27:19we have a role and we have
- 1:27:21content so
- 1:27:23Here I say system
- 1:27:25role.
- 1:27:27and my content can be something like you are a helpful assistant you are
- 1:27:34helpful.
- 1:27:36Assistant
- 1:27:38Your name is
- 1:27:40Neonbot?
- 1:27:43And I say
- 1:27:46introduce
- 1:27:48yourself in the first.
- 1:27:51message and in addition I say give
- 1:27:57short and helpful answers.
- 1:28:02Okay, great. And here I'm going to use triple quotes.
- 1:28:08to have it in multiple lines okay great and let's test our application
- 1:28:16I say let's terminate it first I run it one more time
- 1:28:22Okay, again.
- 1:28:24I say hello
- 1:28:26you can see it is introducing itself okay then I'm going to terminate our program
- 1:28:32and here I'm gonna introduce a new topic and that is really important
- 1:28:37as you can here by adding just a simple system message
- 1:28:41we influenced the model's behavior a lot then that is extremely useful
- 1:28:48and here i need to introduce you a new keyword that is foundation models
- 1:28:55What are foundation models? These
- 1:28:57models, these models that we are working with,
- 1:29:00the models that you use in AI chat pods and these AI chat pod that we have now.
- 1:29:06And why are they called foundation models, because they have a broad knowledge,
- 1:29:12broad knowledge across a wide range of domains.
- 1:29:15The point is that in most real world applications,
- 1:29:19we are not going to use a foundation model because
- 1:29:22we are not dealing with a wide range of tasks, a wide
- 1:29:26range of domains, and we usually have a specific task, and we have a
- 1:29:31specific use case in mind. So we need to somehow make the model more specific,
- 1:29:37and we need adapt its behavior to our requirements.
- 1:29:43And this is really important.
- 1:29:45and you saw just one example of it
- 1:29:48that you provided a system prompt and you changed the model's behavior
- 1:29:53and more accurately we can say this falls under the category of prompt engineering
- 1:29:59we change the prompt in order to get the result that we want
- 1:30:05and this is the first solution when we want to change the model behavior
- 1:30:10in many cases that is enough
- 1:30:13can solve the problem.
- 1:30:15and there are other solutions as well for example RAG systems
- 1:30:19fine-tuning and we go to those solutions when prompt engineering is is not enough
- 1:30:25in that case first we try RAG, retrieval
- 1:30:28augmented generation
- 1:30:30RAG is useful when the model doesn't have that
- 1:30:34information that is necessary for that task
- 1:30:37for example we are designing a chat pod for our company and of course
- 1:30:42A foundation model doesn't have access to our company's documentation.
- 1:30:46So we need to provide that knowledge, that documentation for that chatbot.
- 1:30:52And we do this by using a RAG system.
- 1:30:56and if neither of these approaches solves our
- 1:31:00problem we go to fine-tuning and this is
- 1:31:03more complex and it requires additional infrastructure it needs compute resources it
- 1:31:11needs training data and we need to be specialized in this field so these
- 1:31:15are three levels of
- 1:31:16customization first we try prompt engineering and if it's not enough if we need two
- 1:31:24provide external resources.
- 1:31:26If we need to provide additional knowledge, we consider RAG.
- 1:31:30and if the problem is behavioral, if the behavior
- 1:31:34of the model is a problem, we go to fine-tuning.
- 1:31:38And here, as you can see, prompt engineering was enough.
- 1:31:42We just added a system prompt and in
- 1:31:44many scenarios that's the case.
- 1:31:46Some developers try to make it over complicated, but we shouldn't do that.
- 1:31:53First, we should go to prompt engineering and if
- 1:31:55it's not enough, we consider other solutions.
- 1:31:58And here our goal was to create a branded feeling.
- 1:32:03I mean, we want the user to feel it is chatting with a branded AI assistant.
- 1:32:08And that was our goal. And as you can see, we have fulfilled this goal here.
- 1:32:15Ladies and gentlemen, a quick note about the local language.
- 1:32:19On Planet Chatbot, nobody speaks in words, they speak in tokens.
- 1:32:24A token is a small piece of text, sometimes
- 1:32:27a whole word, sometimes just part of one.
- 1:32:30Before any message reaches the LLM, it's split
- 1:32:33into tokens and every token becomes a number.
- 1:32:37Tokens are also the planet's currency.
- 1:32:40Every token you send and every token you receive has a cost.
- 1:32:45So learning to count your tokens is an essential survival skill
- 1:32:48your captain will now explain.
- 1:32:58Okay, the next very important concept that you need to understand is tokens.
- 1:33:03Up until now, I've sometimes referred to tokens
- 1:33:07as just words, just to keep things simple.
- 1:33:09But from this point forward, I want to use the correct terminology.
- 1:33:14In the context of LLMs, the unit we use to
- 1:33:17measure the input and output is called a token.
- 1:33:21You can think of tokens as the basic unit of text that an LLM processes.
- 1:33:26In natural human language, we think in terms
- 1:33:30of words, but for LLMs, we work with tokens.
- 1:33:34Now, why is it important? Why is it to understand this concept?
- 1:33:38There are several different reasons.
- 1:33:40One of the most important reasons is cost management.
- 1:33:44The provider you are using charges you based on the number of tokens you use,
- 1:33:48the number tokens you send to the model and the number the model generates.
- 1:33:53Another important reason is performance analysis
- 1:33:56For example, if you want to compare several models
- 1:33:59decide which one is the best option for you, for your application
- 1:34:03and in these scenarios, you need to understand how token usage works
- 1:34:07So now let's see an example, let's what a token actually is
- 1:34:12Imagine we send something like this to our LLM I love AI
- 1:34:17Before the model processes this piece of text, it gets broken down into tokens
- 1:34:22the sentence I told you I love AI
- 1:34:24for a simple example it could be something like this
- 1:34:28I one token love one token and AI another token but this is the important part
- 1:34:34a token is not necessarily a complete word a token can be a complete world
- 1:34:40part of a word or even a punctuation mark
- 1:34:43for example you can consider
- 1:34:46a word like interesting it could
- 1:34:49something like this inter one token esting another token so
- 1:34:56don't think of them as exactly as as the same thing as words
- 1:35:01and i can say as a rough approximation you can consider one token
- 1:35:06around three quarters of a word in English language
- 1:35:10but remember this is just an approximation
- 1:35:13one thing that you need to have in your mind is that
- 1:35:17a token can be a whole word,
- 1:35:19part of a word, or a punctuation mark
- 1:35:21you might be wondering why do I keep using can be, could be
- 1:35:26why don't I just tell you how exactly a word is going to be spill it into tokens
- 1:35:31the reason is that different models use different tokenizers
- 1:35:35and each model has its own vocabulary of tokens
- 1:35:39and the model uses this vocabulary when it processes the text every LLM
- 1:35:46has a vocabulary, a collection of tokens. And suppose I give the model this input.
- 1:35:52I am.
- 1:35:53And now the model needs to predict the word that comes next.
- 1:35:57And the model looks at the context of the sentence and
- 1:36:02considers possible next tokens from its vocabulary.
- 1:36:07For example, possible candidates might be tired, happy, sad, and went.
- 1:36:14and each of these possible next tokens gets a probability
- 1:36:18for example, purely for illustration, purely for example I can say
- 1:36:22tired gets 20%, happy gets 21%
- 1:36:28sad gets 10% and went gets 0.00001%
- 1:36:34these numbers are completely made up just for example
- 1:36:38but in this example you saw that I gave a very low probability to went
- 1:36:44because it doesn't fit grammatically
- 1:36:47but it still have a probability and it might be chosen
- 1:36:53and this collection of possible tokens is called a probability distribution
- 1:36:58the model uses this probability distribution to select the next token
- 1:37:03so in this way, HAPPY has a much higher probability in comparison with WENT
- 1:37:10but the point is that WENT still has a chance
- 1:37:14and it's not necessarily impossible and
- 1:37:18uh this probabilistic process this probabilisting nature is one of the
- 1:37:24reasons LLMs can behave in a non-deterministic way
- 1:37:28can be creative in other words the exact details are of course much more
- 1:37:33complicated more complicated that than i simplified it for you but uh
- 1:37:38conceptually this is what's happening the model has a vocabulary of
- 1:37:42possible tokens and it processes the context
- 1:37:46performs a lot of mathematical calculations
- 1:37:50assigns probabilities to possible next tokens
- 1:37:53and then it selects one of those tokens you need to
- 1:37:57remember the parameters i talked about temperature
- 1:38:01and top p these parameters influence this process
- 1:38:04at exactly this stage where the model is working
- 1:38:08with this probability distribution and it's
- 1:38:11deciding which token to generate next okay
- 1:38:14Now there is one more important thing to understand.
- 1:38:18The tokens we talked about aren't what the model actually receives.
- 1:38:22Each token is mapped to a numerical representation, a numerical identifier.
- 1:38:27And it is called a token ID.
- 1:38:29For example, again purely for our example, I can be converted to 101.
- 1:38:36Love can be convert to, for example, 409.
- 1:38:40AI782 and just these examples.
- 1:38:44So ultimately the text we provide, for example, I love AI,
- 1:38:48is converted into an array of token IDs. Finally, that array of
- 1:38:53token IDs is passed to the model and
- 1:38:56The component responsible for doing that is called tokenizer.
- 1:39:01Tokenizer maps a piece of text into tokenize these.
- 1:39:06So the overall process again looks something like this. We have text.
- 1:39:11Then we have tokenization process
- 1:39:14We have tokens Then we token IDs
- 1:39:17And these token IDs are sent to the LLM
- 1:39:21Now, the good news for you is that you usually
- 1:39:24don't need to work with the tokenizer directly.
- 1:39:27In most AI engineering applications, you rarely have to manually tokenize your text.
- 1:39:32And even if you decide to run a model locally, the
- 1:39:36tokeniser usually comes with the model itself.
- 1:39:39And it is provided alongside it, so you don't have to be worried about it.
- 1:39:46So, for now,
- 1:39:47the main thing I want you to remember is
- 1:39:50your text goes through tokenization process
- 1:39:54it gets converted into tokens and these tokens are mapped to
- 1:39:58token ids and that's a basic process and as i mentioned earlier one of the
- 1:40:03first places where understanding tokens becomes really important is cost
- 1:40:08management for example if you look at here look at the open ai pricing
- 1:40:13page you will notice that pricing is based
- 1:40:16on tokens it's saying that i'm charging you
- 1:40:19based on tokens, you might see something like this
- 1:40:22$10 per 1 million token and $50 for output tokens
- 1:40:28So, input tokens are the tokens contained in the text you're sending to the LLM
- 1:40:34your input, your query, and of course, if you're designing your application
- 1:40:39the history is taken into account
- 1:40:42and the response that is generated, the tokens that are used by the LLM to generate
- 1:40:48The response is called output tokens.
- 1:40:51And as you can see here on this pricing page, output
- 1:40:55tokens are more expensive than input tokens.
- 1:40:58And as an AI engineer, tokens need to be important to you.
- 1:41:02You need to take care of things like cost management,
- 1:41:06model limits, context window, performance analysis.
- 1:41:10All these things depend on keeping track of the number of tokens we use.
- 1:41:16And from this point forward, whenever we talk
- 1:41:19about the amount of text, the LLM processes,
- 1:41:22I want you to think in terms of tokens, not just words.
- 1:41:27Oh
- 1:41:33When it comes to counting tokens, there are two
- 1:41:35different places where we might want to count them.
- 1:41:38One is before, when it's before we send the request
- 1:41:41and the other is after we get the response.
- 1:41:44Okay, so let's start with the latter one, with the second one.
- 1:41:49After we receive a response from the LLM, we
- 1:41:52might want to know how many tokens were used.
- 1:41:54For example, we want to how many input tokens did we
- 1:41:58use, how many output tokens did the model generate,
- 1:42:02and finally how many token were used in total. Input tokens plus output tokens.
- 1:42:08Okay, this is actually the easier case because you send a query to an LL,
- 1:42:13the LLL returns that number, it returns a response and
- 1:42:18in the response metadata, the API tells you how many tokens were used
- 1:42:22so this is pretty straightforward, you only need to extract that number
- 1:42:27and as I told you, the other case is when we want to count the number of tokens
- 1:42:33before sending the request and there can be different reasons
- 1:42:37maybe we want to check how many tokens we have before sending a request
- 1:42:41maybe we wanna set a limit for example, in cases that the input is too large
- 1:42:47we may want to prevent the user from sending that request
- 1:42:51in these cases we need to know how many token we have before sending request
- 1:42:58and this is a little more complicated
- 1:43:01because as I mentioned, different models have different tokenizers
- 1:43:05so to count the tokens before sending the request
- 1:43:09we need to have that tokenizer and we need tokenize the text
- 1:43:13so you have many additional steps
- 1:43:16you need to take your text, pass it through the proper tokenizer
- 1:43:21and then after tokenizing it you need to count the number of
- 1:43:25tokens and store it somewhere and use it however you need
- 1:43:29anyway in this session we are going to focus on the second case the easier one
- 1:43:34I'm going to show you where this token where
- 1:43:36this metadata is located in the response
- 1:43:38and then we are gonna keep track of the number tokens
- 1:43:43that is consumed throughout our conversation okay
- 1:43:47and later I will show you how we can calculate the actual cost
- 1:43:52because it was one of the reasons that we wanted to keep track of used tokens
- 1:43:58and at that point I will you how to keep input and output tokens separately
- 1:44:04because the cost is different for input tokens and output token
- 1:44:08in this session we only want to keep the total
- 1:44:12tokens that is used during our conversation
- 1:44:15So first I need to have a variable, have a
- 1:44:20variable to keep track of this number.
- 1:44:23to add the number of used tokens on each iteration of our conversation loop.
- 1:44:29So here I'm going to add a new global variable alongside our other global variables,
- 1:44:35and I'm gonna call it total tokens.
- 1:44:41and I set it to zero.
- 1:44:44now here inside our conversation loop here when we
- 1:44:48get the response i'm going to add the number of
- 1:44:51tokens the number tokens used to this variable
- 1:44:55so here that we are getting the response there
- 1:44:59is a field inside called usage
- 1:45:01and inside that we have total tokens and it represents the number
- 1:45:05of tokens that are used in total so every time we receive the response from the llm
- 1:45:11We extract this value and we add it.
- 1:45:14So here, before returning the response, I say total tokens
- 1:45:20is equal to total tokens plus this number.
- 1:45:25response.usage.
- 1:45:29total tokens and as you can see it is suggesting that
- 1:45:32now I need to tell you that we have a small programming issue here
- 1:45:37Python sees that we are assigning a value to total tokens inside this function.
- 1:45:45and by default
- 1:45:46Python assumes that total tokens is a local variable
- 1:45:51something different from this total tokens that we have up here
- 1:45:55what happens is that on the right hand side we are trying to access total tokens
- 1:46:00before this local variable has been assigned a value
- 1:46:04okay that is a problem
- 1:46:06and that's the reason
- 1:46:08Python will complain about it and this is one of the reasons
- 1:46:12it is told that using global variables is not a good approach.
- 1:46:18But later when we refactor our application, when we make our
- 1:46:23application object oriented, we will solve this problem properly.
- 1:46:27But for now, as a temporary solution, we can explicitly tell Python that we
- 1:46:34want to use the global variable that we defined up here.
- 1:46:39And to do that, it's enough to say
- 1:46:42global before our global variable and this way.
- 1:46:47Python won't create a new local variable called total tokens inside this function.
- 1:46:52Instead, total tokens
- 1:46:54inside this functions refers to the same global
- 1:46:58total tokens variable that we defined earlier.
- 1:47:01Okay, now I want to use it.
- 1:47:03I want print the total number of consume tokens
- 1:47:07after the user decides to end the conversation.
- 1:47:11So after the chat function is called,
- 1:47:15I print and here I use a formatted string and I say consumed tokens
- 1:47:21colon space and here
- 1:47:24I use the placeholder and I put
- 1:47:27total tokens inside. Now let's run our program.
- 1:47:32Okay, I say hello.
- 1:47:34What is the capital?
- 1:47:40France.
- 1:47:42And now I terminate the program and as you can see, the program tells
- 1:47:47us the total number of tokens we consumed during the conversation.
- 1:47:52So now we are not only having a conversation with the LLM, we
- 1:47:56are also tracking how many tokens that conversation is using.
- 1:48:02Oh
- 1:48:07Okay, as you may have realized, our program is growing.
- 1:48:11We are adding new features to it. And if we continue this pattern, I mean, if we
- 1:48:16continue growing the application with the same architecture,
- 1:48:19it's eventually going to become difficult to maintain.
- 1:48:22And you actually saw an example of it in the previous session.
- 1:48:26We reached a point where even though we didn't really want to,
- 1:48:30we had to use a global variable. So now,
- 1:48:33Let's move on to the proper solution and that
- 1:48:37solution is object-oriented programming.
- 1:48:40and at the scale of this project object-oriented programming is enough
- 1:48:45this structure solely is enough, it's more than enough to solve our problem
- 1:48:50and give us the structure that we need and good for our program
- 1:48:54So let's start prefactoring our application. First, I'm going to add a class here.
- 1:49:01I add a new class called
- 1:49:03Chatbot
- 1:49:05and inside I'm going to
- 1:49:07define a constructor for it. So I say def
- 1:49:12in it and
- 1:49:13here we go and here as you can see we have three
- 1:49:19global variables and these were for
- 1:49:23these were our previous implementation and in new
- 1:49:27implementation we need to take these three variables
- 1:49:31and we want to move them inside this constructor and we
- 1:49:36are going to make them the attributes of this class so
- 1:49:41let me fix the problem with indentation now it is okay
- 1:49:46and the only thing left is to add self
- 1:49:50at the beginning of each one oneself here another
- 1:49:54one and another one here now
- 1:49:57i can access these
- 1:49:58attributes from the other methods that i'm going to define inside this class
- 1:50:03what other methods do we need exactly the same functionality we had
- 1:50:07we want that we had a bot function and we had the chat function
- 1:50:13okay what's wrong here
- 1:50:15all messages. Now it's okay.
- 1:50:17It's complaining about that we don't have such a
- 1:50:20variable and that's great because you're refactoring.
- 1:50:25Okay, we have a bot function here and we have the chatbot function here.
- 1:50:28And we are going to move these two functions inside
- 1:50:31our class and make them methods of this class.
- 1:50:35So first I do that for the bot method.
- 1:50:44Okay, let's
- 1:50:46fix the problem
- 1:50:48again with indentation okay now it's okay
- 1:50:52okay I'm
- 1:50:52not going to need this line anymore
- 1:50:55we did all of
- 1:50:56it to get rid of this global variable
- 1:51:00okay and I add self here
- 1:51:04And one here.
- 1:51:05places that it's complaining self.
- 1:51:09dot oh we need to put
- 1:51:13this self as a first parameter of this method as well oneself here
- 1:51:19ok and
- 1:51:21I guess now it's okay.
- 1:51:24yes we need another one here because here it's trying to define
- 1:51:30a
- 1:51:31local total tokens variable here so
- 1:51:34we don't want that so
- 1:51:38and instead of this syntax I use plus equal. Now it's much better.
- 1:51:44Okay, after this I'm going to move
- 1:51:48the chat function inside this class, okay?
- 1:51:53ok the indentation is ok and now it's a method
- 1:51:57the only thing is to add self keyword here and here
- 1:52:03we need to use self before calling this method of the class
- 1:52:08okay now
- 1:52:09it's perfect
- 1:52:11okay now we have these methods inside our class bot method and chat method
- 1:52:19bot handles communication with the LLM and chat handles the conversation loop
- 1:52:24now to actually use these methods first we need to create an object we need
- 1:52:31to create a object from our chatbot class so
- 1:52:35inside the main function where it is complaining about
- 1:52:39this function that no longer exists
- 1:52:44instead of this line we want to create
- 1:52:48an object from this way so I say chatbot here
- 1:52:52parenthesis and now we have an object and I store it in a
- 1:52:56variable called bot. Now I can access and now I have an
- 1:53:02object that I can call these methods on it these two methods that I just defined
- 1:53:10OK, here, messages again, we need a self.
- 1:53:14Save that messages.
- 1:53:17okay yes as i told you i call the chat method on this object and my program
- 1:53:26will work like before hopefully we will test it and we will see
- 1:53:32Now the only thing left is that we still have one line
- 1:53:36here and this is from our previous implementation
- 1:53:40and we no longer need such a variable.
- 1:53:43We no longer have this total tokens here and it is complaining about this matter.
- 1:53:50So we need to do this somehow else.
- 1:53:53I mean rather than keeping this logic outside the class it's better to add a
- 1:53:59new method in our chatbot class whose job is to generate a report.
- 1:54:03By doing this we can also follow an important
- 1:54:07principle in software engineering better.
- 1:54:10This principle is called separation of concerns.
- 1:54:15And the idea is that each part of our application
- 1:54:18should have a clear responsibility.
- 1:54:21For example, here as I told you we have chat
- 1:54:24method that handles the conversation and
- 1:54:27We have bot method that handles the communication.
- 1:54:31And our new report method is going to handle reporting and observability.
- 1:54:38Observability is very important. It's very important in real world applications.
- 1:54:42Whether you are developing the system in the development phase,
- 1:54:47whether you are running it in production.
- 1:54:49In both cases, you want to be able to understand
- 1:54:53what's happening inside your system.
- 1:54:55What's going on.
- 1:54:56so that's really important and you want to be able
- 1:55:01to inspect things like requests responses errors
- 1:55:05token usage and other important metrics so that really important
- 1:55:10and let's do it let's add a new method to this class whose responsibility is
- 1:55:17reporting and observability so i call
- 1:55:21it report i put uh i put self as its
- 1:55:25first parameter and first I'm going to print
- 1:55:29each message in our conversation history. So I say for
- 1:55:34message in self that
- 1:55:37messages
- 1:55:39I'm going to print
- 1:55:41the message
- 1:55:44and after I'm going to print the number of tokens that is used so I just take it and
- 1:55:53paste it here
- 1:55:55Okay, and I put a self at the beginning. Great.
- 1:56:00So now we have a dedicated method responsible for generating the report,
- 1:56:05the token number and the messages that we have in conversation history.
- 1:56:11And finally, we call this method after the conversation loop.
- 1:56:16I say bot.
- 1:56:18report okay,
- 1:56:19so now the overall structure of our application is much better
- 1:56:25it is much cleaner we have a chatbot class and it contains
- 1:56:31three methods and the state of our
- 1:56:34application, I mean the attributes that our application has
- 1:56:39this is much, this is better because each method has a clear responsibility
- 1:56:44and if somewhere needs to be
- 1:56:47changed we go to that specific part of our
- 1:56:50application and now we have a better foundation
- 1:56:53for
- 1:56:54continuing to add new features to our application
- 1:56:57but before finishing this session let's
- 1:57:01test our program to see whether it uh whether
- 1:57:04it works like before or not so i say hello
- 1:57:09What is it?
- 1:57:10Capital of
- 1:57:12France.
- 1:57:14what is its
- 1:57:16population
- 1:57:18Okay, great. It works like before and I terminate the program and
- 1:57:22yeah, the new functionality that we just added.
- 1:57:26Conversation history is printed here.
- 1:57:30messages with different roles, first user, then assistant, first user,
- 1:57:34then assistant. And, um,
- 1:57:37Finally, the number of used tokens is printed here, exactly what we wanted.
- 1:57:44Okay, great.
- 1:57:46you
- 1:57:52okay in this session we are going to add another cool feature to our application
- 1:57:57to the report method we just created at the end of the last session
- 1:58:01and so far our report is able to count the total number of tokens we have consumed
- 1:58:08now we want to take it one step further
- 1:58:12and we want it capable of calculating the cost
- 1:58:16we want calculate the cost as well
- 1:58:19and for this example we will use some hypothetical
- 1:58:23pricing of course if you are working on a real
- 1:58:27project if you're working with a real model you
- 1:58:30need to you need check the pricing and it depends
- 1:58:33on your provider so it's a responsibility for you
- 1:58:36you need find out how much you are charged per
- 1:58:39million input tokens and per million output tokens the
- 1:58:43first step that we need is to track these two separately
- 1:58:47Currently, we are counting only total tokens,
- 1:58:50but we need to keep track of input tokens and
- 1:58:53output tokens separately because as you saw,
- 1:58:57input tokens, and output tokens usually have
- 1:59:00different prices.
- 1:59:02So at the first step, I'm going to add two new attributes to the constructor.
- 1:59:09Here I'm going to add a new
- 1:59:11attribute called input tokens
- 1:59:16And another one.
- 1:59:17called
- 1:59:19output
- 1:59:22tokens
- 1:59:23and I set it to 0 as well.
- 1:59:26Okay now when we get the response from the LLM
- 1:59:29we need to keep track of the input and output
- 1:59:32tokens as well so here
- 1:59:37when we are receiving total tokens let me just duplicate this line
- 1:59:44two times and
- 1:59:46modify the left hand side. I
- 1:59:48change it to Input Tokens.
- 1:59:52and I change this one to output tokens the attribute the attributes we just defined
- 2:00:00and in these cases pay close attention i mean when you copy paste
- 2:00:05always pay close the attention it's very easy to accidentally to accidentally
- 2:00:10introduce a small mistake to a program so you should be really cautious
- 2:00:14now the question is that which attribute
- 2:00:17inside response.usage contains the number of
- 2:00:21input tokens and output tokens as well for input tokens it's prompt tokens
- 2:00:27So here we should say prompt tokens.
- 2:00:31And of course, the exact attribute names can be different depending on the
- 2:00:35wrapper or the library you are using.
- 2:00:37But in the OpenAI wrapper, the number of
- 2:00:40tokens sent to the model is available through
- 2:00:44prompt tokens and usually you don't need to memorize it.
- 2:00:47And after
- 2:00:48using a wrapper or a library, you will get used to it.
- 2:00:52So don't worry about it.
- 2:00:54and for output tokens we have completion
- 2:00:58completion tokens in other words these are the
- 2:01:02tokens that were generated to produce your response
- 2:01:06now we have everything that we need to calculate the
- 2:01:10cost and as i told you we are going to use some
- 2:01:14made up pricing
- 2:01:15let's assume for this example that we are charged five cents five cents per
- 2:01:22million input tokens and eight cents
- 2:01:24twice i mean
- 2:01:26per million output tokens. So we can calculate the cost like this.
- 2:01:32Here I say self.input
- 2:01:36tokens times
- 2:01:390.04 I said 4 or 5, okay
- 2:01:44it's not important, 5, ok plus self
- 2:01:50Output tokens times
- 2:01:530.08 Okay, a
- 2:01:55tiny fraction.
- 2:01:58And we want to print this. So we say print
- 2:02:04and I put it inside and let's add a dollar sign before
- 2:02:10so I use a formatted string
- 2:02:13Okay.
- 2:02:16And here I add dollar sign and
- 2:02:19This needs to be inside a placeholder. Great.
- 2:02:24But there is one important detail here.
- 2:02:26Remember that these rates are per 1 million tokens. So we need to convert it.
- 2:02:31We need to divide it by 1 million.
- 2:02:35So I put the whole expression inside parentheses.
- 2:02:41And then I divide it by 1,000,000.
- 2:02:45Okay, 6.
- 2:02:47And to increase its readability, I add underscores here.
- 2:02:52I'm using these underscores to simply make it easier to read.
- 2:02:57Python allows these unders scores inside numeric literals.
- 2:03:03So this is exactly the same as writing what we had before.
- 2:03:11It's just for reading your number better.
- 2:03:14And now, we need nothing else.
- 2:03:16Let's just
- 2:03:17test our program. I run it one more time.
- 2:03:21Hello?
- 2:03:23How are you?
- 2:03:26OK and it
- 2:03:28enter quit,
- 2:03:30press and as you can see the cost here is calculated for us
- 2:03:35and as we can see, the result is displayed in scientific notation
- 2:03:41this negative e
- 2:03:44This simply means the number is extremely small.
- 2:03:47In this example, we are only using a few tokens.
- 2:03:50So the actual cost is just a tiny fraction of a cent, which is really, really low.
- 2:03:57you
- 2:04:02Now let's move on to the first approach, to the first approach I mentioned.
- 2:04:06I mentioned that there are two places we want to count the token number.
- 2:04:11One of which is before, before we send the user's query to the LLM and one is after.
- 2:04:17In the two previous sessions, we covered the second one.
- 2:04:21And now we want cover the first one, the situation when we want count
- 2:04:26the tokens before we send the user's query to the LLM.
- 2:04:29And there are certain situations we need that.
- 2:04:32We need to track that, track that number.
- 2:04:35And the important question is how can we take a
- 2:04:38piece of text and break it down into tokens.
- 2:04:41As you remember, I mentioned that every model has its own tokenizer.
- 2:04:45For example, OpenAI has its tokenizer, okay.
- 2:04:48another model may use a different tokenizer and currently we are using
- 2:04:53an open AI model so we are going to use the open AI tokenizer but
- 2:04:59what if what if we were using a different model of
- 2:05:02course in that case we would we would need that's
- 2:05:05model tokenizer and we would to go to hugging face
- 2:05:09find the tokenizer associated with that model
- 2:05:12and use it so for now for open AI models I use
- 2:05:15approach appropriate for that I mean using
- 2:05:18the tokenized are associated with open AI and later I will show you
- 2:05:23how to find and access the tokenizer that is for another model that you might use
- 2:05:28so now how can we do that for OpenAI?
- 2:05:31if you're working with OpenAI models you can use TIKTOKEN
- 2:05:34there is a package called tiktoken
- 2:05:36you install it and you can tokenize any piece of text
- 2:05:41so first let me check yeah I'm connected to the internet
- 2:05:45so I'm going to install this new package open terminal and I say
- 2:05:51pipenv.
- 2:05:52install
- 2:05:54tick token okay the package is installed and now
- 2:05:59i'm going to import it so up here i import
- 2:06:04tick token and then at the beginning of my chat method we are going to use it we are
- 2:06:09going to tokenize the piece of text that the user has entered the tick token
- 2:06:14package provides a function called get encoding
- 2:06:17we need an encoding and which encoding
- 2:06:20do we need for this example we are gonna use
- 2:06:23cl100k base and the encoding depends on the
- 2:06:27model the exact model from openai that that you are using so
- 2:06:31up here at the beginning of this method i'm going to use this function
- 2:06:37get encoding function and I'm going to pass what I said cl100k
- 2:06:45base
- 2:06:46okay and we need to keep the results somewhere so i store it in a variable called
- 2:06:52encoding now this encoding object that you are seeing here
- 2:06:57provides me with a method called encode and we can call encode
- 2:07:01pass it the text that we want to tokenize and it will give
- 2:07:05us a token ids and we only need to count them it gives us
- 2:07:09an array and we just need to account the number of elements inside so here
- 2:07:14Before sending that text to the bot method, I'm going to tokenize it.
- 2:07:19So I say.
- 2:07:22encode and I pass
- 2:07:24user input
- 2:07:26and here I just print it
- 2:07:30So I print it
- 2:07:31Now let's
- 2:07:34Let's run the program.
- 2:07:36Okay, I say, now instead of hello, let's say
- 2:07:40which is the capital
- 2:07:43of France.
- 2:07:46and now as you can see before the response we are seeing the token IDs
- 2:07:53for example this token ID that you are seeing here is associated with what
- 2:07:59and this is for is this is there
- 2:08:02now instead of just seeing token IDs we want to know how many tokens we have
- 2:08:08that's pretty easy as a result
- 2:08:10that you are seeing here is just a regular Python list.
- 2:08:13So we can use Python's built-in LEN function to
- 2:08:17count the number of elements inside this list.
- 2:08:20So let's terminate and
- 2:08:22here instead of printing the list itself, I'm going to put it inside
- 2:08:28the len function.
- 2:08:31Okay, now let's run our program one more time. I say hello.
- 2:08:37It says one. Okay, so we made it.
- 2:08:40We can count the number of tokens before we send our request.
- 2:08:44Now, the question is, the question is what is the benefit? What is the
- 2:08:48benefit of knowing the number of tokens before sending the request to the LLM?
- 2:08:52There are actually several possible use cases.
- 2:08:54For example, you can use this as a guard.
- 2:08:58we could say if the number of tokens in the
- 2:09:01user's input is greater than a certain limit
- 2:09:03don't send a request to the LLM.
- 2:09:06For example, let's say we only want to allow prompts within
- 2:09:10a maximum of 10 tokens. For instance,
- 2:09:14So let's do that. First, I'll store this number in a variable called token count.
- 2:09:22Okay.
- 2:09:23Great. And then we can
- 2:09:27add an if statement so i say if
- 2:09:30token count is greater than 10. if it exceeds the limit
- 2:09:37i print something like this i can say your prompt
- 2:09:42exceeded the limit.
- 2:09:46and can say try
- 2:09:49a shorter one.
- 2:09:51please and after exceeding this limit we don't want the program to continue and
- 2:09:57send this prompt to the LLM with this implementation
- 2:10:01it does but we don' t want to do that so here
- 2:10:05we need to use the continue keyword here I say continue
- 2:10:10and by doing this it moves to the next iteration
- 2:10:14and this gives the user another opportunity to enter a shorter prompt
- 2:10:19so let's run our program and test it
- 2:10:23okay here I say hello
- 2:10:24I didn't exit the limit now let me add some gibberish
- 2:10:32to shortly test it. Something more than 10 tokens.
- 2:10:35Okay, now it's definitely more than 10 tokens, okay. As you can see, it says here,
- 2:10:40prompt exceeded the limit and
- 2:10:42That was exactly what we wanted.
- 2:10:45Oh
- 2:10:51Okay, in the previous session, our model
- 2:10:54happened to be from OpenAI, from Open AI family
- 2:10:57and I showed you how to use tiktoken to tokenize your text.
- 2:11:01But what happens if the model we're using isn't OpenAI? Isn't from OpenAi family.
- 2:11:06For example, what if it's Llama or any other model you found on HuggingFace?
- 2:11:11In these situations, the first approach is to relax our requirements a little bit.
- 2:11:17It's true that different models use different tokenizers,
- 2:11:20but in many cases, the difference is not significant.
- 2:11:23It's negligible. For most tasks, we don't really need the exact token number.
- 2:11:29You only need an approximation.
- 2:11:31For example, the OpenAI tokenizer might tell you a
- 2:11:34piece of text contains 10 tokens, for example,
- 2:11:38while the actual tokenizer for your model you're using might count.
- 2:11:448 tokens instead of 10 tokens or 12 tokens so it's not significant
- 2:11:48and that's perfectly fine for most cases so if your task is not sensitive
- 2:11:54if you don't need the exact token number you can simply use TIKTOKEN
- 2:11:58because it's convenient and
- 2:12:00it's easy to work with
- 2:12:02but let's assume that for whatever reason you really need access the tokenizer
- 2:12:08for the exact model you're using for example, let's say our model is Llama
- 2:12:13and we need the exact number. So in these cases, what do we need to do?
- 2:12:19Well, in these situations, first we need transformers,
- 2:12:22and transformers is a library from HuggingFace.
- 2:12:26Not only for this case, I mean tokenization, but for many other
- 2:12:29things, for example, downloading a model, running a model.
- 2:12:32We also need transformes for many tasks.
- 2:12:36It's really worth becoming familiar with it.
- 2:12:38And
- 2:12:38one important thing here is that you need an access token.
- 2:12:42I'm not completely sure whether an access token is
- 2:12:45required here for this tokenization task or not
- 2:12:48but for many other tasks it is required and in some cases it warns you
- 2:12:53it warn you that you are not using a token
- 2:12:56and if you use a token you can have higher
- 2:12:58limits or better access and things like that
- 2:13:01besides some models require you to be authenticated
- 2:13:06otherwise you cannot access that
- 2:13:09model uh it's really good
- 2:13:11create your access token right now go to settings
- 2:13:14create access token and have an access token
- 2:13:16and i've already done that this is my access token
- 2:13:20okay so for the reason i told you i copy this and
- 2:13:24i put it inside my dot n file here i create another
- 2:13:29environment variable called hf token and i set it to
- 2:13:35my HuggingFaceAccessToken and make sure you use exactly this name
- 2:13:39I mean hf underscore token because we are not
- 2:13:42going to read this environment variable directly
- 2:13:45we are NOT going to write it ourselves in our python code
- 2:13:48the Hugging Face Transformers library is going to
- 2:13:51read it and it does it automatically so the approach we read this token is not like
- 2:13:56something like this os.get.env rather than the transformers is going
- 2:14:01to do that so um
- 2:14:04First, I'm going to install this library, this library that I told you.
- 2:14:08So I say pipenv
- 2:14:10install
- 2:14:12transformers. Once that's installed we need to import it.
- 2:14:17So I'm going to import AutoTokenizer class from a transformer. So I say from
- 2:14:24Transformers import
- 2:14:27Auto tokenizer
- 2:14:30now instead of using tick token instead of
- 2:14:33using tick to get here i'm i'm going to use
- 2:14:36this auto tokenizer class that i just
- 2:14:40imported impo i just imported and instead of
- 2:14:44get encoding i'm used i'm gonna use from
- 2:14:48pre-trained, okay, let's remove it
- 2:14:51entirely.
- 2:14:53from pre-trained
- 2:14:55Okay, this method takes the name of the model you want to use.
- 2:14:59So I head over to here, I copy the model name, I paste it and that's it.
- 2:15:06and pay close attention the model identifier needs to be
- 2:15:10the correct identifier that you copy from here
- 2:15:13don't just guess it don't copy it from the provider you're using for example
- 2:15:18Groq it needs to this exact identifier because you're
- 2:15:22using transformers and transformers is from
- 2:15:25hugging face so pay attention to that okay
- 2:15:28now there is nothing left to change and here we can use it like before here i'm
- 2:15:34I print it, I print.
- 2:15:36token count.
- 2:15:39Okay, I run my code to test it. Okay, as you can see, it displays a warning here.
- 2:15:46It's telling that PyTorch should also be installed if you want to use the models.
- 2:15:51But for now, for our purpose, that's perfectly fine
- 2:15:54because we are not trying to load or run the model. We are just using the tokenizer.
- 2:16:00We only want the tokenized. Let's say hello. And as you see, it's working correctly.
- 2:16:07What is it?
- 2:16:08Capital.
- 2:16:10Oh, fuck.
- 2:16:11France
- 2:16:13And as you can see, it's counting token number.
- 2:16:16Here we are using an open AI model, but we are using here, um,
- 2:16:22Llama's tokenizer to count tokens.
- 2:16:25So as you saw, we successfully learned how to use the
- 2:16:29exact tokenizer for the exact model we are using.
- 2:16:32Although here we are an open AI model, we are not using LamaBot.
- 2:16:37I showed you how to use Llama's tokenizer if you wanted to use Llama model.
- 2:16:41And now before finishing the session, let's do something a little more interesting.
- 2:16:47Let's add a limit for the total total tokens.
- 2:16:50total number of tokens our chatbot is allowed to consume
- 2:16:54and as you remember here
- 2:16:57we have an attribute called total tokens so
- 2:17:01as the conversation
- 2:17:03continues and more messages are exchanged this number keeps increasing
- 2:17:08and we can put
- 2:17:09a limit on it in practice you will find that in
- 2:17:12many services that you may use many ai services
- 2:17:15that you might use and you can in those services
- 2:17:19you can consume a certain number of tokens within
- 2:17:22a certain period of time and let's implement that let's a simple version of that
- 2:17:28we only need to add a condition so I say
- 2:17:33if self.
- 2:17:36Why isn't suggesting anything? I don't know
- 2:17:39total tokens
- 2:17:43Great here, then.
- 2:17:44300 or we can just say the 300 is okay okay we will test it
- 2:17:50and in that case I'll print
- 2:17:54Unfortunately
- 2:17:57you
- 2:17:59Exceeded.
- 2:18:01the
- 2:18:02token limit.
- 2:18:04for today.
- 2:18:05and here instead of using the continue keyboard I'm going to use
- 2:18:09the break keyboard because I want to terminate the program
- 2:18:13because we want to terminate our program and
- 2:18:16we don't want to give the user another chance
- 2:18:20and now here as you can see we have two different limits
- 2:18:24the first one checks the number of tokens in the current user prompt
- 2:18:29and if it exceeds 10 tokens, reject that prompt and give the user another chance
- 2:18:37the second one checks the total number consumed
- 2:18:41if it exceed 300, we stop the chat
- 2:18:47and
- 2:18:49Let's test our program. Let's say hello.
- 2:18:53Okay.
- 2:18:55What is the...
- 2:18:58Capital of
- 2:18:59France.
- 2:19:02What is its population?
- 2:19:06okay as you can see here it says unfortunately
- 2:19:10we exceeded the total token limit for today and
- 2:19:15it automatically terminated the program and after
- 2:19:19that we see the report so that's okay and let's
- 2:19:24remove this line that we kept printing the number of token in each query okay
- 2:19:32So this is a simple example, but the idea is the same.
- 2:19:36The idea is same for more sophisticated scenarios.
- 2:19:39And we learned how to have a simple token management system,
- 2:19:44and we can extend it for real world applications.
- 2:19:50you
- 2:19:56Okay, in this session I want to show you how to
- 2:19:58stream the response we receive from the LLM
- 2:20:01So far our application waits for the LLL to generate the entire response
- 2:20:06And then we wait for it to generate entire response
- 2:20:11and we return the complete answer to the user
- 2:20:14But now we want to change this behavior
- 2:20:17Instead of waiting for the entire response, I want
- 2:20:21the LLM to send us what it has generated so far
- 2:20:24and we want to display it to the user as it arrives
- 2:20:27so to do that it's not difficult at all
- 2:20:31we need to set another argument called stream and i set it to true
- 2:20:36by default it is false which means we don't want to stream the response
- 2:20:40but we want do it once we do it the response is no longer returned as before
- 2:20:46previously we could access a response through something like this
- 2:20:51choices.message.content. But when we enable streaming, response becomes an iterable.
- 2:20:58So we have an iterable that we can loop over.
- 2:21:01So for now, I'm going to comment out
- 2:21:05these lines here. And as I mentioned,
- 2:21:09response that we get now is an iterable we can
- 2:21:12we can iterate over it and in each iteration it
- 2:21:17it gives us a chunk and a chunk basically contains a piece of a piece
- 2:21:22of the content that the LLM has generated and
- 2:21:25sent back to us so since response is iterable
- 2:21:29i'm going to loop over it. and i do this using a for loop so i say for
- 2:21:36and I call the loop variable chunk.
- 2:21:40Say chunking.
- 2:21:42response and for each chunk we still have this part i mean choices
- 2:21:48however we no longer have
- 2:21:50message instead we have delta so
- 2:21:53Let's copy it.
- 2:21:56here.
- 2:21:59paste it here instead of response I say chunk and instead of message I say delta
- 2:22:07okay great here i'm also going to add a guard
- 2:22:11to make sure that choices is not mpt choices
- 2:22:15actually exists
- 2:22:16before before trying to access its first item if it doesn't exist i don't want to do
- 2:22:23anything but if it exists i want to access it's first element so here let's say if
- 2:22:31chunk
- 2:22:33choices.
- 2:22:36in that case i'm going to i'm gonna access that and i'm
- 2:22:40going to store it in a variable called delta okay so this delta represents the
- 2:22:47new piece of content generated by the llm it might be a token part of a word a word
- 2:22:53a longer piece of text whatever it really depends on that moment that the ll m is
- 2:22:59generating and now here i'm gong to add another guard
- 2:23:04I'm going to say if at this delta
- 2:23:07It's not empty.
- 2:23:10in that case i'm going to print that chunk of data
- 2:23:12that i received but here we have a problem by default
- 2:23:16python's print function adds a new line after it
- 2:23:19prints something we don't want that new line we
- 2:23:22want the pieces of response that we receive to appear next to each other so
- 2:23:28Delta.
- 2:23:29so i'm going to add another argument called end and
- 2:23:33i set it to an empty string the default value
- 2:23:37is new line backslash n
- 2:23:41And for this reason, it goes to the next line by default.
- 2:23:44So setting it to an empty string prevents Python from moving to the next line.
- 2:23:49And now there is one more thing that is setting flush to true.
- 2:23:55flash to true and
- 2:23:56What does that mean?
- 2:23:57Because Python
- 2:23:58may sometimes buffer the output instead of displaying
- 2:24:02it immediately and we don't want that behavior.
- 2:24:05We want every piece of generated content to be displayed as soon as we receive them.
- 2:24:11Okay?
- 2:24:12But again, we have another problem.
- 2:24:14I also need to store the complete response because we need it for history.
- 2:24:19As you remember, to add history to our conversation, we need to
- 2:24:22store the messages and then with every new query, we
- 2:24:25need to send the entire history.
- 2:24:28But with the code that we have, we are only printing the response.
- 2:24:32We are not actually storing it. So how we can store the completed response manually.
- 2:24:39I mean, here.
- 2:24:40before our loop starts, I can add a new variable called response text.
- 2:24:49points.
- 2:24:50text and I set it to an empty string.
- 2:24:53Now every chunk that arrives I added to this variable. So here
- 2:24:59I say response
- 2:25:01text.
- 2:25:03plus equal delta
- 2:25:07So with this little trick, we solve this problem in this way.
- 2:25:11The user sees the response being generated in
- 2:25:14real time while we also build the complete
- 2:25:17response as well. And there is still one more edge case that we need to handle.
- 2:25:23Remember that we wanted to keep track of how many tokens were consumed.
- 2:25:28And when we use this streaming mode, you don't see the usage information.
- 2:25:33the user info the usage information is typically
- 2:25:36provided in the final chunk with the code we have
- 2:25:39in first iteration in first chunk we have nothing in second and third ones we have
- 2:25:44nothing and eventually when the stream is finished when the final
- 2:25:49chunk arrives the final trunk will contain the information the usage
- 2:25:54information that we need so we need another if statement here and i say if
- 2:26:01chunk.usage
- 2:26:03if in the chunk that has arrived there is this usage field then I'm going to add
- 2:26:12token I'm going to extract
- 2:26:16that information and add them to the attributes
- 2:26:20that we define for tracking token usage.
- 2:26:24Now after the loop, I'm going to use an empty print statement.
- 2:26:30It simply moves us to the next line. After that empty line, we will have our report.
- 2:26:35Okay, and it will be more visually appealing.
- 2:26:39And finally, we need to add the complete
- 2:26:41generated response to the conversation history.
- 2:26:45I don't need these lines anymore. I copied them. This one as well.
- 2:26:49And I comment it out. Okay, this is exactly
- 2:26:53what we want at this point we need to
- 2:26:57add the complete response okay the same name okay that's okay
- 2:27:02and finally we return the response it's not bad to return the
- 2:27:05response but we don't need it anymore because as you can see we are printing
- 2:27:10the response for the user as the chunks arrive
- 2:27:15and here we need actually remove this line we don t want to
- 2:27:20print it anymore
- 2:27:21because if we do that the user will see the response twice
- 2:27:25I only need to call this bot method
- 2:27:30Okay, finally let's run our program and test it
- 2:27:35Okay.
- 2:27:36What happened?
- 2:27:38there is a problem here. Oh, we forgot the comma.
- 2:27:46And here we have a typo.
- 2:27:49okay great I run the program one more time okay I say hello okay
- 2:27:55what's the problem here oh the
- 2:27:58problem is that instead of chunk
- 2:28:03We are using response
- 2:28:07and this is why I told you that pay attention when you copy paste your code
- 2:28:12okay one more time let's
- 2:28:15say hello
- 2:28:17Okay, what is the capital? Let's say, tell me a story.
- 2:28:25okay as you can see the response appears very
- 2:28:28quickly it is actually being being streamed piece
- 2:28:31by piece but it's happening so fast that you may
- 2:28:34not even notice and notice individual chunks and
- 2:28:37to make the streaming behavior more smooth and
- 2:28:41easier to see i'm going to do another little
- 2:28:44visual trick here i'm gonna import time module
- 2:28:49okay and um
- 2:28:52Here right after we receive a new piece of
- 2:28:55response, I'm going to introduce an artificial
- 2:28:58delay.
- 2:28:59So here I'm going to say time
- 2:29:03sleep for 0.05
- 2:29:07seconds this is obviously not something that
- 2:29:11you want to do in real application it's just
- 2:29:14for visual purpose to see that it is actually being
- 2:29:18streamed and let's run our program one more time
- 2:29:24Okay.
- 2:29:27I just say
- 2:29:29Tell me a story.
- 2:29:32okay as you can see this time we can clearly see
- 2:29:35the response being generated piece by piece and
- 2:29:39that's exactly what we were looking for we have
- 2:29:42successfully implemented the streaming for our
- 2:29:44LLM response and you saw that it wasn't difficult at all okay let's
- 2:29:51terminate it consume tokens and calculated costs
- 2:29:55okay now we have everything we wanted to
- 2:29:59Ladies and gentlemen, a message from your captain.
- 2:30:03We've noticed that some passengers are getting
- 2:30:05tired of our long exploration of planet Chatbot
- 2:30:08and can't wait to set off for the next world.
- 2:30:12That's completely understandable.
- 2:30:14Planet Chatbot is the longest stop on this entire journey.
- 2:30:18But please know this, what you learn here is essential
- 2:30:21for your survival on every planet that follows.
- 2:30:24So please be patient, the next world is closer than you think.
- 2:30:37From this session onward, I want to talk to you about a very important concept.
- 2:30:41It's called context window. This is very important.
- 2:30:44Every LLM has a context window and its size is a hard limit.
- 2:30:49It's a hard-limit imposed by the model. In other words, you cannot change it.
- 2:30:54You cannot change the model's context window yourself.
- 2:30:57So when you're using a model, you can check its context window
- 2:31:01and decide whether it's sufficient whether it's good for your application or not.
- 2:31:07Besides checking this limit, you also need to
- 2:31:10implement some strategies to manage this,
- 2:31:13to manage the context window.
- 2:31:15and there are different strategies or different ways to manage the context window.
- 2:31:20This is something that honestly you need to do, you need
- 2:31:23to take care of because that has a lot to do with the cost and other metrics that
- 2:31:28are important to you and your application.
- 2:31:30So first let's discuss what exactly is context window?
- 2:31:35Okay, let's say that we have a model and this
- 2:31:39model has a context window of 128,000 tokens.
- 2:31:44What does that mean?
- 2:31:45It means that in a single request, the model can only process up to approximately
- 2:31:51128, 000 tokens. And what does that include? What does this number include?
- 2:31:58It includes both the input and the output.
- 2:32:02both the input tokens and the output tokens generated by the model.
- 2:32:06So we can say it includes the user's last query, our conversation
- 2:32:11history,
- 2:32:13and the input that the model generates itself.
- 2:32:17So again, the tokens
- 2:32:19that occupy the context window come from three main components.
- 2:32:25New query, or history, and the model's output.
- 2:32:29And for sure, if you have any system messages, those are also taken into account.
- 2:32:34So as long as the total token count fits within this limit, I
- 2:32:39mean fits within the model's context window, we are fine.
- 2:32:43We don't have any problem.
- 2:32:44But if we exceed the available capacity, we run into a problem.
- 2:32:49So we need to be careful, we need implement some
- 2:32:52strategies in our application to handle this limit.
- 2:32:54Even if we choose a model that has a very high context
- 2:32:58window we still need to apply these strategies.
- 2:33:02And even when you open conversation in ChatChipity or cloud,
- 2:33:06don't assume that every message you have exchanged is
- 2:33:09necessarily sent to the model exactly the same form.
- 2:33:12Exactly the same for every time.
- 2:33:14No, these messages, I mean, your conversation history is summarized.
- 2:33:19Some of them may be deleted, maybe summarized in some way.
- 2:33:22And there are different strategies, I'm in.
- 2:33:24And different systems use different strategies.
- 2:33:26But in this tutorial, I am going to introduce two main strategies.
- 2:33:31and the second approach is more common in real world application.
- 2:33:35But first, we will see the first approach. And that approach is sliding window.
- 2:33:40and when you understand this strategy first, it gives us a very good foundation
- 2:33:45and we won't have any difficulty to understand the second approach.
- 2:33:50Okay, let's start with the first approach, sliding window. What does that mean?
- 2:33:56In this strategy we decide how many messages
- 2:33:59we want to keep in our conversation history.
- 2:34:01For example, we might decide to keep only the last five
- 2:34:06messages and ignore the rest, ignore the early messages.
- 2:34:10So as a new message arrives, all their messages are removed.
- 2:34:14One new message comes and one daily message goes.
- 2:34:18And with this strategy, we are somehow sliding a
- 2:34:21window and because of this, these strategies
- 2:34:24often called the sliding window. So the whole point of this strategy is that
- 2:34:29we no longer keep the entire conversation.
- 2:34:32Instead, we will only keep a selected portion of our conversation.
- 2:34:36For example, the last five messages. Okay. And why do we do that?
- 2:34:41Why the last 5 messages? Why the LAST messages? I mean, the assumption is that
- 2:34:47the recent messages are often more relevant, more important.
- 2:34:50And if we keep these messages, our LLM is more likely to have the
- 2:34:55information that is needed to keep track of the conversation.
- 2:34:59And of course, this approach has a limitation.
- 2:35:02We may lose some important information, but we cannot do anything about it.
- 2:35:07We should accept that we are losing some information.
- 2:35:10And if the user refers to something discussed
- 2:35:13a long time ago in messages that we deleted,
- 2:35:16the model doesn't have enough context to understand and answer that query anymore.
- 2:35:22So that's a limitation.
- 2:35:24And that's why the second approach can be
- 2:35:27better because instead of removing messages,
- 2:35:30it summarizes those messages. So we lose less information.
- 2:35:36But first, let's implement the sliding window approach.
- 2:35:42So first I'm going to add a new method for it.
- 2:35:46I'm gonna add a, a new, a method called, uh, let's set it
- 2:35:50up here. Okay. Here.
- 2:35:53I add a new method, I call it
- 2:35:56Trim
- 2:35:58messages.
- 2:36:00OK, self.
- 2:36:02pass okay first I don't implement it so I use
- 2:36:06the pass keyword so Python won't complain
- 2:36:09about it and then I'm going to add another parameter for it which is max number
- 2:36:16and I set a default value for it for example 4
- 2:36:21and this is the number of messages that I want to keep in conversation history okay
- 2:36:27now we're going to call this method we'll call this message inside our bot method
- 2:36:33after the LLM has generated its response and
- 2:36:36we have added the response to our conversation history so at the end of this method
- 2:36:43I'm going to called this method trim messages. Okay.
- 2:36:47This means after each response, we trim the conversation history.
- 2:36:52We are moving that sliding window.
- 2:36:54And with this approach, the window moves forward
- 2:36:58and this way, the conversation progresses.
- 2:37:02This way, we keep the most recent messages and we discard older ones.
- 2:37:07Now let's go back to our method we just defined and let's implement it.
- 2:37:14the first thing that we need to check is whether
- 2:37:18the number of messages is already less than max
- 2:37:21number or not if that's the case we don't need to do
- 2:37:25anything if we already have less than this number
- 2:37:28we don' t need to anything so here i put an if statement and i say if the length of
- 2:37:36or our history.
- 2:37:38is less than this max number that I have here so in that case I return
- 2:37:43and this prevents us from doing unnecessary work
- 2:37:46when the history is still short, is less then four messages
- 2:37:51and there is also another edge case we need to consider
- 2:37:54if you remember our conversation history does not
- 2:37:57only contain user messages and assistant messages
- 2:38:01it also contains system messages and as i
- 2:38:04explained earlier system messages are important
- 2:38:08these messages contain instructions and we need
- 2:38:12these instructions to guide the LLM's behavior
- 2:38:15so we don't want to accidentally or deliberately remove these instructions.
- 2:38:21So the first step is to separate our messages. We need two lists.
- 2:38:27The first list will contain only system messages. And the second list will
- 2:38:33contain all the other messages containing user messages and assistant messages.
- 2:38:38Here we can do this using list comprehensions.
- 2:38:42And this is a very concise syntax and this way
- 2:38:46can create a new list based on an older list. So let's do that.
- 2:38:51First, I'm going to use square brackets and this way I'm telling Python that, uh, I
- 2:38:57am, I am going to this syntax. Then I'm gonna iterate over my conversation history.
- 2:39:03So I say for every single message in my conversation history.
- 2:39:09and then I'm going to add a condition. I'm going to add a
- 2:39:13condition to check if that message has a role of system or not.
- 2:39:18So I say if that message.
- 2:39:23is equal to
- 2:39:24system.
- 2:39:26In that case, I want to have that message. Okay.
- 2:39:30Then I'm going to store the returning list in a variable called system messages.
- 2:39:38Now for the other list I'm going to do the opposite thing.
- 2:39:42I only duplicate this line.
- 2:39:45However, this time I only need to change equality to inequality.
- 2:39:51And this way we get all messages whose role is not system.
- 2:39:56Which means user messages and assistant messages.
- 2:40:00So I change its variable name to other messages.
- 2:40:07okay now we have two lists now how the next step
- 2:40:11is to keep only the last max number messages from
- 2:40:15other messages we need to take these messages
- 2:40:18out and we can do this using slicing syntax
- 2:40:21so here i say other messages
- 2:40:25I add brackets and then I say negative max number to the end.
- 2:40:32So here because we are using a negative index,
- 2:40:36this selects the last max number elements.
- 2:40:39In this case it is four. And we go until the end, so we go all the way to the
- 2:40:45end. So this gives us the last four non-system messages.
- 2:40:49And I store this list inside a variable called trimmed messages.
- 2:40:57Okay, so far so good.
- 2:40:59Now we have all we need.
- 2:41:01We have system messages and we have the messages that we want to keep.
- 2:41:07And here is a great place to use Python's unpacking syntax.
- 2:41:12We want to create a new list, a new conversation history, which consists, uh,
- 2:41:18system messages and trimmed messages so here I unpack all my system messages
- 2:41:26along with
- 2:41:28The messages I want to keep.
- 2:41:31Okay, and then we want to update our conversation history.
- 2:41:36So I say self.messages.
- 2:41:40okay great and that's it
- 2:41:42now the only thing left is to test it
- 2:41:46i run my program okay great
- 2:41:50i say i ask a series of questions
- 2:41:56Great.
- 2:41:57What is that?
- 2:41:59Capital of...
- 2:42:00France
- 2:42:03What is
- 2:42:04Uh...
- 2:42:07what is its population
- 2:42:09okay we had set a token limit and we exceeded that token
- 2:42:15limit so let's increase it where was that here
- 2:42:19Okay, 3000 is enough. Okay, or maybe 10,000. Okay, one more time.
- 2:42:26Okay, again, what is the
- 2:42:28capital of?
- 2:42:30France
- 2:42:32Okay, what is its
- 2:42:35population
- 2:42:37okay here we have a problem and the problem is that
- 2:42:44it belongs to here up here problem is that we forgot to unpack this list and
- 2:42:51we are creating a nested list and then we're
- 2:42:55passing it to messages so it shows that it was
- 2:42:59trying to to call this method it actually called
- 2:43:03it and we had this problem so i run it one more
- 2:43:06time okay say hello
- 2:43:09you can fast forward this part if you like
- 2:43:14Capital
- 2:43:16France,
- 2:43:17what is its population?
- 2:43:21Okay.
- 2:43:22What is its
- 2:43:25president okay now let's terminate the program and see our conversation history
- 2:43:31and as you can see our first message is a system message so we successfully
- 2:43:38separated the messages and we are not removing our system message which is
- 2:43:42important and as we can see our first
- 2:43:45message was hello and here you see that here we have what is its population so
- 2:43:51as you see early messages
- 2:43:53are gone and we are only going to have the last
- 2:43:56four messages okay so we have successfully
- 2:44:00implemented the first approach and in the next
- 2:44:04session we will move to to the next approach
- 2:44:13All right, in this session we are going to look at the second approach.
- 2:44:17The second approach to managing conversation history.
- 2:44:20In the previous approach we removed the older messages
- 2:44:23and kept only the most recent ones.
- 2:44:27As I mentioned the main drawback of this approach is that we lose information.
- 2:44:31Now with our new approach instead of simply deleting the older
- 2:44:35messages we are gonna send them to an LLM and this way we get a summary and we can
- 2:44:41replace those early messages with that summary. This way we lose less information.
- 2:44:47This is a common strategy for managing conversation history.
- 2:44:51And you see this pattern and you see
- 2:44:53this approach, this strategy in many AI applications and projects.
- 2:44:58So the question is, how are we going to implement this?
- 2:45:01There are different approaches, sub approaches we can say here.
- 2:45:05But one option is to summarize the conversation after every, every few messages.
- 2:45:10For example, we could summarize it in every five, every five messages.
- 2:45:15So the first step is to keep track of which iteration, to keep
- 2:45:20track of each iteration of conversation we are currently in.
- 2:45:23I'm going to add a new attribute to my class and that attribute can be
- 2:45:28called iteration, just iteration is good.
- 2:45:32and I set it to 0 as its initial
- 2:45:36value
- 2:45:38Now I'm going to go
- 2:45:41I'm going to go to my...
- 2:45:43conversation loop and here at the beginning
- 2:45:46at the begining of the loop I'm going to increment this attribute
- 2:45:50so I say self.iteration
- 2:45:54plus equal
- 2:45:57plus equal one. I increment it. I increment it by one.
- 2:46:02Now I'm going to define another method
- 2:46:07below this method.
- 2:46:09Summarize.
- 2:46:12messages.
- 2:46:16Okay, and I just put pass here because first I just
- 2:46:20want to define this method without implementation.
- 2:46:24And then here instead of calling the previous
- 2:46:27method, I'm going to call this new method.
- 2:46:31So I say self.
- 2:46:34summarize messages and i remove this one now let's
- 2:46:38implement it first i want summarization to happen
- 2:46:43only every five messages so we can do this
- 2:46:46using a modulus operator if you remember the
- 2:46:49modulus operator gives you the reminder after the division so here i can say
- 2:46:57if self.iteration
- 2:47:01and I say modulus as I told
- 2:47:04you
- 2:47:055 if it's not equal to 0
- 2:47:08I'm going to
- 2:47:10I'm going to return.
- 2:47:11So this means that if the current iteration isn't divisible by five, we return.
- 2:47:19We return immediately without doing anything.
- 2:47:22and summarization will happen on iterations like 5, 10, 15, 20 and so on.
- 2:47:30Next, we need to separate the system messages
- 2:47:33from the other messages, just as we did in
- 2:47:36the previous session. So we can just copy the logic that we had for that.
- 2:47:47Okay, let's fix
- 2:47:48indentation. No, it's okay.
- 2:47:51Now at this point, I want to split other messages into two parts.
- 2:47:55The first part contains the older messages.
- 2:47:59We want to summarize these older messages and the second part contains
- 2:48:03the more recent messages. And we want to keep these messages.
- 2:48:08We want preserve them exactly as they are. How many messages we want summarize.
- 2:48:14It depends on ourselves. It depends our project. For this example,
- 2:48:20I'm going to summarize the first five messages and I will keep the rest as they are.
- 2:48:26So here to create these sub lists, I'm
- 2:48:30going to use bracket notation. So I save other messages.
- 2:48:34I want to summarize the first five
- 2:48:39messages and I also slice
- 2:48:42the rest so from 0 to 5 from index 5 to the end
- 2:48:47here we have two lists and we want to store
- 2:48:50these two inside two variables so a better
- 2:48:54syntax here is to convert these two into a
- 2:48:57tuple now we can unpack this tuple on the left side so here i say
- 2:49:06Early
- 2:49:08messages
- 2:49:11and
- 2:49:12last
- 2:49:14messages.
- 2:49:15Okay, now
- 2:49:16on the right hand side, we have a top hole.
- 2:49:19You're unpacking it into two variables, early messages and last messages.
- 2:49:25I want to leave these last messages unchanged
- 2:49:30and I want summarize these. So we need an LLM call.
- 2:49:35So here I
- 2:49:37just,
- 2:49:40Copy.
- 2:49:41this piece
- 2:49:42of code
- 2:49:44and i paste it here we need to change some
- 2:49:47arguments for example we no longer need the
- 2:49:50streaming key here and here instead of passing the
- 2:49:54entire conversation history i just want to pass
- 2:49:58early messages but we need we need guide we need the llm we need
- 2:50:02to specify its task so here i also add a system message so here instead
- 2:50:07of just this list i create a new list and I put a message as its first element
- 2:50:14and then I unpack this list here
- 2:50:19So here first we have a system message, then we have our messages.
- 2:50:25So here I specify the role.
- 2:50:29I say it's a system message and then
- 2:50:32we have our content
- 2:50:35So I say, summarize these messages concisely.
- 2:50:43keeping
- 2:50:44key facts
- 2:50:46and
- 2:50:48decisions.
- 2:50:51Okay.
- 2:50:52okay as you can see for this example i'm using the same model the same
- 2:50:57model that we used as our main model for conversation but you could also use another
- 2:51:04another model a smaller model a more lightweight
- 2:51:08model or a model that is specifically trained
- 2:51:11trained for summarization it totally depends on you
- 2:51:15and it's not a difficult thing just change it
- 2:51:18and use another model
- 2:51:20and then here we receive a response we need to convert it into a message
- 2:51:26I prefer to use a system message some other people have a different preference
- 2:51:31they save it as an assistant message but I prefer save it
- 2:51:35as a system so here I create a message I specify the role
- 2:51:41I want it to be a system message and then I specify its content
- 2:51:46I say Let me use formatted string and I use triple.
- 2:51:53triple quotes.
- 2:51:55Here I say this is a summary of the five early messages.
- 2:52:03and then down here I put a placeholder
- 2:52:06and then I extract the content of my response I say response.
- 2:52:13choices, choices zero,
- 2:52:15message and content.
- 2:52:18the same as before
- 2:52:19we did it a lot and I store it in a variable called
- 2:52:22summarized message
- 2:52:27Okay, great. Now
- 2:52:28we have converted the LLM's response into a message, into a
- 2:52:32message that can put inside our conversation history.
- 2:52:37and the final step is that we need to construct our new conversation history
- 2:52:43first we need put all system messages at the beginning
- 2:52:48then we want the summarized message and then we
- 2:52:52want the last messages the last
- 2:52:55messages that we that we left here to use them
- 2:53:00so here i create a new array
- 2:53:04with the same syntax like before. I mean unpacking. I have a new array.
- 2:53:09First I put system messages.
- 2:53:14system messages, then I have this single summarized message and then last messages.
- 2:53:23the messages that I didn't want to summarize. Okay,
- 2:53:27that's okay now.
- 2:53:29I need to update the conversation history. So I say self that
- 2:53:34messages
- 2:53:37and I set it to this new array. And that's it.
- 2:53:40We have replaced the older messages with a summary.
- 2:53:43We are also preserving the more recent messages. That's a very good strategy.
- 2:53:49And at the end, I put a print statement.
- 2:53:53This way we can realize that the summarization is taking place.
- 2:53:59I say summarization was
- 2:54:02successful.
- 2:54:03Okay, now the final step is to run our program and test it to see
- 2:54:09whether it works. So here I say hello, create
- 2:54:15What is the capital of France?
- 2:54:21What is...
- 2:54:23its population
- 2:54:27Okay.
- 2:54:28What is its
- 2:54:32Uh,
- 2:54:34with its
- 2:54:36President
- 2:54:39Okay, still waiting for the summarization. Okay. What is its...
- 2:54:45famous dish
- 2:54:49now here we need to have a summary yes summarization was successful and let's
- 2:54:54quit and see the conversation history so here i can we can see that we have our
- 2:55:00system message that we specified ourselves and now here uh we have a system
- 2:55:05message that is the summary that the LLM generated for us and yes we have
- 2:55:11key information and then we have the last messages after that summary so
- 2:55:18As you can see in this strategy, instead of
- 2:55:20simply deleting the earlier messages, we are
- 2:55:23using an LLM to summarize them.
- 2:55:26And this way we lose less information in comparison with the previous approach.
- 2:55:31Welcome back aboard Neon Learn. Planet Chatbot is conquered.
- 2:55:35One world down, nine to go.
- 2:55:39Fasten your seat belt. Next stop, plan a classifier.
- 2:55:42Here you'll teach AI to classify anything
- 2:55:45using lightweight models from Hugging Face.
- 2:55:48No LLM required.
- 2:55:50Your next world awaits. Enjoy the journey.
- 2:56:00Alright, in the next project we are going to talk about sentiment analysis.
- 2:56:04This is one of the most commonly used AI tasks.
- 2:56:08And it means we give our model a piece of text
- 2:56:12and it tells us whether it's negative, positive, or neutral.
- 2:56:18This is something that you will see in many
- 2:56:20different applications, in many AI tasks.
- 2:56:22For example, we can use an AI model to analyze customer comments.
- 2:56:28and this way it can determine whether each comment is positive, negative or neutral.
- 2:56:35Maybe we need this information for ourselves, maybe we
- 2:56:38want to help other customers to make a better decision.
- 2:56:42The application is honestly endless.
- 2:56:45But the important question for us is that how can we implement this task?
- 2:56:50Well, one way is to use an LLM. I mean,
- 2:56:55the same LLM that we used in the previous project, in the chatbot project.
- 2:57:00And another approach is to use models that are specifically designed, that are
- 2:57:05specifically trained for sentiment analysis.
- 2:57:08You might ask, what's the advantage of the second one?
- 2:57:12Because obviously the first one is simpler when we use an LL.
- 2:57:15But what is the advantage the second to use a specialized model?
- 2:57:20The answer is that these models are much smaller,
- 2:57:23they are lighter and as a result, cheaper.
- 2:57:26They are so lightweight that maybe you can run them on your own personal computer
- 2:57:32because they are lightweight parameters are not too many.
- 2:57:35So that's the advantage and that's exactly what
- 2:57:38we are going to do in this second project.
- 2:57:40So first we use an LLM to implement this task
- 2:57:44and then we move on to the second approach.
- 2:57:47which we will use a specialized model and you will see how we do that.
- 2:57:52So let's focus on the first approach and let's implement it.
- 2:57:55Let's go to into code
- 2:57:57and start creating our project.
- 2:58:00So first I create a new file called
- 2:58:03sentiment.
- 2:58:05Analysis
- 2:58:07high and
- 2:58:08here I will follow the same steps that we used in the previous project.
- 2:58:13First, I'm going to import.env because I want
- 2:58:16to access to my environmental variables,
- 2:58:20because I'm, I am going to have an LLM call and then I import OS module to access
- 2:58:27to my environment.
- 2:58:29variables and then I'll import OpenAI because we want to create a client and
- 2:58:34communicate with the LLM so I say from
- 2:58:38OpenAI
- 2:58:40import
- 2:58:41OpenAI class
- 2:58:44So, and finally, I'm going to load the environment variables. So I say.env.
- 2:58:50load.env and then I'll create my main function.
- 2:58:58and finally i'll add our starting point i say if
- 2:59:02double underscore name double underscore is equal to
- 2:59:06double underscore main double underscore in that
- 2:59:10case i want the main function to be executed and
- 2:59:14from the beginning i want to use object-oriented
- 2:59:18structure so here i add a new class called analyzer
- 2:59:23along with its constructor so init and that's it and now
- 2:59:28i go to our chatbot project and i'm going to borrow
- 2:59:34these two.
- 2:59:36paste them here okay fix indentation okay this is
- 2:59:41my api key and that's the same and here instead of
- 2:59:46using this system message i'm going to use another
- 2:59:49system message because it's not a chatbot anymore
- 2:59:52i'm gonna give it an instruction specifically
- 2:59:56for sentiment analysis so here i say classify
- 3:00:00the sentiment of the text as positive negative or neutral okay
- 3:00:08And now we need a method that receives a piece of text as its
- 3:00:13input and sends it to the LLM and returns the model's response.
- 3:00:17So here I say def.
- 3:00:20sentiment analysis
- 3:00:24self parameter should be always self in a class in Python
- 3:00:27and the second parameter is the text that we
- 3:00:30want to figure out what is its sentiment
- 3:00:33now again I go to chatbot and again borrow some code
- 3:00:40okay I copy this and I paste
- 3:00:43it here
- 3:00:45and of course we need to make a few changes first I'm going to add temperature
- 3:00:51and I set the temperature to 0.1 a very low temperature
- 3:00:57and the reason is that for sentiment analysis we generally don't need creativity
- 3:01:02we want the model to behave as deterministically as possible so that's the reason
- 3:01:08then here for the messages we no longer need these two
- 3:01:13we need to unpack our messages here our conversation history that we have
- 3:01:18as an attribute so first i unpack it so i say self.messages
- 3:01:24and then
- 3:01:26we are going to put the text that this function,
- 3:01:29this method more accurately receives.
- 3:01:32But instead of just putting a piece of text here, we
- 3:01:36need to wrap it with that role content structure.
- 3:01:39So here I put curly braces and I say role is user and then I say
- 3:01:47content is
- 3:01:50what I have received
- 3:01:52here okay that's it and finally I get the response and return
- 3:01:56so I say return
- 3:01:58response.choices
- 3:02:00exactly like before choices 0
- 3:02:06Message, Content.
- 3:02:09Now we need to go to our main function and use it.
- 3:02:13First, I need to create an object.
- 3:02:15So I say analyzer and then
- 3:02:17I store this object inside the variable.
- 3:02:22Say analyzer.
- 3:02:24and then I can call the sentiment analysis method
- 3:02:29on this object so I say analyzer.sentiment
- 3:02:33analysis and the piece of text that I want to give it is for example the movie
- 3:02:39is perfect.
- 3:02:42Okay, now let's run our program and see what happens. Run the program.
- 3:02:48Oh, we forgot to print it actually.
- 3:02:50So let's print it or maybe it's better to store the
- 3:02:55result in a variable called sentiment and then print it.
- 3:03:02Okay, one more time.
- 3:03:05And as you can see, it has correctly labeled the
- 3:03:08sentiment of this piece of text as positive.
- 3:03:12Ladies and gentlemen, the captain has switched on the feedback sign.
- 3:03:17If this flight has been worth it, please rate it with a like.
- 3:03:21Want a window seat for every new launch? Join our frequent flyer program.
- 3:03:25Subscribe and turn on notifications.
- 3:03:29And tell us in the comments where should our next
- 3:03:31flight go, the captain is already planning the route.
- 3:03:35Thank you for choosing Neon Learn.
- 3:03:45Okay, now in this session, I want to talk about
- 3:03:48an important concept called prompt engineering.
- 3:03:51This probably isn't a new concept for you.
- 3:03:53We have actually been doing it throughout the course a lot. So what
- 3:03:57does that mean?
- 3:03:58What does prompt engineering mean?
- 3:04:00In simple terms, it means controlling the behavior
- 3:04:04of the model through the input we provide.
- 3:04:07Okay, for example, in the previous project, we
- 3:04:10use the same model to summarize messages and
- 3:04:13In this project you saw that we used that model
- 3:04:16again, but this time for sentiment analysis.
- 3:04:19The model is the same, but the behavior is completely different.
- 3:04:22So where does this different behavior come from?
- 3:04:25It comes from the prompt that we provide.
- 3:04:29So, as a category of prompt engineering, we can refer to few-shot
- 3:04:33prompting, and that's what I'm going to talk about now.
- 3:04:37it means that before the model generates its answer
- 3:04:41we provide it with a few examples of how we expect it to behave
- 3:04:45let me give you an example
- 3:04:47for example we can tell the model that the movie was terrible
- 3:04:51and then we show it that it should classify it as negative
- 3:04:56then again we say that nothing special and we show its label neutral
- 3:05:01terrific positive the movie is terrible
- 3:05:04negative and by giving the model these examples
- 3:05:08we are showing it what the task looks like
- 3:05:11and how we expect it to respond and one of the
- 3:05:14best patterns for doing this in these scenarios
- 3:05:18is using user assistant pattern in other words
- 3:05:21we can put several examples in the conversation
- 3:05:24history with the roles of user and assistant in other
- 3:05:28words again even though these are not actual messages
- 3:05:33we put them and this way we want to kind of control the behavior of the model
- 3:05:38so let me show you here in our conversation history
- 3:05:43after putting the system prompt we add some user resistant messages
- 3:05:50So here I say role is user.
- 3:05:53And here we specify I absolutely loved this restaurant.
- 3:06:00OK, and then we have a message with the assistant role.
- 3:06:07and we say here positive okay another example for for the negative case okay
- 3:06:15Here I copy two of them.
- 3:06:17okay again user has said the wait time was too long was way too long
- 3:06:26So it's definitely negative.
- 3:06:30One more time.
- 3:06:32We can say
- 3:06:35it was an average experience
- 3:06:39nothing special and here we can say neutral okay these are
- 3:06:45not real messages as you can see we didn't
- 3:06:49we didn send this message to the llm and get the
- 3:06:52corresponding response they are made up these are
- 3:06:55for controlling models behavior to show it how it
- 3:06:59should behave what is the structure of output what is
- 3:07:03what are for example possible answers and instead of providing a long explanation we
- 3:07:09just provide some examples now let's run our program
- 3:07:14Okay, as you can see, our program is working like
- 3:07:16before, but let me show you something interesting.
- 3:07:19Imagine we expected a slightly different output format.
- 3:07:22For example, I expected that the model put a
- 3:07:26sentiment plus colon before each response.
- 3:07:30So here I add sentiment
- 3:07:32colon and I repeat that again.
- 3:07:36Okay,
- 3:07:38here.
- 3:07:39and again here now
- 3:07:40let's run our program one more time
- 3:07:43as you can see this time the model has learned that
- 3:07:46we want this part here before the actual answer
- 3:07:50so you can say this is quite useful this is very useful in these scenarios
- 3:07:55in scenarios that explanation is less useful than
- 3:07:59providing some examples it is useful both for us and for the model itself
- 3:08:04the model can better understand
- 3:08:07what you want.
- 3:08:08And one important question you might ask is how many examples are enough?
- 3:08:13Honestly, in many cases, around three to five examples are fine.
- 3:08:18And pay attention when you
- 3:08:20choose your examples, try to cover different scenarios.
- 3:08:24For example, in our example, you
- 3:08:26saw that we include a clear example of positive
- 3:08:30case, negative case and neutral case.
- 3:08:33So you can also add
- 3:08:35couple of more challenging or ambiguous examples.
- 3:08:38For example, this one was a little bit vague because it could be also negative.
- 3:08:44But we showed the model that in these scenarios it should give this answer neutral.
- 3:08:49But someone may say it is negative when there is nothing special
- 3:08:53when an average experience is positive.
- 3:08:57Sorry, it is positive, but it totally depends on us and few-shot prompting is great.
- 3:09:04to specify our tendency, our attitude.
- 3:09:07For example, for this kind of sentiment analysis
- 3:09:10we want it to be neutral instead of negative.
- 3:09:14But keep in mind that generally adding more
- 3:09:17and more examples is not necessarily better.
- 3:09:20In many cases, going beyond 10 examples doesn't provide much additional benefit.
- 3:09:26It only burns your tokens.
- 3:09:28So choose your examples carefully and the goal isn't
- 3:09:32to give the model as many examples as possible.
- 3:09:35The goal is to give a small number of representative
- 3:09:39examples that clearly demonstrate the behavior
- 3:09:42We expect. We want.
- 3:09:45Ladies and gentlemen, congratulations. You've just learned to run sentiment analysis
- 3:09:50with a large language model.
- 3:09:53But before we move on, your captain has a bonus for you.
- 3:09:57You'll learn to solve the very same task with natural language inference.
- 3:10:01Instead of a giant LLM, a small model simply
- 3:10:04checks whether a statement follows from the text.
- 3:10:08Is this review happy or unhappy?
- 3:10:11It's dozens of times lighter, runs on an
- 3:10:14ordinary server, and needs no API calls at
- 3:10:17all.
- 3:10:18Consider it a free upgrade, your captain will now show you how.
- 3:10:29Alright, in the previous session we worked on sentiment analysis
- 3:10:32and we used an LLM to perform this task Now in this session we are going to use
- 3:10:39a model that's much more specifically suited to this kind of task.
- 3:10:43If you want to be precise, sentiment analysis is a well-defined task
- 3:10:47and there are many many models specifically trained for this task.
- 3:10:52For example, these models might be trained on data sets containing
- 3:10:56movie reviews, product reviews, or other types of comments or other information.
- 3:11:01And during training, each example is associated with a label,
- 3:11:06such as positive, negative or neutral.
- 3:11:08And after training, the model learns to classify
- 3:11:11new pieces of text based on those labels.
- 3:11:15So let's take a quick look at HuggingFace and
- 3:11:19search for sentiment analysis or just sentiment.
- 3:11:23Okay.
- 3:11:24As you can see here, there are many models with the word sentiment in their names.
- 3:11:29Previously, Huggingface had a dedicated filter task for that, but
- 3:11:35at some point it was removed under a broader category
- 3:11:39broader category called text classification
- 3:11:42and changes like this happen a lot happen frequently in this community
- 3:11:47and this has two reasons first this field is relatively young changes happen a
- 3:11:52lot and second it's evolving incredibly quickly
- 3:11:56so if you notice a task category has changed has removed has uh whatever
- 3:12:02Don't be surprised and even libraries you use, for example
- 3:12:06You can see that Langchain sees a lot of changes
- 3:12:09I know it can be a little frustrating especially when
- 3:12:13you are following a course, following a tutorial
- 3:12:16and you see that interface is not like what you see in the videos
- 3:12:21but you should be adaptive as developers you need to get used to it
- 3:12:26now the question is that what are we going to do in this session
- 3:12:32honestly i don't want to look at this problem only as a sentiment analysis task
- 3:12:37i want you to learn something more powerful
- 3:12:41something that you can use not only for sentiment
- 3:12:44analysis but also for many other use cases.
- 3:12:47And that brings us to NLI or Natural Language Inference.
- 3:12:51The goal of NLI is to understand the logical
- 3:12:55relationship between two pieces of text.
- 3:12:58These two pieces have specific names.
- 3:13:02The first one is called premise
- 3:13:05and the second one is call hypothesis.
- 3:13:08The premise is the information we start with
- 3:13:12and the hypothesis is a piece of information we
- 3:13:16want to evaluate in relation to the premise.
- 3:13:20And there are three outcomes. The first is entailment.
- 3:13:25The second is contradiction. And the third is neutral.
- 3:13:30Let me give you an example this way you will learn better.
- 3:13:32Suppose our premise is a man. is playing guitar on stage.
- 3:13:39Now let's look at three different hypotheses that each one has a different outcome.
- 3:13:45The first example is a person is performing music. What do you think?
- 3:13:51What do think the result should be? The premise tells us that a man is playing
- 3:13:55guitar on a stage. The hypothesis says the person is
- 3:13:59performing music, so these two statements are consistent and the
- 3:14:04premise supports the hypothesis. So the result is definitely entailment.
- 3:14:09Now, Example 2. A man is sleeping.
- 3:14:12The premise tells us that a man is playing
- 3:14:15guitar while the hypothesis says he's sleeping.
- 3:14:18So in this example, the two statements contradict each other.
- 3:14:23A man cannot be sleeping while playing music. So that doesn't make sense.
- 3:14:27So the result is contradiction.
- 3:14:30And the third example, a man wearing a red T-shirt.
- 3:14:35Does the premise confirm this statement? No. Does it contradict it? Not necessarily.
- 3:14:40So, the result is simply neutral.
- 3:14:44And that's the basic idea behind natural language inference.
- 3:14:48Now,
- 3:14:49the question is, why am I saying that NLI is very powerful?
- 3:14:55because it can be used for many many different applications and use cases
- 3:15:01for example NLI can be use for zero shot classification
- 3:15:06fact checking, question answering, duplicate detection and many other scenarios
- 3:15:12and the use case we are going to focus on in the following sessions
- 3:15:16is zero shot classification so let's understand what it means well traditionally
- 3:15:23If we wanted to build a model for a particular classification task,
- 3:15:28we would usually need to train a model on a labeled dataset.
- 3:15:32Let's give you an example.
- 3:15:34Let's say we want to automatically tag articles on our website.
- 3:15:38We have different articles. We want to tag them.
- 3:15:41And suppose these articles are about programming and
- 3:15:45our labels are JavaScript, Python, Ruby and CSS.
- 3:15:49I mean programming languages and technologies.
- 3:15:53Traditionally, if we wanted such a model, we would collect
- 3:15:57a data set containing perhaps a thousand articles.
- 3:16:01Each article would have a label, a label indicating its topic.
- 3:16:05For example, one article might be labeled JavaScript, another Python, another Ruby.
- 3:16:11And we would then use this label data set to train a model.
- 3:16:16and after training a model at the end we would have a model
- 3:16:20that had learned to classify articles into those categories
- 3:16:25so
- 3:16:25we could give it a new article without a label
- 3:16:29and it would predict the category that article belonged to
- 3:16:32but now let's look at the approach that I want to talk about
- 3:16:36zero shot classification
- 3:16:37with a suitable pre-trained NLI model we don't
- 3:16:40need to have that training process anymore
- 3:16:44and we can use a good NLI model to classify different kinds of label data
- 3:16:51so we can take a general purpose model that has already grasped
- 3:16:55that has understood relationships between text statements
- 3:17:00and give it a set of candidate labels
- 3:17:02so the model can evaluate the input text against those labels
- 3:17:07and determine which ones are the best match
- 3:17:10For example, today I have a set of candidate
- 3:17:13labels like JavaScript, Python, Ruby, and CSS.
- 3:17:16But tomorrow, I might need to classify completely different categories.
- 3:17:20For example health, sports, economics, technology, and so on.
- 3:17:26Now, without changing the model, I only change the labels, the candidate labels.
- 3:17:32And that's really powerful that we don't need a new model.
- 3:17:36We can use the same model.
- 3:17:38So that makes zero shot classification and natural language
- 3:17:42inference extremely useful in a wide range of applications.
- 3:17:47and that's exactly why I wanted to introduce NLI
- 3:17:50instead of focusing only on sentiment analysis
- 3:17:54another advantage is that it's much smaller and I mean
- 3:17:58these kinds of models are smaller in comparison with
- 3:18:02something like the generation models that we used
- 3:18:07for our chatbot we use an LLM these LLMs have
- 3:18:11billions of parameters but these kinds of models
- 3:18:14that we are going to use are much smaller
- 3:18:16for example the model we are
- 3:18:18going to is under 1 GB we can even run it
- 3:18:22locally on our personal computer and we will
- 3:18:26do it in the following sessions and the question is which model are we going to
- 3:18:31use exactly the model I'm going to for this project is distilbert-base-uncased-mnli
- 3:18:40Let's search it more accurately, type form.
- 3:18:44Okay, so that's the model we are going to use.
- 3:18:47And as you can see, it has 67 million parameters, which is
- 3:18:53really lightweight.
- 3:18:58Okay, the model we are going to use belongs to the BERT family. So what is BERT?
- 3:19:04BERT stands for Bidirectional Encoder Representations from Transformers.
- 3:19:09That's quite a long name, I know.
- 3:19:11But BERT was introduced by Google in 2018 and it became an influential model in NLI.
- 3:19:19And many subsequent models and approaches
- 3:19:22were built on top of it, on ideas from BERT.
- 3:19:25So here the important thing to understand is
- 3:19:29that BERT is an encoder-based transformer
- 3:19:32and it learns conceptual representations of language
- 3:19:36and one of its key characteristics is that it processes text bidirectionally
- 3:19:41and because of that it has this term in its name
- 3:19:45Now let's look at what does that mean in total Consider these two sentences
- 3:19:50I see the blue sky and I feel blue In the first sentence, blue refers to color.
- 3:19:57In the second sentence, blue describes a feeling of sadness.
- 3:20:01So, the word is the same but the meaning is different.
- 3:20:04The meaning changes depending on the context.
- 3:20:07And the word context you hear a lot refers to that.
- 3:20:10If you see, if you hear that a model learns the context of the text,
- 3:20:15the term context means that. So, to summarize, context means.
- 3:20:21means the surrounding information that helps us to understand
- 3:20:26what a word or sentence actually means.
- 3:20:28So we should consider other words when we want to grasp the meaning of a word.
- 3:20:33And that's really important.
- 3:20:35Earlier approaches often struggle to capture this information.
- 3:20:40They couldn't do that effectively.
- 3:20:42But the transformer architecture revolutionized
- 3:20:45everything and introduced the powerful
- 3:20:47technique a powerful mechanism called attention
- 3:20:51and it was described in the influential paper
- 3:20:54attention is all you need and as i told you attention helps a model to understand
- 3:21:00the context better and it determines which parts
- 3:21:04of the input are relevant to one another and the
- 3:21:08BERT model builds on the transformer architecture
- 3:21:12and uses information from both sides from both
- 3:21:15directions and this can help the model understand the context much better.
- 3:21:21So that's the meaning of BERT and what does
- 3:21:25Distilbert mean? As you can see it's not only BERT, it's
- 3:21:30DistilBERT. We should know that what distil mean actually in
- 3:21:35in the world of AI. It refers to a technique. We have a technique called
- 3:21:39Knowledge Distillation.
- 3:21:41Knowledge Distillation is a technique used to
- 3:21:44train a smaller model to reproduce the behavior of a larger model.
- 3:21:49To replicate capacities of a large model.
- 3:21:52For example, imagine we have a model with billions
- 3:21:55of parameters and we want to create a smaller one.
- 3:21:58And this way we can train a small model using the
- 3:22:01information that the larger model produces.
- 3:22:05And it enables the smaller model, to learn to approximate some of its capacities and
- 3:22:11This way we can reduce the model size and this way we need
- 3:22:15less computational power, fewer computational requirements
- 3:22:20and we can kind of retain as much useful performance as possible
- 3:22:25and this DistilBERT here is a small version derived from the BERT model we talked about
- 3:22:30and it uses this approach, knowledge distillation
- 3:22:34and this DistilBERT was introduced by Hugging Face
- 3:22:37and the goal was obvious to be more lightweight and it's lighter than the BERT model
- 3:22:44so it was about this part, this DistilBERT
- 3:22:46and another important thing, another important concept I want you to understand
- 3:22:51is about three different categories that we have
- 3:22:54which are encoder, decoder and cross-encoder the
- 3:22:58BERT model is an encoder
- 3:23:01encoder model, encoder-based model. But what exactly does an encoder mean?
- 3:23:06In simple terms, an encoders processes input
- 3:23:10and it tries to take out representations. Okay?
- 3:23:14And a decoder, on the other hand, is used to generate output.
- 3:23:19For example, a GPTS style model that you see are decoder only
- 3:23:23and are designed to generate information, to generate text data in general.
- 3:23:28And the key point here is that BERT is not designed to generate text.
- 3:23:33It is designed to understand text. And because of that, it's an encoder model.
- 3:23:39And for our NLI task, that's exactly what we need.
- 3:23:42We don't need a model that can generate text, we needed to
- 3:23:46understand the relationship between two pieces of text.
- 3:23:49And help us determine whether one is entailment, contradiction, neutral, and so on.
- 3:23:55So here we can use an encoder based model
- 3:23:58rather than a generative model, a generative
- 3:24:00element. Of course, different AI tasks require different architectures.
- 3:24:04For example, if we had machine translation,
- 3:24:07in that case, we needed both because we need
- 3:24:10encoder to process the source language and take out the context.
- 3:24:15And then we use decoder to use that information to generate in the target language.
- 3:24:21That was an example where we need both components.
- 3:24:25And there's still one category called cross encoder.
- 3:24:29A cross encoders processes two pieces of text together.
- 3:24:32That's important. It does it together.
- 3:24:34And it allows the model to compare the two inputs directly
- 3:24:38and learn how they are related to one another. This is very useful in many tasks.
- 3:24:45For example,
- 3:24:46In RackSystems, we want to know whether a
- 3:24:49document is relevant to search query or not.
- 3:24:52Or many other situations that's really important that we need to know
- 3:24:56whether a piece of text answers a question or no.
- 3:25:01And in these cases, what matters to us is the
- 3:25:03relationship between two pieces of text.
- 3:25:06And in this situations, cross-encoder is one option that helps us.
- 3:25:16Now that we understand the theory behind this task,
- 3:25:20let's see how we can use this model in our code.
- 3:25:23If you go on the model's page on Hugging Face and click on Use this model and click
- 3:25:29on Transformers, you can see here we have two options, Pipeline and Auto Classes.
- 3:25:34Auto Class give you more control,
- 3:25:36more control over how the model is loaded and used.
- 3:25:40However, they also require more code and involve a little more complexity.
- 3:25:46The pipeline option on the other hand is much simpler.
- 3:25:49As you can see, we can specify the task and the model
- 3:25:53and the pipeline takes care of most of the implementation for us
- 3:25:57which is really great.
- 3:25:58But one important point is that the pipeline
- 3:26:01itself also uses the underline model classes.
- 3:26:05It simply provides a higher level interface that makes things easier for us.
- 3:26:10So in this project, which approach are we going to use? we are going to pipeline
- 3:26:15because it's simpler and that's enough that's for our project and it's
- 3:26:19simpler for you now what we are
- 3:26:22going to do now is to replace the LLM we used in our previous session
- 3:26:27with our pre-trained HuggingFace model what you saw here
- 3:26:30and to use models from Huggingface we need to install transformers library
- 3:26:35which we have already installed in an earlier session We also need PyTorch.
- 3:26:39We also needed PyTorch, which is one of the
- 3:26:42libraries that Transformers uses to run model.
- 3:26:44So we need it. And before the session, I installed it.
- 3:26:49So you can install it using this command, pipenv install torch.
- 3:26:54By running this command you can't install PyTorch. Okay.
- 3:26:58now let's go back to our code and first i'm going
- 3:27:02to import pipeline i imported it before starting
- 3:27:05recording the session i imported pipeline from
- 3:27:08transformers and now let's find the part of our
- 3:27:11code which was for the llm so now we are going
- 3:27:15to replace it with our classifier so here
- 3:27:17instead of having this lln i'm gonna have a classifier
- 3:27:22and I set it to pipeline exactly what you see here we need a pipeline
- 3:27:28the first argument we need to provide is the type of task that we want to perform
- 3:27:34in our case the task is zero shot classification so here you see it is suggesting us
- 3:27:39I'm going to have this task zero shot classification
- 3:27:43next we need specify which model we want to use so I go here and I copy it
- 3:27:49to avoid any typo
- 3:27:51So I paste it here and this way you specify the model that you're going to use.
- 3:27:57And that's pretty much it.
- 3:27:59The pipeline handles the main steps required to load
- 3:28:02the model and perform zero shot classification.
- 3:28:05We don't need to implement those steps ourselves, which is really good.
- 3:28:08As a beginner, these types of things are
- 3:28:11really scary, but it makes it easier for us.
- 3:28:14So that's really great.
- 3:28:15now let's move let's move to our sentiment analysis method here we need to we need
- 3:28:22to change this LLM call that we have here i'm going to remove the code that sends
- 3:28:27the request to an LL and instead we are going to use the model that we chose
- 3:28:32first we need
- 3:28:33to define our candidate labels which labels do we want to use positive negative
- 3:28:38and neutral these are all labels so here i define a new variable called
- 3:28:44labels it's a list of three labels positive
- 3:28:48negative and neutral okay next we'll use the
- 3:28:52classifier we initialized in our constructor and
- 3:28:56as a first argument we pass the text that was
- 3:28:58provided to our model and as the second argument we
- 3:29:02passed the candidate labels we just defined here
- 3:29:05let's do it here i say self.classifier okay and as first argument i say
- 3:29:13text, the text that I'm receiving in this method.
- 3:29:16And as a second argument, I specify the candidate label.
- 3:29:21So I say candidate
- 3:29:23labels, and I set it to these labels.
- 3:29:27And I store the result in a variable called result.
- 3:29:35And finally, I will return the result.
- 3:29:40and that's all we need to change notice that we have kept the same method and the
- 3:29:45same overall structures we have simply changed the implementation behind it
- 3:29:49that's the only thing we do previously we used an LLM and we had an LLL call
- 3:29:54now we are using a pre-trained model and we do it locally we run a model
- 3:29:59locally and we get it from HuggingFace and run it and use it now let's run our
- 3:30:03program and see what happens okay here I run the program
- 3:30:08One thing to keep in mind is that the first
- 3:30:10time you run the program, the model needs to
- 3:30:12be downloaded. So the first run will take longer than usual.
- 3:30:16And once the model has been downloaded and cached,
- 3:30:19subsequent runs generally won't need to download
- 3:30:22it again. Okay. You can see the result here.
- 3:30:25It says that with this probability 0.98% this
- 3:30:29text you see here, the movie is perfect,
- 3:30:33positive.
- 3:30:33And there is one important detail here.
- 3:30:36When you run the program, the model's weights
- 3:30:39are loaded into your computer's memory.
- 3:30:42So this way the model can perform the task and can perform inference.
- 3:30:47And when the program finishes, the memory can be released.
- 3:30:51So that adds to
- 3:30:52the time you're waiting here but when you run this program on a server
- 3:30:57it's up 24 7 and you don't have to wait the same time that you wait here
- 3:31:02now let me show you something interesting i'm going to change the labels
- 3:31:07but i won't change the model or the implementation of our method
- 3:31:12just instead of specifying these labels i say finance
- 3:31:18health,
- 3:31:20and sort.
- 3:31:23And here, instead of saying the movie is perfect, I say
- 3:31:28Fitness activity.
- 3:31:31It's good for our
- 3:31:34body okay that's it and everything is like before
- 3:31:38Okay, I run the program. Okay, notice what we have done here.
- 3:31:43We have changed the task from sentiment analysis
- 3:31:45to classifying text into broader categories
- 3:31:48related to fitness, health, sports. And as you can see, it works correctly.
- 3:31:53This piece of text is related to health and it correctly labels it as health.
- 3:31:59As you saw, we didn't train a new model and we
- 3:32:01didn't change the underlying classification
- 3:32:04implementation. We only changed. the candidate labels and the input. That's it.
- 3:32:08And that's the power of zero shot classification.
- 3:32:11We can use the same model for different classification
- 3:32:15tasks, simply by providing a different set
- 3:32:17of candidate labels.
- 3:32:19And that is one of the reasons NLI-based models
- 3:32:23are so useful and powerful in AI engineering.
- 3:32:26Welcome back aboard Neon Learned.
- 3:32:29Planet Classifier is conquered, two worlds down, eight to go.
- 3:32:33Fasten your seat belts. Next stop, Planet Embeddings.
- 3:32:37Here, you'll master embeddings and build search
- 3:32:39that understands meaning, not just keywords.
- 3:32:45Your next world awaits. Enjoy the journey.
- 3:32:48Oh
- 3:32:54Okay, in our next project, we are going to explore embeddings.
- 3:32:57Embeddings are one of the fundamental concepts you need to understand.
- 3:33:01You need to understand when working with LLMs and AI engineering in general.
- 3:33:05They are not only an important concept in many AI
- 3:33:08engineering applications, such as RAG, RAG systems,
- 3:33:11but they can also be used as a foundation for tasks in their own right.
- 3:33:17and that's something we are going to explore in this project.
- 3:33:20So let's start by understanding the idea behind embeddings.
- 3:33:24Let me start with an example.
- 3:33:26As humans, we receive information in many different forms.
- 3:33:29It can be text, images, audio, and many other types of data.
- 3:33:33Now, when we hear a sentence, when when we read something or receive information
- 3:33:39in some other way, our brain processes that information and forms
- 3:33:44a mental representation of what it means.
- 3:33:47We still don't fully understand every detail of what happens inside the human brain.
- 3:33:52But the important point is that we form an
- 3:33:55understanding of the information we receive.
- 3:33:58For example, when I tell you I like black cats, you
- 3:34:01might immediately imagine a black cat in your mind.
- 3:34:05The concept of embeddings aims to achieve something similar for computers.
- 3:34:10The idea is to take an input.
- 3:34:12and convert it into a numerical representation that
- 3:34:16captures important information about its meaning.
- 3:34:19And
- 3:34:20question is that how do
- 3:34:22we represent that meaning?
- 3:34:24We represent it using numbers. Computers only deal with numbers.
- 3:34:30And for embedding the way we do it, we use a pre-trained model.
- 3:34:34This model receives a piece of data as input and converts it into vector.
- 3:34:40which is simply a list of numbers.
- 3:34:42For now, imagine that this vector has only two dimensions. What does that mean?
- 3:34:48It means we can represent the vector as a
- 3:34:51point on a two-dimensional coordinate system
- 3:34:54with an x-axis and y-axis.
- 3:34:57Now, suppose we give the model the word cat.
- 3:35:01The model converts it into a point in this two-dimensional space.
- 3:35:06If we give it the word dog, we get another point.
- 3:35:09We can do the same thing with ship and car. Now, here's the important part.
- 3:35:15Because cats and dogs are semantically related,
- 3:35:19a well-trained embedding model will generally represent them
- 3:35:23in a way that makes their vectors relatively close.
- 3:35:27Similarly, the representations of ships and cars
- 3:35:31may be closer to each other than they are to the representation of cats and dog.
- 3:35:36In other words, the model learns to represent
- 3:35:39relationships between concepts in a numerical space.
- 3:35:43Of course, this is a simplified illustration.
- 3:35:46Real embedding spaces are much more complex,
- 3:35:49and the relationships between vectors depend on how the model was trained.
- 3:35:54But this gives you the basic idea of embedding models. And you should know that
- 3:36:00Having this concept, I mean having a model that can convert data
- 3:36:04into meaningful numerical representations is extremely powerful
- 3:36:08because it allows us to build many different types of applications.
- 3:36:12For example, embeddings are useful for semantic search, which is finding
- 3:36:17information based on meaning rather than just matching exact keywords.
- 3:36:23Or recommendation systems to find products, articles or
- 3:36:27other items that are similar to something a user likes, a user prefers.
- 3:36:32Spam detection, which is representing messages in a way that can help a
- 3:36:37classifier distinguish spam from legitimate messages.
- 3:36:41Or fraud detection, using embeddings as features
- 3:36:45to help identify suspicious patterns or
- 3:36:48RAG systems, the application that we want to have.
- 3:36:51Finding relevant documents to provide as context to an LLM.
- 3:36:56Of course, building a high-quality embedding model from scratch isn't easy.
- 3:37:01I simplified it.
- 3:37:02It requires substantial mathematical knowledge, large amounts
- 3:37:07of training data, computing resources, and time of course.
- 3:37:11But here's the good news.
- 3:37:13We are working in AI engineering, and we are not
- 3:37:16going to train a foundation model from a scratch.
- 3:37:19Our goal is to use these models, to learn how to use
- 3:37:23these models effectively. So we don't need to train them, we just want to use them.
- 3:37:28And there are many, many commercial and freely available embedding models
- 3:37:33that we can use directly in our application. So, we don't need to build
- 3:37:37an embedding model ourselves.
- 3:37:39Another point is that, I mean, another thing that I simplified was about
- 3:37:43dimensions. I told you that we had a two-dimensional space.
- 3:37:48In reality, embedding models typically represent text in
- 3:37:52spaces with hundreds or sometimes thousands of dimensions.
- 3:37:56Each input is converted into a vector containing
- 3:38:00many numerical values, several hundreds.
- 3:38:02And again, another thing is that don't assume
- 3:38:06that the input has to be a single word.
- 3:38:08It could be a singled word, of course.
- 3:38:11It could be a sentence, a paragraph, or even a longer document,
- 3:38:15several pages, depending on the model and its limits.
- 3:38:19Embedding models have maximum input lengths, just like other language models.
- 3:38:24As long as the input fits within the model's supported
- 3:38:28length, we can generate an embedding for it.
- 3:38:31For longer documents, we may need to split
- 3:38:34the text into smaller chunks first and then
- 3:38:37we do the task. And embeddings aren't limited to text.
- 3:38:41There are models designed to generate embeddings
- 3:38:44for other types of data, including images,
- 3:38:46audio, and some multimodal models can even represent different types of
- 3:38:51data in a shared embedding space, making it
- 3:38:54possible to compare information across modalities.
- 3:38:59to compare a video with a piece of text to find
- 3:39:02out whether that video is related to that
- 3:39:05text.
- 3:39:06And as I mentioned earlier, there are many embedding models available.
- 3:39:11For example, OpenAI offers
- 3:39:12embedding models in addition to its generation models.
- 3:39:15Different embedding model in OpenAI produce
- 3:39:18vectors with different numbers of dimensions.
- 3:39:20For example, there is a model that produces 1500 dimensional
- 3:39:25vectors and there is another larger model that
- 3:39:28produces 3000 dimensional vectors.
- 3:39:31And generally speaking, larger embedding vectors
- 3:39:34require more storage and computational resources.
- 3:39:38A model with more dimensions. may capture useful distinctions more effectively.
- 3:39:43So another way we can say that it's stronger,
- 3:39:46but more dimensions don't automatically mean
- 3:39:49better results.
- 3:39:50The quality of the model depends on many factors
- 3:39:53like the way it is trained and so when you
- 3:39:56want to use it in your tasks, there are some consequences. considerations.
- 3:40:00And in addition to OpenAI, another provider you will often hear is Coheer.
- 3:40:06Its embedding models are used in applications
- 3:40:09such as Semantic Search and Retrieval Systems.
- 3:40:12And for enterprise projects where quality and
- 3:40:15reliability are important, commercial embedding
- 3:40:18models can be a very good option. It is worth investing.
- 3:40:23but if you want to keep your costs down there are
- 3:40:26many excellent open source embedding models
- 3:40:29some of them are small enough to run directly on
- 3:40:33your personal computer and in this project we will
- 3:40:36use a model that we can do we can download it and use it on our own computer
- 3:40:49For this project we are going to use a very popular embedding
- 3:40:52model called All Mini LM and let's search for it in Hugging Face.
- 3:40:56And now let me explain where this name comes from.
- 3:40:59First let's look at this part, All.
- 3:41:02This refers to the model being designed as a general purpose sentence embedding
- 3:41:07model and it is trained on a broad collection of data sets and that's why
- 3:41:13It can be used for a broad range of tasks, including
- 3:41:17semantic search, center and similarity, and clustering.
- 3:41:21And you can use it for texts about health, technology,
- 3:41:25sports, and many other subjects and domains.
- 3:41:29And next we have Mini-LM.
- 3:41:31Mini-lm is a family of compact transformer models associated with Microsoft.
- 3:41:38The model we are using builds on another model called Mini LM L6 H384
- 3:41:46and it's a smaller encoder model designed to
- 3:41:49be effective and small at the same time.
- 3:41:51And then we have L6.
- 3:41:54The L6 indicates that the underlined transformer encoder has six layers
- 3:42:00and you should know that neural network models
- 3:42:03are made up of layers that progressively transform the input to the output.
- 3:42:09And finally, what does V2 mean?
- 3:42:12It identifies this particular version of the model.
- 3:42:17So we have a general purpose embedding model based on a compact six layer
- 3:42:22transformer encoder with version 2.
- 3:42:25And one more useful detail is that this model produces 384 dimensional embeddings.
- 3:42:32It's relatively small and fast, which makes it a good choice for learning
- 3:42:37and for even many real world applications.
- 3:42:40So,
- 3:42:46To work with embedding models, we can use a
- 3:42:48Python library called sentence transformers.
- 3:42:51So let's start by installing it by installing sentence-transformers.
- 3:42:55So on the terminal, I say pipenv install sentence-transformers.
- 3:43:03Okay, great.
- 3:43:04And while it is installing it, I'm going to
- 3:43:08create a new Python file called embedding.py.
- 3:43:12So here I say embedding dot py.
- 3:43:15hi okay here first i'm going to import dot env so
- 3:43:19i say import dotenv and then i load environment
- 3:43:24variables from our dotenv file so i'll say dotenv
- 3:43:27dot load dotenv notice that we don't need to
- 3:43:31import os module here we are not going to read the token ourselves
- 3:43:36The sentence transformers library is going to do that.
- 3:43:39Next, we are going to import sentence transformer itself.
- 3:43:43As you can see, it was successfully installed and now I can import it.
- 3:43:48I can say from a sentence transform. Import.
- 3:43:53sentence transformer it's a it's a class and we are going to create an
- 3:43:58instance of this class let's do it let's create an instance of
- 3:44:02this class and tell it which model we are
- 3:44:04going to use to avoid any typo i'm going to copy this name from here i paste it
- 3:44:12now let's have a few sentences a few
- 3:44:16sentences that we want to convert into embeddings i guess or some
- 3:44:21examples here so yes all I need are these examples
- 3:44:26so I copy these examples and I paste them here
- 3:44:31and I store the result of this line in a
- 3:44:33variable called model because it's going to
- 3:44:36be
- 3:44:36the model that we use to embed our query or our sentences
- 3:44:41Now let's create our main function.
- 3:44:44I define my main function as keyword and our usual entry point.
- 3:44:49I say if double underscore name
- 3:44:52double underscore is equal to double underscore main double underscore.
- 3:44:58In that case, I want the main function to be executed,
- 3:45:01something that we have had a lot in this course.
- 3:45:05Now inside main we can call the model and pass in our list of sentences.
- 3:45:10These sentences I mean.
- 3:45:12So here I use this model.
- 3:45:14This model provides a method called encode and
- 3:45:17it gets the sentences that we want to embed.
- 3:45:20And it's nicer if we move these sentences
- 3:45:24inside this function because it would be the only
- 3:45:28place that we are going to use these sentences.
- 3:45:32And I'm going to store the embeddings in a variable called embeddings.
- 3:45:36So they say embeddings now. I'm gonna print embeddings shape.
- 3:45:42I don't want to see the embedding themselves. I only want to the dimension.
- 3:45:48So I run the program and the first time you run it, it may take a little
- 3:45:52longer because the model needs to be downloaded, loaded and initialized.
- 3:45:57And on subsequent subsequent runs, it will usually be faster.
- 3:46:01I'm waiting for it to be executed. Okay. Weights are loaded. Okay.
- 3:46:06And you can see this number here, four and 384. What does that mean?
- 3:46:11The first number, I mean four, tells us the
- 3:46:14model generated four, four embeddings, one
- 3:46:18embedding for, for each sentence.
- 3:46:19Here we had four sentences and now we have four embedding.
- 3:46:24And the second number 384 tells us that each embedding contains three
- 3:46:30384 numerical values.
- 3:46:33In other words,
- 3:46:34each sentence has been converted into a vector with 384 dimensions.
- 3:46:42You can think of each vector as a point in a 384 dimensional space.
- 3:46:47Since we have four sentences, now we have 4 points in that space.
- 3:46:53And that brings us to the next question.
- 3:46:56What interesting things can we actually do with these embeddings? Let's find out.
- 3:47:02Oh
- 3:47:08In this session, we are going to learn the basics of Semantic Search.
- 3:47:11Let's say we have a collection of sentences and
- 3:47:14we receive a new sentence called a query.
- 3:47:16Our goal is to find the sentence in our collection
- 3:47:20that is most similar in meaning to our query.
- 3:47:23One way to do this is to calculate the similarity
- 3:47:26between the query and each of those sentences in our
- 3:47:30collection and then return the most similar one. But the question is, how
- 3:47:35How do we do this? How do measure this distance? How do we calculate the similarity?
- 3:47:41In the previous lesson, we converted each of our four
- 3:47:46sentences into a point in a 384 dimensional space.
- 3:47:51And we did it using an embedding model. Now we can do the same thing with our query.
- 3:47:57We convert it into a vector in the same embedding space. And then in that space,
- 3:48:03we can compare it with the embeddings that we generated for our sentences,
- 3:48:08from our sentences from our collection. So the idea is simple.
- 3:48:13find the sentence whose embedding is most similar
- 3:48:17to the query embedding and return that sentence.
- 3:48:21But this raises an important question.
- 3:48:23How do we measure similarity between two vectors?
- 3:48:27There are several metrics we can use but one
- 3:48:30of the most common is cosine similarity.
- 3:48:33Cosine similarity measures the cosine of the angle between two vectors.
- 3:48:38In a simplified geometric picture, vectors pointing in
- 3:48:42similar directions tend to have higher cosine similarity.
- 3:48:46And
- 3:48:46the score ranges from minus 1 to 1.
- 3:48:50A score of 1 means the vectors point in the same direction.
- 3:48:55A score means they are orthogonal, meaning there is no directional similarity.
- 3:49:02and sq means they point in opposite direction.
- 3:49:06And for a purpose, a higher cosine similarity
- 3:49:10means a better match, means those two sentences
- 3:49:13are closer. Alright, let's see how this works in code.
- 3:49:18First, I should import util again from sentence transformers and the
- 3:49:23This module provides several useful functions. One of which is cosine similarity.
- 3:49:29It calculates cosine similarity between vectors. Let's remove these lines here.
- 3:49:33The lines that we have for encoding the sentences and printing their shape.
- 3:49:38Because we are going to handle the embeddings inside another function.
- 3:49:43So I create another function for finding similarity.
- 3:49:48Let's call this function find closest. closest. So I define a new function called
- 3:49:54find
- 3:49:56closest and this
- 3:49:57function takes a query and a list of sentences and returns the sentence that is most
- 3:50:05similar to that query so first we have query
- 3:50:10and second we
- 3:50:12have sentences
- 3:50:13okay
- 3:50:15pass and then inside my main function i'm gonna call this function i am gonna
- 3:50:22say find closest
- 3:50:24I'm calling it, I give it a query or we can put the query inside the variable.
- 3:50:30I say I define
- 3:50:32a new variable called query and I set it to I am
- 3:50:36feeling great today.
- 3:50:40So I pass a query here as its first argument, and then I pass my collection
- 3:50:47of sentences.
- 3:50:49And then I store the result in a variable called close a sentence, and
- 3:50:55then I print it.
- 3:50:57I print closest sentence. Okay, that's pretty much it.
- 3:51:01Now the only thing left is to implement our find closest function here.
- 3:51:07So here we need to embed both query and sentences.
- 3:51:12So here I say model.encode.env.
- 3:51:15and I send sentences and I do the same for the query.
- 3:51:22and I store these two into separate variables, I call it.
- 3:51:29embeddings and
- 3:51:30I call the second one query
- 3:51:32embedding.
- 3:51:34query embedding.
- 3:51:36okay that's it now we have the embedding for every
- 3:51:39sentence in our collection as well as the embed
- 3:51:42for our query our next task is to find the sentence
- 3:51:46with the highest cosine similarity score to do
- 3:51:49that we need to keep track of two things the index
- 3:51:52of the closest sentence and its similarity score
- 3:51:56so here we need two variables first closest index
- 3:52:00and we need another variable called closest score
- 3:52:05First, we initialize closest index to zero.
- 3:52:08In other words, we are assuming that the first
- 3:52:11item is the item that we are looking for.
- 3:52:13And we also initialize the closest score to
- 3:52:16the lowest possible cosine similarity score,
- 3:52:19which is negative one. Now, this is the procedure.
- 3:52:22We need to iterate through all sentences.
- 3:52:25And if we find a closer embedding, so we update these two variables.
- 3:52:31I mean index and the score. So we need to iterate over it.
- 3:52:35Let's use enumerate function because we also need the index.
- 3:52:39So I call enumerated and I give it my embeddings
- 3:52:44and it gives me index as well as each vector
- 3:52:48in my embedding. So I save for index as there as vector.
- 3:52:54in my embeddings.
- 3:52:57Okay, inside this loop, we'll calculate the cosine
- 3:53:00similarity between the current sentence and our query.
- 3:53:04So here I use the util module that we imported and I call
- 3:53:08cosine similarity function and I give it the vector that we
- 3:53:12have in this iteration as well as the query embedding.
- 3:53:16Then I use the item method and I store the result in a variable called similarity.
- 3:53:25okay this item method you see here extracts the
- 3:53:29scalar similarity score from the returned tensor
- 3:53:33yes it returns a tensor but we need a score and
- 3:53:37this is a conversion next we need to check we need
- 3:53:41to check whether the current sentence has a higher
- 3:53:44similarity score than the best score we have seen
- 3:53:47so far or not so if it does we update both closest index and closest score.
- 3:53:54So here we need an if statement.
- 3:53:56So I say if similarity
- 3:53:58is greater than closest score, pay attention.
- 3:54:03In that case we're gonna update both these two
- 3:54:08i copy and paste them here here i put similarity and
- 3:54:12then i say index the index that we have in this
- 3:54:16iteration and the similarity that we calculated
- 3:54:20okay so with this logic in our code as we continue
- 3:54:24through the loop we keep comparing each sentence with our query and we keep updating
- 3:54:30these variables and when we are done we have a closest index and index not
- 3:54:37in text.
- 3:54:38we have closest index and closest core and then we can return
- 3:54:42so here at the end after this loop i'm going to return
- 3:54:47sentences but the index that we found we calculated so here i use
- 3:54:54closes index okay that's pretty much it we have implemented a basic semantic search
- 3:55:00function from scratch and let's run and see what happens okay i run the program my
- 3:55:05query was i'm feeling great today let's see what will happen okay here we have a
- 3:55:11problem what is a problem oops my bad here we need to say index and here we
- 3:55:17need to save similarity one more time i hope this time everything is all right
- 3:55:24Okay, as you can see, the closest sentence to my query is today is a sunny day.
- 3:55:31Okay, we implemented this function manually, so you
- 3:55:34could understand what's happening behind the scenes.
- 3:55:38But I should tell you, we don't have to do that.
- 3:55:40I mean, Sentence Transformers already provides a utility
- 3:55:44function called semantic search that can do the same for us.
- 3:55:48There is an important difference, however.
- 3:55:51semantic search i mean this utility function that
- 3:55:54i'm telling you doesn't take raw sentences and you
- 3:55:57need to embed your query and your sentences your
- 3:56:00collection of sentences first and then you can
- 3:56:03pass them as the argument of this function
- 3:56:07this utility. function so here first i say
- 3:56:11model.encode and you can borrow some code from here yes that's better
- 3:56:19Okay.
- 3:56:21I paste it here.
- 3:56:23Okay.
- 3:56:24I embed both my query and my sentences.
- 3:56:29Then everything is ready to use this utility function. So I say util.
- 3:56:34semantic search and first i send query query embedding then i send
- 3:56:39embeddings sentence embeddings and we could we could use better names
- 3:56:45here but it doesn't matter and then we specify the top k and i set it
- 3:56:50to two and the top K you see here tells the function how many of the
- 3:56:55most similar results we want the functionality that we didn't have
- 3:57:01in our function in our own function it always
- 3:57:04returns the most similar sentence but here you can
- 3:57:08receive the top k most similar sentences
- 3:57:12Anyway, let's store the result in a variable called results.
- 3:57:16Then I print it and let's run our program to see what happens.
- 3:57:21Okay, as you can see here, the output is a little
- 3:57:24different from our manually implemented function.
- 3:57:27Instead of returning the sentences themselves,
- 3:57:30SemanticSearch returns a list of result dictionaries.
- 3:57:34Each result indicates an ID, which is the
- 3:57:37index of the matching sentence and a score.
- 3:57:41which represents its similarity to the query.
- 3:57:43For example, you can see here that it says the closest
- 3:57:47one has the index 3, which is the last item.
- 3:57:50And that is the similarity score.
- 3:57:54So that brings us to the core idea behind Semantic Search,
- 3:57:59which is encoding text into embeddings, compare
- 3:58:03those embeddings using a similarity metric,
- 3:58:07and retrieve the text that best matches the meaning of our query.
- 3:58:13Welcome back aboard NeonLearn.
- 3:58:15Planet Embeddings is conquered, three worlds down, seven to go.
- 3:58:20Fasten your seat belts. Next stop, climate red.
- 3:58:24Here, you'll build an AI support agent that answers from real company data
- 3:58:28and begin your journey with Langchain.
- 3:58:32Your next world awaits. Enjoy the journey.
- 3:58:41Okay, from this point on, we are going to start
- 3:58:44learning about RAG systems, which stands
- 3:58:46for Retrieval Augmented Generation.
- 3:58:49RAG is one of the most important topics in AI engineering.
- 3:58:53You will see it mentioned frequently in job
- 3:58:56descriptions, and many companies are looking
- 3:58:58for engineers who know how to build RAG-based applications. But what exactly is RAG?
- 3:59:05What problem does it solve and how does it work?
- 3:59:08Okay, let's start with a problem we discussed
- 3:59:11earlier and it relates to the limitations of
- 3:59:14LLMs.
- 3:59:15Remember when I described an LLM as something like a person, a person who has read
- 3:59:22a huge amount of information available on the internet.
- 3:59:26That analogy helps us understand
- 3:59:28why LLMS can answer so many general questions.
- 3:59:32They have been trained on enormous amounts of data.
- 3:59:36they can often provide useful answers across a wide range of topics.
- 3:59:41But, of
- 3:59:42course, they have limitations.
- 3:59:44And the first limitation is something called the knowledge cutoff.
- 3:59:49A model's knowledge cut off refers to the point in time beyond which
- 3:59:53its training data may not provide reliable information about events.
- 3:59:58For example, if a model's training data only covers information up to 2018,
- 4:00:04it might not know about events that happened in 2019 or later.
- 4:00:10So if you go beyond knowledge cut off and if you ask a
- 4:00:14model about a recent event it hasn't learned about,
- 4:00:18it may not have the information it needs to answer the question correctly.
- 4:00:23And another limitation is private access.
- 4:00:27Models don't have access to private or company-specific information.
- 4:00:31For example,
- 4:00:33the model doesn't know about your company's internal policies.
- 4:00:37It doesn't have access to
- 4:00:38your customer records or the details of your business.
- 4:00:42So as a result, there are many situations
- 4:00:45in which an LLM cannot give us a good answer and that's because it doesn't
- 4:00:51have the relevant information and that is
- 4:00:54exactly the problem RAG is designed to solve.
- 4:00:57And
- 4:00:57One way to think about RAG is as a method for improving an LLM's response
- 4:01:02by giving it access to relevant information at the time we ask a question.
- 4:01:07If the model doesn't have up-to-date information, we can do that.
- 4:01:11We can retrieve that information for the model and provide it as context.
- 4:01:15If it doesn't know our company's internal policies, we
- 4:01:19retrieve the relevant policies and provide them to the model.
- 4:01:23This way the model can use information that wasn't necessarily part of its training
- 4:01:28and we kind of compensate for that. And let's look at a more tangible example.
- 4:01:35Imagine that I own a gym and I want to build a chatbot assistant for my business.
- 4:01:40My gym has a website and I'd like to add a
- 4:01:43chat bot button to the corner of the page.
- 4:01:46So visitors can click on and ask a question, something that you have seen a lot.
- 4:01:52Now imagine a customer asks, I attended one session, but I wasn't satisfied.
- 4:01:59I'd like a refund. What should I do?
- 4:02:01So the LLM for sure doesn't know about our gym's cancellation or refund policies.
- 4:02:08So we need to provide that information somehow.
- 4:02:11Otherwise, the model might give a generic answer.
- 4:02:14And this brings us to another important problem.
- 4:02:18Hallucinations.
- 4:02:19In these scenarios, hallucination happens a lot.
- 4:02:23And as we discussed earlier, LLMs are trained to predict and generate text.
- 4:02:28And they are very good at this task.
- 4:02:31And they try to give natural and convincing answers.
- 4:02:36But sounding convincing doesn't necessarily mean being correct.
- 4:02:41When a model doesn't have enough information, it might still generate
- 4:02:46an answer instead of clearly acknowledging that it doesn't know.
- 4:02:50For example, our gym chat pod may respond.
- 4:02:54I'm sorry, you had a bad experience, we know.
- 4:02:57You can contact the management for a refund.
- 4:02:59And that's terrible because our gym might not have a refund policy.
- 4:03:04So, chatbot is offering something that doesn't exist.
- 4:03:08So, it's giving incorrect information and this is an example of hallucination.
- 4:03:13The model generates information that isn't supported by facts.
- 4:03:17And RAG systems are going to help reduce this problem.
- 4:03:21And it is done by retrieving relevant information, of course,
- 4:03:25from a trusted knowledge base like our policy documentation.
- 4:03:29And provided to the model before it generates the response.
- 4:03:33For example, instead of asking the LLM to
- 4:03:36answer the customer's question in first place,
- 4:03:39we can first retrieve our gym's actual cancellation policy.
- 4:03:44Then we can provide both the customer question and the relevant policy.
- 4:03:48And now the model has the information that it
- 4:03:52needs to generate a more accurate answer.
- 4:03:55And before going any further, there is something important to understand.
- 4:04:01I mean Retrieval Augmented Generation is a general concept.
- 4:04:06There isn't just one way to implement it.
- 4:04:09The best implementation depends on the task,
- 4:04:12on project, the data we are working with.
- 4:04:14And a RAG system contains several components and
- 4:04:18those components need to work together effectively.
- 4:04:23And it brings us to another important thing about RAG systems, which is monitoring.
- 4:04:29It's really important to monitor the system. It's very important to evaluate it.
- 4:04:34Simply implementing something that runs without errors
- 4:04:37doesn't necessarily mean we have built a good RAG system.
- 4:04:41For example, one factor that can have a significant
- 4:04:44impact is chunk size, and we will explore it.
- 4:04:47But at this point, I'm telling you that observability and
- 4:04:51evaluation is an important part of implementing RAG systems.
- 4:04:55Now, before finishing this lesson, I'm going to elaborate on rag words.
- 4:05:00I mean, let's break down the three words that make up rag.
- 4:05:04Retrieval, augmented, and generation.
- 4:05:07Let's start with the first word, retrieval.
- 4:05:10Retrieval means finding and collecting the
- 4:05:13information that's relevant to a user's question.
- 4:05:16Let's return to our example, our gym example.
- 4:05:19Suppose we have a 100 page PDF containing all the gym's guidelines and policies.
- 4:05:26Okay, a customer asks about cancellations.
- 4:05:30And here's a point where we don't need every single page of that PDF.
- 4:05:34We only need the sections related to cancellation and refund policies.
- 4:05:39So the process of finding those relevant sections is called information retrieval.
- 4:05:45And in a RAG system, the component responsible for
- 4:05:49finding that information is called retriever.
- 4:05:52Okay.
- 4:05:53The retriever searches our knowledge base and returns the
- 4:05:57information that is most relevant to the user's query.
- 4:06:01So the retriever's job is to find the right information and bring it back.
- 4:06:06And as we will see how well it performs, this job has a
- 4:06:10significant impact on the quality of the final answer.
- 4:06:14So this component is extremely important. The second word is augmented.
- 4:06:20Augmented means enhanced or supplemented.
- 4:06:23Before we implement our RAG, our LLM might not
- 4:06:27have had access to our GIMS' private policies.
- 4:06:31But now, we have retrieved the relevant information and can provide it to the model.
- 4:06:36We are kind of augmenting the model's input with
- 4:06:39additional context, giving it information that it needs.
- 4:06:43So, as a result, the model has a better basis for generating its response.
- 4:06:49And finally, we have generation. This is the part that the LLM generates an answer.
- 4:06:54So let's put the three parts together. First, we receive a query from the user.
- 4:06:59And in our example, the user asks what they should do if
- 4:07:03they want to cancel a gym membership or request a refund.
- 4:07:06We send the query to a retriever.
- 4:07:08The retriever searches our knowledge base, which contains the gym's PDF guidelines.
- 4:07:15and then returns the relevant information about cancellations and reforms.
- 4:07:20Now we have two things, the original query from the user and the relevant
- 4:07:24information retrieved from our knowledge base. And then we combine these two pieces
- 4:07:30and we send it as a prompt to our LLM. So the LL M uses the query and the
- 4:07:35retrieved context to generate a response. Here's a complete flow.
- 4:07:40Before we had a RAG system, we simply send the user's query to the LLM.
- 4:07:44But as we have seen, that approach can fail when
- 4:07:47the model doesn't have enough information.
- 4:07:49However, with RAG, we retrieve the relevant information.
- 4:07:53We provide it to the model as a context.
- 4:07:56And this gives the model a better chance of producing a correct and useful answer.
- 4:08:02And that's the whole idea behind retrieval augmented generation.
- 4:08:17Okay, now that we understand the overall idea,
- 4:08:20let's take a closer look at the retriever component and how it works.
- 4:08:25Imagine our knowledge base contains the same 100 page PDF.
- 4:08:29You might ask, why do we need a retriever at all?
- 4:08:32Why don't we simply send the entire PDF to the LLM along with the user's query?
- 4:08:37Wouldn't it be better to give the model all the information
- 4:08:41so it can find the answer itself? Well, not necessarily.
- 4:08:45There are several catches, there are several important
- 4:08:49reasons why this isn't always a good idea.
- 4:08:51First, remember that every LLM has a context window.
- 4:08:55The context window limits how much information
- 4:08:58the model can process in a single interaction.
- 4:09:01So, this includes the prompt, the conversation history, retrieved
- 4:09:06information, and the space needed for the generated response.
- 4:09:10So if we put the entire 100-page PDF into the prompt,
- 4:09:15we could consume a large part of our context window.
- 4:09:19In the worst case, the input could exceed the
- 4:09:22model's limit and the request might fail.
- 4:09:26even when the entire document fits, using so much context may be unnecessary.
- 4:09:32So, it brings us to the second reason, cost.
- 4:09:36When it's unnecessary, we are burning our tokens.
- 4:09:39When we use a paid API, we're charged based on token usage.
- 4:09:44So, if we send the entire knowledge base with query question,
- 4:09:48we may repeatedly pay to process large amounts of
- 4:09:51information that aren't relevant to the question.
- 4:09:54even if we are running the model locally, we
- 4:09:58are wasting hardware, time and computational
- 4:10:01resources in general and it increases latency
- 4:10:04in our system so we cannot respond to other
- 4:10:07users.
- 4:10:08To make it more clear, let's look at an example.
- 4:10:13our knowledge base contains 100,000 tokens.
- 4:10:16Now, if 10 users ask questions and we send the entire knowledge base with query,
- 4:10:22that could mean processing around 1 million input tokens.
- 4:10:26So ignoring other stuff, it could be 1 million
- 4:10:30input tokens, but what if each question only needs a small section of documents?
- 4:10:35For example, it only needs around
- 4:10:37100 tokens instead of 100,000 tokens.
- 4:10:40So in that case, across 10 requests, that's only around
- 4:10:441,000 token.
- 4:10:46So you can compare 1 million input tokens and 1, 000 input tokens.
- 4:10:51That's a big difference.
- 4:10:53So the principle is clear.
- 4:10:55Retributing only the relevant information can greatly reduce
- 4:11:00the amount of unnecessary context.
- 4:11:03So it was the second reason. And the third reason is answer quality.
- 4:11:08Giving the model more information doesn't
- 4:11:11automatically mean giving it better information.
- 4:11:14Imagine our gym has a swimming pool, a weight
- 4:11:17training area, and several other facilities.
- 4:11:20Our PDF contains information about all these stuff.
- 4:11:23Now a customer asks about the cancellation policy as I told you.
- 4:11:28So if we send the entire PDF,
- 4:11:30The model also receives information about pool depth, swimming rules,
- 4:11:35children's safety, gym equipment, and many other topics that are irrelevant.
- 4:11:40So, I acknowledge that modern LLMs use attention mechanism, and this mechanism
- 4:11:46helps them process relationships between different parts of the input.
- 4:11:52But irrelevant information can still make it harder
- 4:11:55to focus on the information that actually matters.
- 4:11:59So that's really important and we don't just
- 4:12:02want to give the model more information.
- 4:12:05We want to get it the right information and
- 4:12:08that's where retrieval becomes so important.
- 4:12:12Now after knowing that retriever and retrieval augmented generation is important,
- 4:12:17the question is that how do we solve this problem? If the problem is
- 4:12:22We cannot give the model the entire document. How do we solve this problem?
- 4:12:28What should we do?
- 4:12:29One approach is to use the embeddings that we just learned in the previous project.
- 4:12:34Let's say our knowledge base is that 100-page PDF.
- 4:12:39The first thing we can do is to split it into smaller pieces.
- 4:12:43For example, we might split it into paragraphs or sections.
- 4:12:47And this process is called chunking.
- 4:12:49For example a 100-page PDF might be converted
- 4:12:53into hundreds or even thousands of chunks.
- 4:12:56And remember, our knowledge base doesn't have to be a PDF.
- 4:13:00It could contain Excel files, web pages, internal
- 4:13:04documentation or many other types of data.
- 4:13:08And once we have split the context into chunks,
- 4:13:11we generate an embedding for each chunk.
- 4:13:14In other words, we convert each chunk into a numerical vector in an embedding space
- 4:13:21that we saw in the previous project.
- 4:13:23Now our knowledge base is represented by a
- 4:13:26collection of vectors instead of simple text.
- 4:13:29And each vector is associated with its original text. Now what happens?
- 4:13:35What happens when a user asks a question? That's really simple.
- 4:13:38We take that question. and pass it through the embedding model that we have.
- 4:13:42So we convert the query into a vector in the same space.
- 4:13:47Then we compare the query embedding with the embeddings for chunks.
- 4:13:52We can use a similarity measurement, such as the cosine that you saw.
- 4:13:57So we can find which chunks are most relevant to the query.
- 4:14:02And then the system receives the top matching chunks.
- 4:14:06Another question that you might have is why do we
- 4:14:09retrieve multiple chunks instead of just one chunk?
- 4:14:13because the information might be spread across several parts of our knowledge base.
- 4:14:19For example, one chunk might describe the cancellation
- 4:14:23deadline while another explains the refund conditions.
- 4:14:27So retrieving multiple relevant chunks can give the model
- 4:14:31a more complete picture which is exactly what it needs.
- 4:14:36So once retrieval is complete, we have the
- 4:14:39original query and the relevant text chunks.
- 4:14:42So we combine these two.
- 4:14:44We combine retrieved chunks and the query in
- 4:14:47a single prompt and send them to the LLM.
- 4:14:50And the LLLM can use the retrieved information to generate an augmented answer.
- 4:14:56So let's summarize the process into this. First we have chunking.
- 4:15:00We split the knowledge base into smaller pieces. Then we have embedding.
- 4:15:05We convert each chunk into a vector. Then we query embedding,
- 4:15:10we convert the query into a Vector using the same model.
- 4:15:14And we have retrieval, we use a similarity metric
- 4:15:18and we find the chunks whose embeddings are most relevant to the query.
- 4:15:23And finally we have generation.
- 4:15:25We provide the query and retrieve chunks to the LLM.
- 4:15:28so it can generate an answer.
- 4:15:31And that's the core idea behind a basic embedding based RAG system.
- 4:15:36And in upcoming lessons, we will take a closer look at the Retriever
- 4:15:40and explore how to implement these steps in action.
- 4:15:47Ladies and gentlemen, a quick update from the control tower.
- 4:15:52Right now, AI engineer is the fastest growing job in the United States,
- 4:15:56ranked number one on LinkedIn's jobs on the rise list, two years in a row.
- 4:16:02And the top skills listed for this role, Lang Chain and Rag.
- 4:16:06The exact skills you're building on this journey.
- 4:16:09Companies are looking for people who can build with AI. The runway is ready.
- 4:16:15So stay consistent, one lesson at a time, one project at a time.
- 4:16:20Every hour you invest is fuel for your future. Your effort will pay off.
- 4:16:27you
- 4:16:33All right, from this point on, we are going to build a RAG project from scratch.
- 4:16:39The first thing that we need is a knowledge base.
- 4:16:41And for this project, we're going to use a PDF containing
- 4:16:46information about the symptoms of different diseases.
- 4:16:50The document is 25 pages long and contains information about approximately
- 4:16:56100 different diseases, it has a paragraph describing each disease.
- 4:17:02and I will upload this PDF and include a link
- 4:17:05in the description of this YouTube video.
- 4:17:08So you can download it and follow along with me. So that's our knowledge base.
- 4:17:14And this is a great document for practicing RAG
- 4:17:17because it gives us a collection of information
- 4:17:21that we can split into chunks, converts them into embeddings,
- 4:17:25and eventually used to retrieve relevant information.
- 4:17:29After that, we will move on to vector databases and
- 4:17:33I will explain why they are useful in RAG systems.
- 4:17:37Not only useful, but also necessary.
- 4:17:41So let's get started
- 4:17:43and let's create a new Python file called rag.py.
- 4:17:49And as usual, I want to have my main function.
- 4:17:54and my entry point. So I say name is equal to main
- 4:18:01in that case execute my main function so the
- 4:18:05first task is to read the pdf file there are
- 4:18:09several ways to do this but for this project
- 4:18:13i'm going to use a package called py pdf
- 4:18:16and we will also use langchain a very popular framework in ai engineering so first
- 4:18:23let's install these two so i open up my terminal and here i say pipenv install
- 4:18:33LangChain as well as PyPDF.
- 4:18:36Before start coding, let me briefly explain what
- 4:18:40LangChaine is and why we are going to use it.
- 4:18:44LangChains is a widely used framework that helps us
- 4:18:47build AI applications and it makes it easier for us.
- 4:18:51We can easily build AI application in an organized way.
- 4:18:55It provides reusable components that can help us write less code
- 4:19:00and we can integrate different parts of our application
- 4:19:04and it makes it easier to maintain and extend our application.
- 4:19:08We are going to use LangChain to implement parts of
- 4:19:11our RAG system, but keep in mind that it is not limited to RAG systems.
- 4:19:15It can be used for many other AI engineering tasks.
- 4:19:19And you will also hear about LangGraph. What is the difference?
- 4:19:24LangGraph is particularly useful when we need more explicit control when
- 4:19:29we are dealing with multi-step workflows including agentic system.
- 4:19:34Later we will take a look at LangGraph as well.
- 4:19:37Here LangChain is enough for us, is sufficient, is
- 4:19:40all we need for implementing this RAG project.
- 4:19:44And my general recommendation is to start with simpler tools.
- 4:19:49As long as they meet your needs, that's okay.
- 4:19:52You don't need to move to something more sophisticated.
- 4:19:56I see that a lot, but some developers prefer to overcomplicate a problem.
- 4:20:01So first use LangChain and when LangChains is not enough, you can go with LangGraph.
- 4:20:07Okay, back to our code. The first thing we need to do is to read our PDF
- 4:20:12file and
- 4:20:12we want to extract its text. So, as I told you, I'm going to use PyPDF.
- 4:20:18After that, I will use Langchain document object.
- 4:20:22What exactly is a document object?
- 4:20:25In Langchain, a document is a standard data structure
- 4:20:29that contains two main things, page content and metadata.
- 4:20:33Page content is the actual text or content of the document.
- 4:20:38and metadata is additional information about the content.
- 4:20:42For example, the metadata might include the source files name, the page number,
- 4:20:47or other information that helps us identify where the content came from.
- 4:20:52And this standard format is useful because it allows different
- 4:20:56components to work with information in a consistent way.
- 4:21:00It doesn't matter whether our original knowledge base comes
- 4:21:04from a PDF, a spreadsheet, a web page, or another source.
- 4:21:08We can extract the content.
- 4:21:10represent it using the same document structure and this becomes particularly
- 4:21:15useful when we start connecting components
- 4:21:18together in a RAG pipeline the output from one
- 4:21:22component can become the input to another component
- 4:21:26and that's the purpose of document object
- 4:21:29so let's start by reading our knowledge base which is a pdf
- 4:21:34so first i import pdf reader from pypdf so i say from
- 4:21:39pypdf import pdf reader then we need to import that document object that i told you
- 4:21:46it's a class so i say from langchain core
- 4:21:51documents okay import document okay that's
- 4:21:56it
- 4:21:57One thing worth mentioning here is that you may encounter older
- 4:22:01tutorials that import components from the Langchain community.
- 4:22:05Currently, the policy of Langchain is to provide standalone packages.
- 4:22:10So instead of having Langchain Community which was previously contained many, many
- 4:22:16tools and methods and functions and classes, now we have standalone packages
- 4:22:22So when you are following older tutorials, always check the current
- 4:22:26documentation to see which package provides the components you are using.
- 4:22:31So if you see LangChain Community, at that point search for and Standalone Package.
- 4:22:37So let's create a function to read our PDF.
- 4:22:40I define a new function called readPDF file which takes
- 4:22:44file name or more accurately it's the address of our file.
- 4:22:48Next, inside this function, we're going to create an empty list called docs.
- 4:22:54The process is that we'll create a PDFReader object and pass it
- 4:22:58our file path and it gives us access to the pages of our PDF.
- 4:23:03So first I create this object, I say PdfReader and I pass our path.
- 4:23:09and we will get a reader, a reader object and then using this reader
- 4:23:14we can access to pages of our document this way so reader.pages
- 4:23:19it is an iterable, we can iterate over it so I'm going to loop over it
- 4:23:26and since I also need index, I use enumerate it gives me both page and index
- 4:23:32so I save for I and page in these pages
- 4:23:37and inside the loop, we'll create a document object for each page, I mean.
- 4:23:43As we discussed earlier, each document contains the page's content and its metadata.
- 4:23:48I told you that it has two components.
- 4:23:51So we can extract the text from the current page and we can use the name of our file
- 4:23:56as well as its page using this index to be used as our metadata.
- 4:24:01So I'm going to create an instance of this document object for each each page
- 4:24:06and it has two components page content so first i
- 4:24:10extract it using the extract text method and it
- 4:24:14gives me the actual content of that page and here
- 4:24:18i use an or logical operator a page might be
- 4:24:22not so in that case i set it to an empty string and as it's metadata
- 4:24:28we have two pieces of information first we have
- 4:24:31source i give it the file name and then i specify
- 4:24:34the page number that this page comes from so i
- 4:24:38say page and i set it to i plus one because
- 4:24:41indexes start from zero okay now we have all
- 4:24:45we need and following that we can append
- 4:24:49this document that we created to our list so first i set
- 4:24:53store it in a variable called doc and then I
- 4:24:56say docs.append and I add this doc to it.
- 4:25:00And finally once we have processed every page, I mean after
- 4:25:04this loop, we return this docs list that we created.
- 4:25:08That's our PDF reader.
- 4:25:10Now before we test the function, let's take a
- 4:25:13moment to understand why we are storing metadata.
- 4:25:16I mean source and this page number here.
- 4:25:19Metadata is pretty useful when you are developing a RAG system.
- 4:25:23You need it in many different stages, from development
- 4:25:27to testing to evaluation and other stages.
- 4:25:29So we kind of need to track where our information comes from.
- 4:25:33For example, imagine that our RAG System answers a
- 4:25:37question using information retrieved from page 14.
- 4:25:41And if we preserve the source page number,
- 4:25:44we can identify the original location of that information.
- 4:25:48It can help us provide references to users.
- 4:25:51So it makes it more factual and it makes debugging easier as well.
- 4:25:56For example, if our chatbot produces an incorrect answer,
- 4:26:01we can investigate which document and page were retrieved as context.
- 4:26:06And that makes it easier to understand whether
- 4:26:08the problem came from the retrieval process,
- 4:26:11the source material, or the model's response.
- 4:26:14So metadata isn't just extra information, it's an important part of
- 4:26:19making our RAG system easier to inspect, evaluate and
- 4:26:23maintain as well as providing user with factual information.
- 4:26:27This way users can better trust us when we provide the reference.
- 4:26:31Let's test our program up to this point. So here I'm going to call this.
- 4:26:37I call this function, the function that we built.
- 4:26:40And as its argument, I pass this PDF.
- 4:26:44Okay, I say symptoms.pdf. Okay, and it gives me a list of documents.
- 4:26:51So I store it in a variable called docs.
- 4:26:54And
- 4:26:54then I print the length of this list as well as its first item.
- 4:27:01I say docs zero. Now let's run our program.
- 4:27:06Okay.
- 4:27:07As you can see here we have 25 pages.
- 4:27:11Okay, it was the length and the first item in this
- 4:27:15list, which is the first document actually we have.
- 4:27:19So first we have its content and then we have the metadata.
- 4:27:23It's the source that we have and this is the page number.
- 4:27:27So here we had 25 document objects and each document
- 4:27:32object contains both content and metadata.
- 4:27:36So we have successfully read our PDF and convert
- 4:27:39its pages into a LangChain document objects.
- 4:27:42This is our first step toward building the RAG pipeline.
- 4:27:52Alright, what have we done so far?
- 4:27:55We started with a 25-page PDF file and converted into 25 document objects.
- 4:28:01Now, you might think we can simply take these document
- 4:28:04objects and pass them to our embedding model.
- 4:28:07The model would convert each document into a
- 4:28:10vector in our 300-dimensional embedding space.
- 4:28:12But there is a problem.
- 4:28:14A full page is too much information to represent as a single embedding.
- 4:28:18and this is where domain expertise becomes important.
- 4:28:23If you look at our PDF, you will notice that each page can contain information about
- 4:28:29four or five different diseases. If we embed the entire page as a single unit,
- 4:28:35the resulting vector has to represent all of that information at once.
- 4:28:40And the problem is that the embedding might not
- 4:28:42capture the specific information we are looking for.
- 4:28:45For example, if a user asks about the symptoms of a particular disease,
- 4:28:50we want to retrieve the section that describes that disease,
- 4:28:54not an entire page containing several unrelated diseases.
- 4:28:58And here's the important part.
- 4:29:00If we don't store our information at the right level of detail,
- 4:29:05we cannot expect our retrieval process to be accurate either.
- 4:29:09So, instead of treating an entire page as one
- 4:29:12document, we want to split it into smaller pieces.
- 4:29:16Ideally chunks that contain a paragraph or a short section about a single disease.
- 4:29:23And this process is called chunking. Now,
- 4:29:28Splitting text isn't particularly difficult.
- 4:29:31I mean, we could write a simple function to break a string into smaller sections.
- 4:29:37but in real-world projects, developers usually rely on existing text splitters.
- 4:29:42Langchain provides several text splitters that
- 4:29:46work directly with its document objects.
- 4:29:49So since we are already using Langchain and its document object,
- 4:29:54it makes sense to use its built-in tools and follow the framework's conventions.
- 4:29:59So to use these text split-ers, we need to install an additional Langchain package.
- 4:30:05which is Langchain text splitters. So here,
- 4:30:09I'm going to install pipenv install langchain text splitters.
- 4:30:16Okay, and before we implement this, let's take a moment
- 4:30:21to understand the basic approaches to text splitting.
- 4:30:25The simplest approach is to split text based on a fixed size.
- 4:30:29For example, we could say that every chunk should
- 4:30:33contain 1000 characters or approximately 100 tokens.
- 4:30:37But there is a problem with this approach.
- 4:30:39It can split the text in the middle of a sentence or even
- 4:30:43in the middle of a word. Imagine that a sentence starts at the end of one chunk
- 4:30:49and continues in the next chunk. So the meaning can become harder to understand
- 4:30:55when those chunks are retrieved separately. To address this problem,
- 4:30:59we can use a strategy called recursive character splitting.
- 4:31:04The idea is to try splitting the text using natural boundaries,
- 4:31:09starting with larger sections, and gradually moving to smaller ones.
- 4:31:14For example, the splitter can first try to separate the text at paragraph breaks.
- 4:31:20If a resulting section is too large, it tries sentence-level separators like period,
- 4:31:26and if the text is still too large, it can fall back to smaller separators,
- 4:31:32such as spaces or individual characters.
- 4:31:35In this strategy, the main principle is to preserve
- 4:31:38the natural structure as much as possible,
- 4:31:41while keeping each chunk within the target size.
- 4:31:44So, this is why recursive splitting is a useful
- 4:31:47general purpose strategy for many RAG applications.
- 4:31:52And for this project, we'll use recursive characters. character text splitter.
- 4:31:57So let's import it. I'm going to say from LangchainTextSplitter,
- 4:32:03import recursive character textSplitters.
- 4:32:06And despite the word character you see here,
- 4:32:09this splitter can also work with token-based length measurements.
- 4:32:14and in many cases, measuring chunk size in tokens is more useful,
- 4:32:19more useful than measuring it in characters because language models process tokens
- 4:32:24and have token-based context limits
- 4:32:26and they describe everything with tokens rather than characters.
- 4:32:30So it's better to use tokens instead of characters for splitting our chunks.
- 4:32:35Now you might ask if you are splitting our chunk based on tokens,
- 4:32:40which tokenizer should be used? Honestly,
- 4:32:42For this task we don't necessarily need the
- 4:32:45exact tokenizer used by our eventual LLM.
- 4:32:48Here we need an estimate. We are looking for a practical estimate of chunk size.
- 4:32:53So we can use a tokenizer such as tiktoken even if our model is not an open AI model.
- 4:33:00So here inside our main function I'm going to use this text splitter
- 4:33:07and
- 4:33:08it gives me a method called from tick token encoder
- 4:33:12which uses tick token tokenizer to split our chunks
- 4:33:16and i set chunk size to 110 and chunk overlap to 10 i will explain don't
- 4:33:23worry okay so let's look at these two parameters
- 4:33:28that might have confused you the first one is chunk size
- 4:33:33we have set it to 110 which means
- 4:33:35We want each chunk to contain approximately 110 tokens.
- 4:33:40The second parameter is chunk overlap, which we have set to 10.
- 4:33:44This means that adjacent chunks can share approximately 10 tokens.
- 4:33:50It helps us preserve some context across chunk boundaries.
- 4:33:55For example, imagine a simplified case where the
- 4:33:59first chunk contains tokens 0 through 109.
- 4:34:03So the next chunk might begin around token 100, 10 tokens earlier.
- 4:34:09So it shares the final 10 tokens of the previous chunk.
- 4:34:14And the following chunk can then continue from around token 200.
- 4:34:20And the main question is that why do we need chunk overlap?
- 4:34:25Let's say an important sentence begins near the end
- 4:34:29of one chunk and continues into the next junk.
- 4:34:32and without this strategy, without overlapping,
- 4:34:35the two pieces might be retrieved separately
- 4:34:38and the relevant chunk could be incomplete.
- 4:34:42However, with overlap, some of that information
- 4:34:45is repeated in the neighboring chunk,
- 4:34:48and this increases the chance that a retrieved chunk will
- 4:34:52contain enough surrounding context to be useful to the LLM.
- 4:34:57And of course, overlap isn't free.
- 4:34:59It creates some duplicated text, which means additional storage and embedding work.
- 4:35:05And too much overlap can also lead to redundant retrieval results.
- 4:35:10So you should be careful, it depends on your task.
- 4:35:13We need to find a reasonable balance and keep in mind that a
- 4:35:18good retriever is built through experimentation and evaluation.
- 4:35:22The values we are using here are starting points, not universal rules.
- 4:35:27Depending on content of your documents and the kinds of questions users ask,
- 4:35:33you might get better results with smaller
- 4:35:36chunks, larger chunks or a different overlap.
- 4:35:39The right way to decide is to inspect the chunks,
- 4:35:42take the retrieval results and adjust these numbers based on what you observe.
- 4:35:51Now let's store what it returns in a variable called splitter.
- 4:35:56and now now that we have configured our splitter
- 4:35:59and we have it we can use the splitter it gives
- 4:36:02a method called split documents and we can pass
- 4:36:06our documents to it so i call splitter.split
- 4:36:09documents and it receives our docs and it returns our chunks
- 4:36:17Here, notice that we are passing the entire
- 4:36:19list of documents, not just a single document.
- 4:36:22Langchain will split their contents into smaller
- 4:36:26chunks and return a new list of document objects.
- 4:36:29So that's really important. These chunks retain their associated metadata.
- 4:36:34They retain their structure, which were document objects,
- 4:36:39And that's really useful because as you can see, we simply pass 25
- 4:36:43document objects and we will receive some chunks as document objects.
- 4:36:48So here I print the length of this list to see how many chunks we have.
- 4:36:55And then I print. The first chunk.
- 4:36:59okay so let's run our program okay and as you can see
- 4:37:05here we have 183 documents here and this is our
- 4:37:10first chunk as and as we can see it's a document
- 4:37:14object it has content and it has method metadata
- 4:37:18Okay, let's summarize what we have done so far. We started with a 25 page PDF file.
- 4:37:24We converted it into 25 document objects, one for each page.
- 4:37:29Then we split those documents into smaller chunks,
- 4:37:32creating a new list of document objects called chunks
- 4:37:36that are better suited for embedding and retrieval.
- 4:37:39And our pipeline now looks like this.
- 4:37:42A 25 page pdf, 25 document object,
- 4:37:47Then we have chunking, then we have a smaller document objects now.
- 4:37:51After these steps, we are ready for the next step,
- 4:37:55which is converting these chunks into embeddings.
- 4:37:59Ladies and gentlemen, this is a flight update from the cockpit.
- 4:38:03We've now been flying for about four hours and we have roughly four more hours
- 4:38:08until our final destination.
- 4:38:10We're currently exploring planet Rag, one of the
- 4:38:13most important worlds on this entire journey.
- 4:38:17So please return your seat to the upright position
- 4:38:20and give your full attention to your surroundings.
- 4:38:23What you learn on this planet is one of the most valuable skills in AI engineering.
- 4:38:36Alright, we have reached the point where we have 183 document objects.
- 4:38:41Now, we want to map the content of these documents
- 4:38:45into our high dimensional embedding space.
- 4:38:48More precisely, we wanted to convert each chunk into a vector in that space.
- 4:38:54And this isn't anything new.
- 4:38:56We have already done this in an earlier lesson using the AllMiniLM model.
- 4:39:02And we are going to use the same model here. But there is a difference here.
- 4:39:06Previously, I loaded the model using the sentence transformers library.
- 4:39:11This time, I want to use LangChains integration instead.
- 4:39:15So in order to keep our application consistent, I wanted to do that.
- 4:39:20And this is one of the frustrating things about AI
- 4:39:24engineering, especially when you are new to this field.
- 4:39:28you might notice that different developers use different
- 4:39:32libraries and approaches to accomplish exactly the same task.
- 4:39:36It makes you confused and frustrated.
- 4:39:39But honestly, there is usually a reason behind the approach that we choose.
- 4:39:44You need to consider the architecture of your application first.
- 4:39:48In our case, we are using Langchain. Okay, so the best option is this
- 4:39:54bridge that is between Langchain and HuggingFace
- 4:39:58and it makes our application more consistent.
- 4:40:02So let's do it.
- 4:40:03Let's first install the package that is for
- 4:40:06using Huggingface models in our application,
- 4:40:09in a Langchain application. So I say pipenv install langchainhuggingface.
- 4:40:15Okay, so that's the whole point of this package.
- 4:40:18It's a bridge between Huggingface and Langchain.
- 4:40:23Now let's import the embedding class. There is a class called HuggingFaceEmbeddings.
- 4:40:29And whenever you want to load an embedding model, you're going to use this class.
- 4:40:34So I say from LangChainHuggingFaces, importHugingFaceEmbeddings.
- 4:40:41This is a that we need. And then we can initialize our model using this class,
- 4:40:47so I create an object using this glass.
- 4:40:51and we need to specify some fields and the first one is our model
- 4:40:57here I say model name which is allminiLM let's copy its name so
- 4:41:03Okay, I copy its identifier and I paste it here.
- 4:41:08Okay, this tells LangChain which embedding model we want to use.
- 4:41:12And then I'm going to configure one additional
- 4:41:15option, which is normalize embeddings.
- 4:41:18Let's set it first and I will explain what is happening.
- 4:41:21So I say encode keyword arguments and then it is a dictionary
- 4:41:27and I say normalize Embeddings and I set it to true.
- 4:41:33This thing that you're seeing here is related to linear algebra,
- 4:41:37but I want to try my best to simplify it.
- 4:41:40Okay, remember that each embedding is a vector in a high dimensional space.
- 4:41:45You can think of a vector as an arrow that
- 4:41:48starts at the origin of coordination system
- 4:41:51and points toward a location in that space.
- 4:41:54And different vectors can have different lengths.
- 4:41:57And when we set normalize embeddings true,
- 4:42:01The embedding model scales each vector so that its length becomes 1.
- 4:42:07This process is called L2Normalization.
- 4:42:10And the question is why would we want to do that?
- 4:42:13Well, in our RAG application, we will eventually compare vectors
- 4:42:17to find the text chunks that are most similar to the user's query.
- 4:42:22And when vectors are normalized,
- 4:42:25calculating cosine similarity becomes more convenient.
- 4:42:29In fact, for unit length vectors, cosine
- 4:42:32similarity is equivalent to their dot product.
- 4:42:35So it would be more efficient if we normalize our vectors. Now we
- 4:42:40have the model.
- 4:42:41I store it in a variable called model. Now we can use our model.
- 4:42:47Okay, let's get back to our main function.
- 4:42:50I remove these print statements here that we use to inspect our chunks.
- 4:42:55Now we want to use our model.
- 4:42:57Langchain gives us a method called embedDocuments and we can use it
- 4:43:02to generate embeddings in one call, which is really convenient.
- 4:43:06However, there is one important detail.
- 4:43:08EmbeddingDocuments expects a list of strings,
- 4:43:11not a list a LangchainDocument objects.
- 4:43:14our chunks variable contains a document object.
- 4:43:17So first we need to extract the text from each one.
- 4:43:22And we can do this using list comprehension. So I say for each
- 4:43:27chunk in our chunks list, I'm going to access its page content.
- 4:43:33So I say
- 4:43:35chunk.
- 4:43:36page.
- 4:43:38content.
- 4:43:39Okay. That's it.
- 4:43:41And I store it in a variable called texts and
- 4:43:44this creates a list containing the text content
- 4:43:47of every chunk. Now we can pass it to the method that I mentioned.
- 4:43:52I say model embed documents and I pass this list
- 4:43:56that I just created and I store the result
- 4:44:00in a variable called vectors and that's it!
- 4:44:04The model will generate an embedding vector for each text chunk.
- 4:44:07Now, you might be wondering, what happened
- 4:44:10to the consistency we talked about earlier?
- 4:44:13Didn't we say that we want to preserve LangChain's document format?
- 4:44:18And now, what are we doing here? Yes, you are right.
- 4:44:21For now, I'm extracting the text and generating
- 4:44:24these embeddings separately to show you.
- 4:44:27But in the next session, we will use a different approach.
- 4:44:30let's print these vectors just like before i'm
- 4:44:34going to first print the length of these vectors
- 4:44:38and second we
- 4:44:40are going to print the first item of this list
- 4:44:45so I say vectors 0 let's run our program
- 4:44:50everything is in order and
- 4:44:52here we can see that we have 183 vectors the
- 4:44:56same as the number of chunks that we had
- 4:45:00and this is the embedding of the first chunk so the first
- 4:45:05trunk which is something like these tokens these words
- 4:45:09are converted to this vector that you're seeing here
- 4:45:14at the end okay so now our pipeline looks like this
- 4:45:19First we had PDF, we converted our PDF to document
- 4:45:23objects, we went through chunking, and then we converted
- 4:45:27our chunks into embedding vectors.
- 4:45:29And in the next step, we look at how to store these vectors
- 4:45:34and use them to retrieve the most relevant chunks
- 4:45:37when a user asks a question.
- 4:45:43Ladies and gentlemen were now cruising smoothly at full speed.
- 4:45:48Every like and every subscription is fuel for this rocket
- 4:45:51and helps us reach more travelers like you.
- 4:45:55Know someone who'd love this journey, share it
- 4:45:57with a friend, and bring a co-pilot on board.
- 4:46:00Have a question, drop it in the comments. The captain is listening.
- 4:46:04Now sit back, relax, and enjoy the rest of your flight.
- 4:46:10you
- 4:46:16Alright, we have now converted our entire knowledge
- 4:46:19base into vectors in a high dimensional space.
- 4:46:22The next step is to take a user's query, convert it into a
- 4:46:26vector, compare it with the vectors in our knowledge base,
- 4:46:30and retrieve the closest matches.
- 4:46:33We have already implemented this approach in our
- 4:46:35previous project, so the idea itself isn't new to you.
- 4:46:38But here's the thing, in real-world drag applications, we generally don't store
- 4:46:44all are vectors in a Python list and we don't manually
- 4:46:48compare the query against every single vector.
- 4:46:51Instead, we use a specialized database designed for this kind of work, this kind
- 4:46:56of task, and it's called a vector database. A vector
- 4:47:00database is designed to store vector embeddings
- 4:47:04and support similarity search over them.
- 4:47:07There are quite a few options available, including Pinecone, Weaviate, Chroma,
- 4:47:13and also PGVector which adds vector search capabilities to PostgreSQL
- 4:47:19and along with many other options that emerge these days.
- 4:47:23And for this project I'm going to use ChromaDB
- 4:47:26because it's really simple and it's a very good option for learning
- 4:47:32and even in real-world projects.
- 4:47:34You can develop real-world projects with Chroma and there is nothing wrong with it.
- 4:47:38But before doing that, before seeing that in code, let me
- 4:47:42explain what we need a vector database in the first place.
- 4:47:46Why can't we simply keep all our vectors in a simple
- 4:47:50Python list and search through them ourselves?
- 4:47:54There are different reasons, or it's better
- 4:47:57to say there are three main considerations.
- 4:48:01The first one is memory usage. Second, persistence.
- 4:48:05and search performance, the third one.
- 4:48:07And let's start with memory.
- 4:48:09In our previous example,
- 4:48:11I'm in project, we stored all our vectors in a python list.
- 4:48:15Those vectors occupy memory while our
- 4:48:18program is running.
- 4:48:19And when the program terminates, that in-memory list disappears.
- 4:48:23When you don't use
- 4:48:24a vector database, memory becomes a serious
- 4:48:28thing, especially when your data set grows.
- 4:48:31With a few hundred
- 4:48:32or a few thousand vectors, a simple Python list is perfectly fine.
- 4:48:37But imagine having 10 million vectors.
- 4:48:39Our all-mini-LM model produces 384 dimensional vectors.
- 4:48:44If we store 10 million vector using 32-bit floating-point numbers, the
- 4:48:50raw vector values alone would take up approximately 14 GB of memory.
- 4:48:55before accounting for metadata indexing and other stuff.
- 4:48:58So, if you have a computer with 16 GB of RAM,
- 4:49:02storing everything in memory could quickly become impractical.
- 4:49:06A vector database can persist data on disk on the other hand,
- 4:49:10which helps us manage datasets that don't fit entirely in memory.
- 4:49:14Now, let's talk about search performance. If you have only a few hundred vectors,
- 4:49:20comparing our query against every vector is usually fast enough.
- 4:49:24But what happens when we have millions or tens of millions of vectors?
- 4:49:28In that case, a brute force search compares a query against every vector.
- 4:49:33And as the data set grows, that can become expensive.
- 4:49:37Vector databases can use specialized indexing and search
- 4:49:41techniques to make retrieval much more efficient.
- 4:49:44So in those scenarios, we don't have a problem anymore.
- 4:49:48And let's talk about the techniques that these databases use.
- 4:49:53to improve performance. One widely used family of techniques is called
- 4:49:57approximate nearest neighbor or ANN.
- 4:50:01And that is, instead of calculating the exact distance to every single vector,
- 4:50:09ANN algorithms search for vectors that are likely to be among the nearest neighbors.
- 4:50:15This often provides a useful trade-off between speed and search accuracy.
- 4:50:20Now that we understood why we are using vector databases
- 4:50:24let's integrate chroma into our application since we are use Langchain
- 4:50:30we'll install its dedicated chroma integration
- 4:50:33so here first I say pipenv install langchain chroma
- 4:50:39okay once it's installed we need to import it
- 4:50:43so I say from langchainchroma import chroma
- 4:50:48Okay, as you can see it is installed and it is a
- 4:50:51class, so we need to create an object of it.
- 4:50:54So I say Chroma and we need specify some parameters.
- 4:50:59So the first thing is collection name.
- 4:51:01A collection name is a named group of records stored in Chroma.
- 4:51:05You can think of it as a way to organize the data
- 4:51:08belonging to a particular application or use case.
- 4:51:11For a project, let's call the collection name medical data, it's a descriptive name.
- 4:51:17So here I say collection name and I set it to medical data
- 4:51:23and the point is that if this collection already exists in the configured database
- 4:51:28Chroma can connect to it otherwise it will create it So it is our collection name
- 4:51:34and then we need to specify which embedding model should be used
- 4:51:38and I'm going to use this model that we constructed here
- 4:51:42So here i say embedding function
- 4:51:45and I set it to the model that we defined on the top.
- 4:51:49By passing this model to Chroma, we are telling
- 4:51:52the vectoristore that we are going to use this
- 4:51:54model and this is where things become more convenient.
- 4:51:58Previously, we extracted the text from every
- 4:52:01chunk and manually called embed documents.
- 4:52:04This way we generated embeddings ourselves.
- 4:52:07Now, Chroma can take care of these steps for us.
- 4:52:12When we add our documents, it will use the configured
- 4:52:16embedding function to generate their vectors.
- 4:52:19And later, when we submit a text query, it can use
- 4:52:22the same embedding model to embed that query.
- 4:52:25And we no longer have to take up these stuff.
- 4:52:29The next parameter is persist directory. Let me set it to period slash
- 4:52:35We want our data to remain available after our Python program terminates.
- 4:52:40So we don't want to store our data on RAM.
- 4:52:43We want a folder here is created and our data is stored in that directory.
- 4:52:49So here we'll specify a directory where Chroma can persist the database.
- 4:52:54So this tells Chroma to use a directory named Chroma inside our program.
- 4:53:00project directory and if this directory doesn't exist,
- 4:53:05Chroma can create it and the database files will be stored there,
- 4:53:09allowing us to reopen the persisted collection when we run the application again.
- 4:53:15Now there is one more important configuration we need to discuss.
- 4:53:20The distance metric.
- 4:53:22When we compare two vectors, we need a way to measure how close they are.
- 4:53:26There are different metrics and we talked about cosine similarity.
- 4:53:31And in this project, we also want to use cosine.
- 4:53:35But the question is that where should we specify this configuration?
- 4:53:40There is a field called collection metadata.
- 4:53:43It takes a dictionary, a key is HNSW space
- 4:53:49and I set this key to cosine
- 4:53:52but pay attention, this cosine you see here is cosine distance not cosine similarity
- 4:53:58we discussed earlier that cosine similarity compares the angle between two vectors
- 4:54:04now Chroma uses cosine distance it is calculated in this way
- 4:54:101 minus cosine similarity. And another common option is a squared L2 distance,
- 4:54:17which measures the sum of the squared differences
- 4:54:20between corresponding vector components.
- 4:54:23And by default, Chroma uses squared L 2 distance for its standard configuration.
- 4:54:29And there is a very important detail here.
- 4:54:32The distance metric is a collection level configuration.
- 4:54:36I mean, if the collection already exists,
- 4:54:39Passing different metadata doesn't automatically
- 4:54:42change the metric used by the existing collection.
- 4:54:45You should decide on the metric when creating a collection.
- 4:54:48So changing it later doesn't work.
- 4:54:52And it's a decision that should be made when you create a collection
- 4:54:56and I store the result inside a variable called vectorStore.
- 4:55:02It's a typical name when creating an instance from Chroma. Okay.
- 4:55:07Now let's go back to our main function.
- 4:55:10Remember these lines we had in the previous session.
- 4:55:13And I told you that it's not the approach that we will take and I remove them.
- 4:55:18And as I mentioned earlier, we are going to use Chroma and
- 4:55:22Chroma is going to take this responsibility to manage vectors,
- 4:55:27storing vectors, using the embedding model and other stuff.
- 4:55:31So I say vectorStore and it provides a method
- 4:55:34called addDocuments and I'm passing my chunks.
- 4:55:38Okay, that's it.
- 4:55:40We are passing our list of LangChain document
- 4:55:43objects and Chroma uses the configured embedding
- 4:55:46model to generate the embeddings and stores the
- 4:55:49corresponding text and metadata alongside them.
- 4:55:53So you're not only storing the text, so you're
- 4:55:57also storing the metadata in your database.
- 4:56:00And as you can see here, we no longer extract text from our document objects.
- 4:56:06We are using our document object.
- 4:56:08We are passing our document-objects and this is the consistency I was talking about.
- 4:56:13Okay, now let's verify that our documents have been stored.
- 4:56:17And for that, Chroma provides a method called get.
- 4:56:22which allows us to retrieve stored records from the collection. So let's do that.
- 4:56:27I say vector store dot get, and this gives me all the
- 4:56:32vectors and I store them in a variable called all docs.
- 4:56:38So here the result is a dictionary-like structure containing
- 4:56:43fields such as the record IDs, document texts, and metadata.
- 4:56:48Yes, every document has an ID and we will see that in detail later.
- 4:56:54And like before, let's print the number of
- 4:56:57stored records and inspect the first document.
- 4:57:00Here I say print, I want the length of the document we have here.
- 4:57:05As I told you, it's a dictionary and one key is documents. These are my documents.
- 4:57:12Now I want to print the first document.
- 4:57:14So I remove the length function and here I
- 4:57:17use bracket notation to access to the first
- 4:57:20element in this list. And now let's run our program.
- 4:57:25Fortunately, we don't have any error here and it
- 4:57:29says that we have 183 vectors in our database.
- 4:57:32And it is a text of the first document.
- 4:57:36But there is an issue here.
- 4:57:38Before we finish, I want to show this problem and
- 4:57:41we will address this problem in the next session.
- 4:57:45So let's run the program one more time.
- 4:57:47Okay, wait a little bit. You will see that this number has twice.
- 4:57:53We had 183 and now we have 366.
- 4:57:59That means that we have added another 183 records. If you run.
- 4:58:04program again the count can increase again okay the
- 4:58:08reason is that remember we are using a persistent
- 4:58:12database okay
- 4:58:14and I told you that each vector has an ID.
- 4:58:18The records from the previous run are still there
- 4:58:21and every time we call add documents
- 4:58:24and everytime we do that without supplying stable IDs
- 4:58:29Langchain generates new IDs.
- 4:58:32So the previous 183 vectors had different IDs
- 4:58:37and these new vectors that we are storing again have new IDs, okay?
- 4:58:43And Chroma, therefore, has no reason to treat those
- 4:58:47new records as updates to the existing ones.
- 4:58:51So, it can store them as separate entries,
- 4:58:54even when their content is exactly identical.
- 4:58:58So, as you can see, this creates duplicate records in
- 4:59:02our collection, and that can affect retrieval quality.
- 4:59:06when we eventually search for chunks similar
- 4:59:09to a query, we might retrieve multiple copies
- 4:59:12of the same information instead of getting a
- 4:59:15more useful selection of relevant content.
- 4:59:18So we have a problem here and we have to solve it.
- 4:59:21And in the next lesson, I will show you how to deal with this scenario.
- 4:59:35So, how do we solve the problem we discussed in the previous lesson?
- 4:59:39One approach that Chroma supports is to provide
- 4:59:43an ID for every document we want to store.
- 4:59:46In this way, when we try to add a document using
- 4:59:49an ID that already exists in the collection,
- 4:59:52Chroma won't create another record with that same ID.
- 4:59:56Instead, adding a document with an existing ID updates the existing record.
- 5:00:02So, in other words,
- 5:00:03to solve this problem, we need to generate
- 5:00:06an ID for every chunk before storing it in
- 5:00:09our vector database.
- 5:00:11So we had 183 chunks and we'll generate an id for each one.
- 5:00:17At the end, we'll have 183 IDs for each of these vectors.
- 5:00:24And this gives us a consistent way to identify
- 5:00:27our chunks and prevent duplicate records every
- 5:00:30time we run the program.
- 5:00:32And now the question is how are we going to generate these IDs?
- 5:00:36For this project, I'm going to use Python's built-in hashlib module
- 5:00:40and that means we don't need to install any additional packages.
- 5:00:44We just need to import this module. So on the top, I say import hashlib.
- 5:00:51Now let's create a function that does that for us. I mean generating IDs.
- 5:00:57So I define a new function
- 5:01:00called generateId
- 5:01:01and it receives our documents and it can generate an ID for each one.
- 5:01:06So at the end, I want to have a list of IDs. So I need to return a list.
- 5:01:13Okay. And this is a situation that using a list comprehension is a great choice.
- 5:01:20So I say for each doc in my docs, I want to do this with that doc.
- 5:01:26First, I wanted to access its content.
- 5:01:29So I said doc page content, but
- 5:01:32I don't want its actual text.
- 5:01:35I want its bytes.
- 5:01:36I want the encode method to access it's bytes because when we want to use a hash,
- 5:01:43algorithm, we don't want its actual text, we want its bytes.
- 5:01:46It's related to cryptography. So here I say encode.
- 5:01:51It's a piece of text and every string offers this method.
- 5:01:54And I give it as input to MD5 algorithm.
- 5:01:59So I say hashlib MD5 and I paste it here.
- 5:02:03So I want to hash this piece of texts. And finally, I want hexadecimal. So
- 5:02:10I say hex digest.
- 5:02:11So at the end, each id is a hexadecimal string
- 5:02:16containing characters from 0 through 9 and a through
- 5:02:20f.
- 5:02:21Now let's use this function in our existing pipeline.
- 5:02:24We need to generate an id for each chunk.
- 5:02:27So here I call this function that I just created and I pass my chunks and I get ids.
- 5:02:34So I store it in a variable called
- 5:02:37ids and now to test it let's print these ids to
- 5:02:41make sure everything is working correctly so
- 5:02:44here i print ids i only print the first item for now i'm going to comment out
- 5:02:52the lines after so
- 5:02:53i run the program
- 5:02:56now you can see that it's working correctly
- 5:02:58and here i have a hexadecimal string so it's
- 5:03:01working and now i have an id for each chunk okay now
- 5:03:06the next step instead of adding our documents
- 5:03:09without ids we'll pass both the chunks and their
- 5:03:13corresponding ids to the vector store so here i
- 5:03:17return these back and here when i'm adding documents i add ids as well so here I say
- 5:03:24IDs and I pass these IDs okay that's it now chroma receives an explicit ID for
- 5:03:32every document we want to store and if you run the ingestion process again and
- 5:03:37the same chunk produces the same ID chroma can identify the existing record
- 5:03:42and it won't store it again anymore let's run our program one more time to
- 5:03:48verify everything is working but before that i'm
- 5:03:51going to delete my database from here so i'm
- 5:03:54going to
- 5:03:54delete it i
- 5:03:56run my program one more time it says we have 183 correct and i will run the
- 5:04:03program one
- 5:04:04more time to verify this number won't increase wait again
- 5:04:10As you can see, the number of stored records remains the same.
- 5:04:14And that means we have successfully prevented repeated ingestion.
- 5:04:18And that's the purpose of this session.
- 5:04:22Oh
- 5:04:27All right, we are getting closer and closer to
- 5:04:30the implementation of a complete RAG system.
- 5:04:34At this point, we have our vector database and we can search through our vector
- 5:04:39database to find the chunks that are most relevant to a user's query.
- 5:04:44So let's implement it.
- 5:04:46First, I'm going to create a conversation loop similar
- 5:04:50to the one we implemented in our chatbot project.
- 5:04:53So let's borrow some code from that project.
- 5:04:57I want this part. Okay. That's enough. Yeah.
- 5:05:01So here I don't want these lines anymore.
- 5:05:04And also this line, it was just for verifying that everything was working correctly.
- 5:05:10And I paste my loop here and this line is irrelevant here.
- 5:05:14And yes, we get
- 5:05:16an input from the user and we have this guard.
- 5:05:20And now we have the user's input and this is going to be our search query.
- 5:05:26now we want to pass this query to our vector store to our
- 5:05:31vector database and retrieve the chunks that are most similar to it so that's a
- 5:05:35process and the question is uh which method
- 5:05:39which method should we use lang chain provides a method called
- 5:05:43similarity search we have two methods similarity search and similarity search
- 5:05:49with score so if you want the score as well we need to use
- 5:05:54similarity set with course and that's a method
- 5:05:57we are going to use in this project so
- 5:05:59I use this method, I say vectorStore similaritySearchWithScore, similaritySearchWithScore
- 5:06:06and I give it
- 5:06:08the user's query and I specify k, I set k to 3
- 5:06:14okay, this tells the vectorStore to return up to 3 matching chunks
- 5:06:19okay, each result will contain two things
- 5:06:23the retrieved document and its associated distance score
- 5:06:28So here we have two pieces of information and remember that because
- 5:06:32later we will use both pieces of information.
- 5:06:35So I store it in a variable called retrieved chunks.
- 5:06:40And there we go.
- 5:06:42And here's something that's really important when
- 5:06:45developing a RAG system, which is observability.
- 5:06:48During development, it's a good idea to inspect
- 5:06:51the documents your retriever returns.
- 5:06:54You can print them to the console. We can also save them to a file.
- 5:06:59Ideally, when you are developing a RAG system, it is better to
- 5:07:03save them somewhere so you can track the history of changes and
- 5:07:07the chunks that your retrievers returns after each change.
- 5:07:11But the point is that we shouldn't develop a RAG system
- 5:07:15without checking what's going in and what's coming out.
- 5:07:19For now, I'm just going to print the retrieve chunks.
- 5:07:22But in a real project, keep in mind that it's
- 5:07:25better to save them in a file or a database,
- 5:07:28or ideally use a tool like a length miss that is for this task.
- 5:07:33I mean, observability and keeping track of what's happening.
- 5:07:37So here we want to iterate over
- 5:07:40these retrieved chunks and print corresponding information.
- 5:07:45So here I'm going to say for chunk.
- 5:07:49in retrieved chunks or it's better to use
- 5:07:53enumerate to access to index as well so i say
- 5:07:59enumerate and in addition to chunk i also
- 5:08:03have the index so the index is the first thing we
- 5:08:07want to print so i said print and here i use a formatted string
- 5:08:12and then i put some hyphens here and then
- 5:08:16i say chunk
- 5:08:18placeholder, my index, and then another four
- 5:08:21hyphens and backslash end to go to the next line.
- 5:08:24Now remember that I told you each result in this list is a tuple.
- 5:08:30The first element is a retrieved document and the second element is the score.
- 5:08:35Let's first print the score, here I add another print
- 5:08:40statement and I say again a formatted string.
- 5:08:44I say score and placeholder and I say chunk. Okay, the second item.
- 5:08:50The second item is the score.
- 5:08:52And then let's format it to three decimal places.
- 5:08:56So I put a colon here and then I say.3f.
- 5:09:01This way we'll have three decimal place. And now let's print the metadata.
- 5:09:06The metadata associated with the document.
- 5:09:09we stored two metadata fields source and page so
- 5:09:14here i add another print statement again i use
- 5:09:19a formatted string and i say metadata say source i
- 5:09:23place a placeholder here and there i say chunk the
- 5:09:28first item in this tuple is our document and each
- 5:09:32document has a metadata and our metadata is a dictionary
- 5:09:37so
- 5:09:37one field was source and another field was page so
- 5:09:43let's duplicate this line and i say page and i
- 5:09:48change this key to page and finally let's print
- 5:09:51the actual text the first 200 characters of the
- 5:09:55chunks text so here i say print chunk again
- 5:09:58first item is our document this time instead
- 5:10:02of metadata i want to access
- 5:10:05to its page content and I just want the first
- 5:10:09200 characters and then I put a backslash
- 5:10:13in here to go to the next line and this
- 5:10:16gives us enough information to inspect the retrieved result
- 5:10:20without printing the entire content of every chunk.
- 5:10:23Now let's put everything together and run our
- 5:10:26program to see whether we have a problem or not.
- 5:10:30It asks me to enter a question. Okay, great.
- 5:10:34Here, let's try a query that should be a good test for semantic search.
- 5:10:39Here I ask, what are the symptoms of high blood pressure?
- 5:10:45and I press enter and as you can see I received three chunks
- 5:10:52here's what makes this a useful test the wording of our query doesn't necessarily
- 5:10:58have to match the wording in the pdf exactly we
- 5:11:01are asking about high blood pressure while the
- 5:11:04document uses the term hypertension and as you
- 5:11:08can see a retriever is retrieving the exact
- 5:11:12document that is related to high blood pressure
- 5:11:15and i guess it is the first disease here
- 5:11:18Yes, as you can see, it's a first page, not the first disease.
- 5:11:23Okay, often called the silent killer.
- 5:11:26And this was the exact document we were looking for.
- 5:11:30It's the most relevant chunk in our document.
- 5:11:32Now, I guess this is a good point to talk a
- 5:11:36little about how we evaluate a RAG retriever.
- 5:11:40When developing a RAG system, we need to measure
- 5:11:44how well it retrieves relevant information.
- 5:11:47and there are different evaluation metrics and frameworks for doing that
- 5:11:52but one useful starting point is to examine the highest rank result
- 5:11:57in our example, the first result is especially important
- 5:12:01because it's a chunk our retriever ranked as the closest match to the query
- 5:12:06if that first result contains the information we are looking for
- 5:12:11that's a positive sign and in formal retrieval evaluation
- 5:12:16This idea is related to metrics such as MRR, which is Mean Reciprocal Rank,
- 5:12:24which evaluates how highly the first relevant
- 5:12:28result is ranked across a set of queries.
- 5:12:32And another useful metric is HitRate, which checks whether
- 5:12:37at least one relevant result appears among the top results.
- 5:12:42For now, we are doing a simple manual inspection
- 5:12:46and we are not dealing with these metrics
- 5:12:48but they are essential in real-world projects.
- 5:12:51And in this example, our retriever has returned the chunk
- 5:12:54that was about hypertension.
- 5:12:57and this was the best chunks that we could retrieve
- 5:13:01so our retriever is working correctly
- 5:13:05Ladies and gentlemen, we're now beginning our
- 5:13:08climb as we prepare to leave planet RAG.
- 5:13:11Our exploration here is almost complete, just a little further.
- 5:13:15And believe us, the rest of this journey is even more exciting than you think.
- 5:13:21Our next stop is Planet Agent.
- 5:13:22There, you'll learn to build robots equipped
- 5:13:25with tools that can get real work done for you.
- 5:13:28Please keep your seat belts fastened. Next stop Planet Agent.
- 5:13:40Okay, in this session we are going to complete the final piece of the puzzle.
- 5:13:44I mean sending our retrieved chunks along with the user's query to an LLM.
- 5:13:48And one of the most important parts of a RAG
- 5:13:51system is the instruction we give the model.
- 5:13:54If you remember, we used the system message to define how our chatbot should behave.
- 5:14:00We have the same prompt here telling a model how should
- 5:14:04produce, how should respond in an augmented way.
- 5:14:08So here I've already prepared a system prompt to save your time.
- 5:14:12And as you can see here, the most important part of this prompt is that
- 5:14:17we are instructing the model to ground its answer in the retrieved context
- 5:14:22instead of relying on its general knowledge.
- 5:14:25So this is a fundamental principle of RAG.
- 5:14:28We want the model use the information we have provided
- 5:14:32rather than simply generating an answer based on what it already knows.
- 5:14:37or making up unsupported assumptions.
- 5:14:40So look at here, I'm saying that answer only using the sources I've provided.
- 5:14:45Don't rely on your own medical knowledge. Don't make things up. Okay.
- 5:14:50And we have also told the model what to do when the
- 5:14:54retrieved documents don't contain enough information.
- 5:14:57It should explicitly say that it doesn't have
- 5:15:00enough information to answer the question.
- 5:15:03And finally, we have asked it to cite its sources.
- 5:15:07This helps us to trace an answer back to the relevant document,
- 5:15:11making it easier to verify where the information came from.
- 5:15:15And now, finally, let's add the LLM to our project. To keep our focus on RAG,
- 5:15:21I'll use the same OpenAI Wrapper used in the previous project. However,
- 5:15:26LangChain has its own integrations for working
- 5:15:29with language models and since we are using
- 5:15:32LangChains the best practice is to use these integrations but as i told you because
- 5:15:38we want to keep our focus on RAG I'm going
- 5:15:41to use what you are familiar with so first
- 5:15:44I'm gonna add.env so on the top I say import.env and here I
- 5:15:50load environment variables I say
- 5:15:54dot env dot load dot env and since i'm going
- 5:15:58to read keys here below dot env i import os
- 5:16:03and I also import OpenAI class.
- 5:16:06So I say from OpenAI import Open AI class.
- 5:16:10Now we need to create
- 5:16:12the client. So let me borrow some code.
- 5:16:16I guess we had another name.
- 5:16:18Yeah, we called it LLM in
- 5:16:21our chatbot project.
- 5:16:23So I paste it and I store it in a variable called client. Okay, everything is right.
- 5:16:29Now, let's create a function called LLM.
- 5:16:33This function is going to receive two pieces of information, query and context.
- 5:16:39Okay.
- 5:16:40The query is a question that the user has asked and the context contains
- 5:16:45the relevant chunks that our retriever fetched from our vector database.
- 5:16:50So I define a new function, I call it LLM.
- 5:16:54It receives two information,
- 5:16:57query and context. Here we go.
- 5:17:00First, I'm going to put our system prompt here.
- 5:17:04Here's our system prompt.
- 5:17:06And as you can see here, we are injecting our context inside our system prompt.
- 5:17:13So this is where our context goes.
- 5:17:16But besides our system prompts, we also need a user message because we are going
- 5:17:21to ask something to have that. We need a list of messages.
- 5:17:25The first message is a system message. And the second message is the user message.
- 5:17:29So I create a list called messages. It has two messages.
- 5:17:34The first message is a system message. So I set role to system.
- 5:17:39And the second one is content. Okay. I set it to our system prompt.
- 5:17:45I duplicate this line. I change it to user.
- 5:17:48And my content is going to be the query that this function receives.
- 5:17:54Let's send the query to the LLM. So I go here. I search for LL M dot chat.
- 5:18:00I need this part.
- 5:18:02Copy it. I paste it here.
- 5:18:04I change it to client and I'm going to use the
- 5:18:08same model and here instead of this, I'm
- 5:18:11going to the messages list that I created on the top
- 5:18:16and I set temperature to 0.7.
- 5:18:20And here we are sending our request and we get the response.
- 5:18:25So I store it in a variable called response.
- 5:18:29Now we will return the response to the user.
- 5:18:32So to the main function actually, because we're going to call it from the main
- 5:18:36function.
- 5:18:37So I say return response.choices k0 the syntax that we had.
- 5:18:42Message. It's a message and I want its content.
- 5:18:46that's our llm function it takes the user's question and the retrieved context
- 5:18:52sends both to the model and returns the answer now let's use this function in our
- 5:18:58main function okay here inside our main function but there is an important detail
- 5:19:03here the
- 5:19:04context should be a simple text
- 5:19:08not a list of LangChain document objects here as you can see
- 5:19:12We have a list of document objects with their scores, but we need to parse them.
- 5:19:16We need to combine all the information into a simple string.
- 5:19:20So after retrieving these chunks, let me remove these lines.
- 5:19:25It was for observability in the previous session.
- 5:19:28So first I create an empty list called parts.
- 5:19:31Now I'm going to iterate over these chunks.
- 5:19:35take out the information that I need and import in a string.
- 5:19:39So we will eventually have three strings.
- 5:19:42And finally after iterating over this retrieve
- 5:19:45chunks, we can concatenate these three strings
- 5:19:48that we will have in this list.
- 5:19:50So here, like before, we are going to iterate over
- 5:19:54this retrieved chunks and I will use enumerate.
- 5:19:57to have indexes as well.
- 5:19:59So I use enumerate and we have an index and each chunk.
- 5:20:04Okay, I use
- 5:20:05pass here for now.
- 5:20:08And it is a tuple here, we can unpack it instead of using indexes.
- 5:20:13It's a better syntax.
- 5:20:14So I unpack its two elements into two variables called doc and score.
- 5:20:21Okay, for each chunk I want to extract its chunk number, page,
- 5:20:26source and the content itself.
- 5:20:28So I use a formatted string and I say source.
- 5:20:33I place a placeholder here. I say I plus one because our indexes start from zero.
- 5:20:40Then I have the page.
- 5:20:43I say page and I want to have a placeholder here as well
- 5:20:47so I say doc.metadata it is located in our metadata and
- 5:20:52then since it is a dictionary I use the get method
- 5:20:56so I save page and the second argument a question mark so if the page metadata
- 5:21:04isn't available we'll use a questionmark instead okay and after another pipe
- 5:21:11again and I say retrieval score and as you remember
- 5:21:16it's not similarity score it's cosine distance
- 5:21:20so because of that I'm saying retrieval a
- 5:21:23score not cosine score so here again braces
- 5:21:27and I said score and three decimal places here
- 5:21:31And finally we want to have our actual content.
- 5:21:34So I put backslash in here and then my actual page content.
- 5:21:39So I say doc page content
- 5:21:42Okay, that's pretty much it.
- 5:21:44The only thing left is to concatenate all these that we have.
- 5:21:48So we can use the join method that strings provide.
- 5:21:52So I call join and I pass it our list here.
- 5:21:57And as a separator, I use double backslash and
- 5:22:00a couple of hyphens and double back slash
- 5:22:03and again.
- 5:22:04Now we are ready for the final step, which is calling
- 5:22:08our LLM function with the original user query.
- 5:22:11and the context just created. So I call it, I say,
- 5:22:17LLM and I provide our user input and the contexts as well.
- 5:22:22I forgot to store it in a variable called context. Okay, it's our context.
- 5:22:28Okay, everything is fine and let's print the result.
- 5:22:32Okay, let's run our program.
- 5:22:34oh what happened oops here the problem is that we are loading environment variables
- 5:22:40after we want to use them so here was my bad
- 5:22:44okay let's run our program one more time
- 5:22:47I ask what
- 5:22:49are the symptoms of high blood pressure
- 5:22:54oh
- 5:22:55it says I don't have enough information so it's weird
- 5:23:00let's keep track what's happening so here we have the context
- 5:23:05first let's print context
- 5:23:08It's better if I investigate the problem myself and I will pause the video and I
- 5:23:14will get back to you.
- 5:23:16Okay, hello again. I quickly found the root of the problem.
- 5:23:20And the problem is that here I forgot to append this string
- 5:23:24that I generated to this list that I had on the top.
- 5:23:28So I say parts that append and that's pretty normal.
- 5:23:35Again we try to run our program. What are the symptoms of high blood pressure?
- 5:23:43okay as you can see now it answers our question
- 5:23:46this is a part of our print statement and the
- 5:23:50answer starts from here high blood pressure can
- 5:23:53cause symptoms such as blah blah blah and
- 5:23:56the interesting part is that it's mentioning our
- 5:23:59sources it says i'm saying that according to
- 5:24:03source one and if you check let me bring our pdf
- 5:24:08As you can see, this response is augmented based on this information.
- 5:24:12You can see headaches, dizziness, and you can see these keywords here.
- 5:24:17You can see dizziness, headaches. So everything is working.
- 5:24:22Our application retrieves relevant chunks from the
- 5:24:25vector database, combines them into a context,
- 5:24:28and sends that context along with the user's question to the LLM.
- 5:24:33Here we have an augmented answer as a result.
- 5:24:36successfully constructed all pieces of our RAG system.
- 5:24:41But before finishing this session, there is one important thing I want to mention.
- 5:24:46As you can see with this implementation, let me bring our main function.
- 5:24:50With this implementation every time we start the program,
- 5:24:55we repeat the entire document processing pipeline.
- 5:24:59We read the PDF, we convert its pages into documents, generate IDs,
- 5:25:04split the documents into chunks and send those chunks
- 5:25:09to the vector store
- 5:25:10even though we have added
- 5:25:12safeguard to avoid creating duplicate records we are still repeating a lot of
- 5:25:17unnecessary work and that's not a good design for a
- 5:25:20real world application a better approach would be to
- 5:25:24separate to separate document ingestion from querying for example
- 5:25:30We could create one script responsible for processing the PDF,
- 5:25:34generating chunk and IDs and these stuff,
- 5:25:37and another query for our main application. Welcome back aboard Neon Learned.
- 5:25:42Planet Rag has conquered four worlds down, six to go.
- 5:25:48Fasten your seat belts. Next stop, Planet Agent.
- 5:25:51Here, you'll build an AI agent that uses real tools,
- 5:25:55checking the weather and searching the web on its own.
- 5:25:59Your next world awaits. Enjoy the journey.
- 5:26:09Alright, from this point on, we are going to
- 5:26:11start talking about AI agents, which are
- 5:26:14a very important topic in AI engineering.
- 5:26:17I'm sure you have been hearing a lot about AI
- 5:26:20Agents and that proves that that's a burgeoning
- 5:26:23area in this field.
- 5:26:25We are going explore what AI Agentes are, what
- 5:26:28they do, what problems they can solve, and of
- 5:26:31course just like in previous sessions, we will
- 5:26:34also build practical projects to put what
- 5:26:37we have learned into practice. So let's start with a fundamental question.
- 5:26:42What exactly are AI agents and what problems do they solve?
- 5:26:47The answer is that up to this point, we have built a chatbot.
- 5:26:51At the beginning of this course, we created a simple chatbot
- 5:26:55that took an input from the user and generated a response.
- 5:26:59pretty straightforward, right?
- 5:27:01But as we continued, we discovered an important problem.
- 5:27:06We saw that for certain tasks, the model doesn't have enough information, doesn't
- 5:27:11have enough relevant information more accurately to provide a good answer.
- 5:27:16And how did we address that? We introduced RAG or Retrieval Augmented Generation.
- 5:27:23It allowed our application to retrieve relevant information from external documents
- 5:27:29and provide that information to the LLM as context
- 5:27:32so LLm could generate better answers.
- 5:27:35But now we have another problem or it's better to say we have another limitation.
- 5:27:40Our application is still primarily designed to take a query and generate a response.
- 5:27:46For many tasks, we need something more. Sometimes we need the model to take action.
- 5:27:52For example, we might want our application to retrieve real-time information, to
- 5:27:58perform calculations, to query a database, to interact with an external service.
- 5:28:04In these situations, simply generating an answer isn't enough for us.
- 5:28:09And our application needs a way to perform actions.
- 5:28:13And this is where AI agents kick in.
- 5:28:16At a high level, an AI agent uses an LLM to decide what action to take.
- 5:28:22question is how do we make this possible?
- 5:28:25We provide the LLM with a set of available tools,
- 5:28:28along with descriptions of what those tools do and what inputs they require.
- 5:28:34The model can then decide
- 5:28:36whether one of those tools would help answer the user's question or not.
- 5:28:41And when a tool is needed,
- 5:28:43the model can request that the application call
- 5:28:46it, or code executes the tool, gets the result,
- 5:28:50and makes that result available to the model.
- 5:28:53That's the basic idea behind tool use in AI agents.
- 5:28:57Now let's look at a practical example because this
- 5:29:01concept will become much clearer once we walk
- 5:29:04through the process and see an example. Let's
- 5:29:07start with the limitation we have already discussed.
- 5:29:10The knowledge cutoff. A model is trained on data available up to a certain point,
- 5:29:16and its built-in knowledge might not include recent events or real-time information.
- 5:29:21For example, suppose we ask an LLM, what's the weather like in Paris today?
- 5:29:26The model can't answer the question using its training knowledge.
- 5:29:30It needs access to current weather data.
- 5:29:33Now you might be asking, but ChatGPT can answer questions about today's weather.
- 5:29:38That's true.
- 5:29:39And the reason is that ChatGPT connects to external
- 5:29:42services and it is provided with up-to-date information.
- 5:29:46But if you simply run a language model locally
- 5:29:50without connecting it to proper tools, it won't
- 5:29:54automatically have access to real-time weather. So it cannot answer your question.
- 5:29:59It might say something like, I don't have enough information.
- 5:30:03I don't have up-to-date information.
- 5:30:05As I told you, we can address this limitation
- 5:30:08by giving our application access to a weather
- 5:30:11tool. And this brings us to an important point. which is a misconception.
- 5:30:17There is a common misconception about how tool calling works.
- 5:30:21Some people think when we provide a tool to an LLM,
- 5:30:25the model itself executes the tool. That's not what happens.
- 5:30:30A language model only generates token and when it's configured to use tools,
- 5:30:36it can generate a structured tool call request that specifies
- 5:30:41which tool it wants and what arguments to pass to it.
- 5:30:45So it is your application that is responsible for parsing that tool request
- 5:30:51and it's responsible for executing the actual function or external operation.
- 5:30:56Let's make this more concrete with an example.
- 5:31:00Imagine we have an LLM and we have a tool or in other words a function.
- 5:31:05they are the same.
- 5:31:06And this function is responsible for retrieving
- 5:31:09the current weather for a given city.
- 5:31:12So we have a getWeather function.
- 5:31:14It takes the name of a city and it returns the
- 5:31:17weather conditions in that particular city.
- 5:31:20And when we use the LLM, we provide the user's
- 5:31:24query along with a description of this function.
- 5:31:28That description tells the model that a tool called getWeather exists.
- 5:31:33It accepts a city name as its input and it
- 5:31:36returns weather information for that city.
- 5:31:39And notice the distinction here.
- 5:31:41We are providing the model with the tools name,
- 5:31:44description and its input requirements.
- 5:31:47And we are not providing the implementation at all. That's the whole point.
- 5:31:52The LLM doesn't have access to implementation and it
- 5:31:55doesn't make sense to provide it because it's useless.
- 5:31:58It's our own responsibility to call that tool.
- 5:32:01Now let's consider two different user queries.
- 5:32:04The first query can be what is the capital of France?
- 5:32:07Okay, the model already has enough information about the capital of countries.
- 5:32:13It doesn't need current weather information, doesn't need this tool.
- 5:32:16So it answers the capital of France is Paris and we have no tool call here.
- 5:32:22So here tool call is not necessary.
- 5:32:26For another query, we can say, what's the weather like in Paris today?
- 5:32:31This time, the model recognizes that it needs current weather information.
- 5:32:36And instead of immediately generating a final answer, it requests a
- 5:32:41call to the get weather tool and supplies Paris as the argument.
- 5:32:45Now, it's our application's responsibility to handle that request.
- 5:32:50So, because it's really important, let's walk through it step by step.
- 5:32:55So first, the user asks about the current weather in Paris.
- 5:32:59We send the query to the LLM and provide the definition of Get Weather Tool.
- 5:33:04Okay, the model determines that it needs current weather information in a city
- 5:33:10and returns a request to call Get Weather with Paris as the input.
- 5:33:16Now our code receives the tool called Request, identifies the requested function,
- 5:33:23passes Paris to it and executes the function.
- 5:33:27And the weather service might return something like 22 degrees Celsius.
- 5:33:32Okay, now that we have the weather resolved,
- 5:33:34we send it back to the LLM as part of the conversation history.
- 5:33:39Now the model can finally use the returned weather information
- 5:33:43to generate a response such as
- 5:33:45the current temperature in Paris is 22 degrees celsius.
- 5:33:49Is there anything else you'd like to note
- 5:33:52And that's it.
- 5:33:53We have gone from a user question to an action and then a final answer.
- 5:33:58If you look closely, you will notice that this process requires two LLM calls.
- 5:34:05The first call allows the model to decide whether it needs a tool.
- 5:34:10If so, it requests the tool call.
- 5:34:13Our application then executes that tool and obtains the result
- 5:34:18and the second LLM call provides a model with that result, with that information
- 5:34:23so it can generate a final response.
- 5:34:27And of course, it's obvious that not every request requires exactly two calls.
- 5:34:32If the model doesn't need a tool, it might answer in a single call.
- 5:34:37If it needs multiple tools, it may require several
- 5:34:40tool calls and additional LLM interactions.
- 5:34:43And this is one reason agent-based applications
- 5:34:47can be more expensive than simple chatbots.
- 5:34:51They may require multiple model calls, external
- 5:34:55API requests, and additional processing.
- 5:34:59And at this point you might be wondering if our
- 5:35:02application has to manage all these steps?
- 5:35:05Doesn't the code become complicated?
- 5:35:08Yes, exactly.
- 5:35:09We need to handle tool call requests, identify the
- 5:35:12correct function, validate and pass its arguments,
- 5:35:16execute it, return the result to the model again,
- 5:35:20and determine whether the model is ready to provide final answer or not.
- 5:35:25Besides that we also need to consider
- 5:35:28error handling, unexpected tool inputs, API failures, and other edge cases.
- 5:35:34And this is exactly where frameworks such as LangChain and LangGraph become useful.
- 5:35:40LangChains provides abstractions and integrations
- 5:35:45that make it easier to connect LLMs with tools
- 5:35:48and build tool calling applications and so on.
- 5:35:51and LangGraph helps us build more explicit workflows,
- 5:35:55agent systems, support more complex execution patterns.
- 5:36:00Oh
- 5:36:06I know the concept we discussed in the previous
- 5:36:08lesson might seem a little complicated at first,
- 5:36:11but don't worry.
- 5:36:12With this hands-on project, we will put all those concepts into practice,
- 5:36:17and you will develop a much deeper understanding of how everything works.
- 5:36:22So don't worried. We will take it step by step, and you'll master it.
- 5:36:26So first, I'm going to create a new file called agent.py, and...
- 5:36:33In this file, as usual, first we have our main function.
- 5:36:37okay and our usual entry point which is if double
- 5:36:41underscore name double underscore is equal to
- 5:36:45double underscore main double underscore in that
- 5:36:48case i want my main function to be executed okay
- 5:36:52that's it the next thing we need is an llm so far
- 5:36:55we have been using the open ai python wrapper to
- 5:36:59communicate with our language model now in this
- 5:37:02project we want to switch to langchains wrapper
- 5:37:06So we are going to use Langchain's wrapper for Groq.
- 5:37:10And this gives us the opportunity to learn another widely used integration.
- 5:37:14Besides, it keeps our implementation more consistent
- 5:37:17because we are currently using Langchain.
- 5:37:20So to get started, we need to install its package, which is Langchain Groq
- 5:37:26So here, pipenv install Langchain Groq.
- 5:37:31Now let's
- 5:37:32load our environment variables just as we did in previous project.
- 5:37:36So I import.env first and then I use it to load my environment variables.
- 5:37:43Next, we need to import ChatGroq from the library that is being installed.
- 5:37:49Okay, now we can import it.
- 5:37:51So here I say from LangChain, from LangChainGroq import ChatGroq.
- 5:37:57Okay, that's the class we need.
- 5:37:59One thing to keep in mind is that when we switch wrappers,
- 5:38:02the parameter names and the way responses are returned can change.
- 5:38:07The underlying concepts remain the same,
- 5:38:09but the interface isn't necessarily identical.
- 5:38:13For example, we can still configure parameters such as temperature
- 5:38:17and the maximum number of generated tokens,
- 5:38:20although the exact parameter names depend on the
- 5:38:23integration and the model provider and now let's
- 5:38:26create an instance of this class so here i say
- 5:38:30chat Groq and i store it in a variable called
- 5:38:33llm it's going to be our ll okay first we need to specify
- 5:38:38which model we are going to use so i set model to
- 5:38:42let's see here you know in this project i copied the model name from here which was
- 5:38:48Open.
- 5:38:50yeah open AI GPT OSS okay I want to use this
- 5:38:53model okay this is our model now let's set the
- 5:38:57temperature okay I set the temperature to 0.5
- 5:39:01okay and that's our basic LLM configuration
- 5:39:05You might remember that when we used the OpenAI
- 5:39:08Wrapper, we had to specify a custom base URL.
- 5:39:11That's because we were using the Open AI client to
- 5:39:14communicate with Groq's OpenAI-Compatible API.
- 5:39:17However, here we are using the dedicated LangChain integration for Groq.
- 5:39:22So we don't need to configure that base URL ourselves and
- 5:39:27it is already configured by internal implementation.
- 5:39:31However, make sure that the API key is available in your.env file.
- 5:39:36You need to have this API key here with exactly this name.
- 5:39:40And this integration can read this environment variable automatically.
- 5:39:44So we don't have to pass the API explicitly in our Python code.
- 5:39:48Now that our LLM is configured, let's see how we can use it.
- 5:39:52With the OpenAI client, we previously used a
- 5:39:56call like client.chat completions create.
- 5:40:00Now with LangChain we can use a simpler interface
- 5:40:03we only need to call the invoke method
- 5:40:06so here in our main function we can call this method
- 5:40:11and pass my query for example i say what is the capital of France okay
- 5:40:17and i get the response and then i print it
- 5:40:22Okay, this is a message, but I only want its content, not its role.
- 5:40:26Now let's run our code.
- 5:40:29okay as you can see it works correctly you see
- 5:40:32how straightforward this is we call llm.invoke
- 5:40:35passing our question and receive a response that's
- 5:40:38it the return the value you see here is just
- 5:40:41a langchain message object as you see langchain gives
- 5:40:45us a relatively concise interface and we don't have
- 5:40:49to manually construct an open ai chat completions
- 5:40:53request like before okay before we move on
- 5:40:56let's take a closer look at the response object
- 5:40:59this response we have here like open ai response
- 5:41:02contains a lot of metadata things like token usage
- 5:41:06and other kinds of information for example
- 5:41:10here i say response metadata
- 5:41:13Okay, as you can see here, it can contain information such as
- 5:41:17the model identifier, the name of the model we are using, and
- 5:41:21a lot of other information that might be useful for you.
- 5:41:25As another example, let's print usage metadata.
- 5:41:30So if I print it out, you see that it says our
- 5:41:34input tokens was 78, output tokens, total tokens,
- 5:41:38and it gives you information about your usage.
- 5:41:42And finally, let's look at another thing that is pretty related to our topic.
- 5:41:48I mean, AI agents. So here I print
- 5:41:51tool calls and if I printed
- 5:41:54you see that it's an empty list because our LLM doesn't want us to execute any tool
- 5:42:00and we didn't give it any tool as well.
- 5:42:04Oh
- 5:42:11If you remember, when we used OpenAI Wrapper, each
- 5:42:14message had two main fields, role and content.
- 5:42:18The role could be user, assistant or system, depending on the type of message.
- 5:42:23We have the same concept here. In LangChain, we have three classes.
- 5:42:27These classes are human message, AI message and system message.
- 5:42:31And from their name, you can probably guess what each one corresponds to.
- 5:42:35Human message is equivalent to user role.
- 5:42:38AI message corresponds to Assistant role and System message corresponds
- 5:42:43to the system role and to use these three classes you can import them from
- 5:42:48LangChain course. So on the top I say from
- 5:42:52LangChain, core, messages, okay, import,
- 5:42:56system message, human message and AI message.
- 5:43:00And we have also tool message here. It is related to when we have a tool call.
- 5:43:05So whenever we need one of them, we can use this class to
- 5:43:10create that type of message that we want.
- 5:43:13So simply we create an instance, for example, for when we want to have
- 5:43:18have a human message or a message that its role is user.
- 5:43:21I use human message and I pass the content that I want to its constructor.
- 5:43:27So here, for example, I say what is the capital of France.
- 5:43:33and this represents the same kind of message we
- 5:43:36had with open ai format something like this that
- 5:43:39we had before we had role the role is user and
- 5:43:43we have content which is this message so
- 5:43:47i'm trying to say you these two are the exact
- 5:43:50same concepts with only different syntaxes okay
- 5:43:53and this was about these three classes okay
- 5:43:59now to build an AI agent we will start by creating a
- 5:44:02regular chatbot just like the one we built at the
- 5:44:05beginning of this course in our first project then
- 5:44:08we will extend it by equipping it with a set of
- 5:44:12tools okay that's a basic idea we start with a chatbot
- 5:44:15that can communicate with an LLM and then we
- 5:44:18will give it the ability to request actions through
- 5:44:22the tools that we provide so let's implement it in
- 5:44:25And I want it to be object oriented. So first I create a class called Agent.
- 5:44:31And this class has a constructor.
- 5:44:36and first we will initialize our LLM using the
- 5:44:39wrapper that I showed you in the previous
- 5:44:42session so here I'm going to take it cut it and paste
- 5:44:46it here
- 5:44:48okay and i add self keyword at the beginning of
- 5:44:51it then we want to have our conversation history
- 5:44:55so i say self messages it is a list and as you
- 5:44:58remember the first message should be a system message
- 5:45:02so i use the system message class here and i
- 5:45:06pass something like this give short answers
- 5:45:10next we want have a method a method called chat
- 5:45:13loop and it is basically our conversation loop
- 5:45:17So here I add this method. I say def chat loop.
- 5:45:22our conversation loop is going to sit here so
- 5:45:26let me borrow some code from maybe here
- 5:45:29okay i copy these lines and i paste them here
- 5:45:36after receiving a message from our user we are
- 5:45:38supposed to add it to our conversation history
- 5:45:41so this is my conversation history and i'm going to append
- 5:45:46this message i mean what the user has entered to it
- 5:45:50so i say human message
- 5:45:52so you see that instead of using the previous syntax for creating messages now
- 5:45:57we have this syntax which is nicer in my opinion and i put users input here then
- 5:46:04We are going to have an LLM call and I told
- 5:46:07you that you should use the invoke method.
- 5:46:10So here I say self.llm invoke and
- 5:46:14here I'll pass our conversation history, which
- 5:46:19is our messages list and
- 5:46:22I'll get a response and I store it in a variable called response.
- 5:46:27And then I'll add the response to my conversation history.
- 5:46:30So I say self.messages or conversation history append this message.
- 5:46:36And in the previous lesson, I showed you that this is basically an AI message. Okay.
- 5:46:42And now let me print its content.
- 5:46:46Say response.content.
- 5:46:48Okay, let me make them more organized.
- 5:46:51Okay, that's nicer and I don't need this line anymore.
- 5:46:55Okay, now we have to create an instance of it. So, store it in a variable.
- 5:47:02and then I'll call its ChatLoop method.
- 5:47:05Now let's run our program and test it with a few questions.
- 5:47:10So here I say, let's
- 5:47:12start with the greeting.
- 5:47:14And I asked what is the capital of France?
- 5:47:19Who is its president?
- 5:47:22to test whether it has memory or not. So it has a memory. Okay, great.
- 5:47:26We have a chatbot that can answer questions
- 5:47:29and maintain its conversation history across
- 5:47:31multiple turns. At this point, we still have a regular chatbot.
- 5:47:35We haven't given it any tool yet. And that's what we will do next.
- 5:47:39In the next session, we will equip it with the tools that we wanted to have.
- 5:47:44And yes, in the next section, I will show you that.
- 5:47:48Ladies and gentlemen, this is a message from your captain.
- 5:47:52You may have noticed a few meteors passing close to our ship.
- 5:47:56The fear of forgetting what you've learned. The fear never becoming an AI engineer.
- 5:48:01The feeling that all your effort is for nothing.
- 5:48:04If they've scared you, that's completely normal.
- 5:48:07Everyone who has traveled this path to becoming an AI engineer has seen them too.
- 5:48:13The only real danger out here is turning off your engines.
- 5:48:17So just keep going at full speed. These meteors can't catch you.
- 5:48:31Okay, from this point on, we are going to start implementing our agent.
- 5:48:35But before we do that, I'd like to spend a little time explaining the difference
- 5:48:40between LangChain and LangGraph because it
- 5:48:43is relevant to what we are about to build.
- 5:48:47You've already seen a little bit of LangChains, but you might not be familiar
- 5:48:51with LangGraph yet. If you search online, you might get the impression that
- 5:48:57whenever you want to build an AI agent, you should use LangGraph
- 5:49:01but that is not necessarily true
- 5:49:04just because you are building an agent doesn't
- 5:49:06mean you automatically need LangGraph
- 5:49:08why? because LangGraph can introduce additional complexity
- 5:49:13and if your task doesn't require that level of control, that level complexity
- 5:49:20adding complexity might not make sense
- 5:49:23So if all you need is an LLM call with a few tools,
- 5:49:27LangChain already provides a convenient way to do that. You don't need LangGraph.
- 5:49:32So in fact, LangChains has a feature called Create
- 5:49:36Agent and this Create Agent uses LangGraph
- 5:49:39behind the scenes.
- 5:49:41So it manages all that complexity.
- 5:49:43You can use this higher level interface
- 5:49:46without having to work directly with LangGraph
- 5:49:49and it makes your application less complex.
- 5:49:53This way you can build an agent without explicitly
- 5:49:56importing LangGraph or managing its lower level
- 5:49:59details yourself which is really good and convenient.
- 5:50:02So let me summarize. LangChains create agent gives you a convenient
- 5:50:07higher level interface for building tool using agents. While LangGraph
- 5:50:12gives you more direct control over the agent's control flow and execution.
- 5:50:16So in this project, we will use create agent for creating our agent.
- 5:50:20And later in another project, I will show you how to use LangGraph.
- 5:50:25So let's get started. First, we need to import create agent from LangChainAgent.
- 5:50:31So here on the top, I say from LangChainAgent.
- 5:50:36Import, create agent
- 5:50:39Okay, now the question is what exactly are we going to build?
- 5:50:43We are going to create an agent with a weather tool.
- 5:50:46I don't want to make it over complicated.
- 5:50:48I want to focus on the concept of agent itself.
- 5:50:51So I don't want to really connect my application to a weather service.
- 5:50:56So I will create a simple function that accepts a city name
- 5:51:00and it always returns the same weather information.
- 5:51:03It always says the weather is sunny and the temperature is 22 degrees Celsius.
- 5:51:09And later we can replace this function with a real weather API.
- 5:51:13And keep in mind that whenever I say tool, I'm referring to a simple function.
- 5:51:18Okay, so here I'm going to define this tool,
- 5:51:21which is a function and I call it get weather. It takes a city name.
- 5:51:26It is a string and then it returns a dictionary. Here I say dict.
- 5:51:32Oh, S, T, R.
- 5:51:34and then we'll specify a description and here I use triple quotation
- 5:51:39and I say get the current weather for a city
- 5:51:44use imperative sentences when you are documenting like this
- 5:51:48and then I specify the arguments
- 5:51:51the only argument that we have is city which is the name of the city
- 5:51:57then I'll return a dictionary containing a city
- 5:52:01which is the city that the user has asked for and another key called weather
- 5:52:07and I set it to sunny 22 degrees Celsius
- 5:52:11the most important detail here is the description you are seeing here
- 5:52:15in Python it is called Dockstring and it is a standard way to document a function
- 5:52:21so other users, other developers can understand
- 5:52:25what it does and how they should use it.
- 5:52:28But when we are working with AI agents, that
- 5:52:32description has another important purpose.
- 5:52:36Langchain can use this description along with the function name and its input schema
- 5:52:42to provide the LLM with the information it needs.
- 5:52:46and with this information, LLM can figure out when it should call a tool.
- 5:52:52So you should keep in mind that the model uses this information
- 5:52:56and it influences its decision,
- 5:52:59its decision whether the tool is relevant to the user's request or not.
- 5:53:03That's really important and it's not just for documenting your functions necessary.
- 5:53:09For example, if the user asks about the weather in Paris,
- 5:53:12This description helps the model understand that get weather is needed here.
- 5:53:17And if the description is vague or misleading, the
- 5:53:21model might fail to call the appropriate tool.
- 5:53:24So you should be really careful.
- 5:53:26And before we continue, there is another important point I want to mention.
- 5:53:31When building agents, the Model you choose matters. A small model
- 5:53:35might work well for simple questions and answers,
- 5:53:38but they might struggle with more complex tool calling tasks, something like this.
- 5:53:43So, in some cases, it might fail to recognize when a tool is needed,
- 5:53:48or it might generate invalid arguments or request a tool that doesn't exist.
- 5:53:53For example, a model with 8 billion parameters might work for simple tasks,
- 5:53:59but it isn't guaranteed to follow every instruction, in these scenarios.
- 5:54:04I mean, when you have some tools and you want to have an agentic behavior.
- 5:54:09So use a model that is strong enough.
- 5:54:12enough.
- 5:54:13okay let's get back to our agent class previously
- 5:54:16we initialized our message history with a system
- 5:54:20message and it simply instructed the model to
- 5:54:22give the short answers now we need to update it
- 5:54:25and tell the model use appropriate tools whenever needed so
- 5:54:30i'm going to say use the provided tools whenever need it okay that's it
- 5:54:38and this instruction gives the agent some basic
- 5:54:41behavioral guidance, exactly what we need.
- 5:54:44Notice that we are not telling that it always should use a tool.
- 5:54:48Maybe user ask something that doesn't need a tool, so that's really important.
- 5:54:52Don't assume that when we provide tools, model
- 5:54:56always needs to use one of those tools.
- 5:54:59So that's very important.
- 5:55:00It might request one of the tools, several of those tool or
- 5:55:04maybe doesn't need to request any of those tools.
- 5:55:07Okay, now this is a point where we need to construct our agent in this agent class.
- 5:55:14And to do that, I told you that we want to use createAgent function.
- 5:55:18So here in our constructor, I'm going to add another attribute
- 5:55:24called self.agent using this createAgen that we just imported.
- 5:55:29First, we need to specify the model that we are going to use.
- 5:55:33So as a model, I'm going to this model. So here I say self.llm
- 5:55:39And then we want to provide the tools that we have.
- 5:55:43So here I'm going to set this tools argument and
- 5:55:48I set it to a list and I add my tools here.
- 5:55:51Currently I only have one tool. So here, I say get weather.
- 5:55:56That's all we need to do at first.
- 5:55:59LangChain takes care of much of the orchestration. Yeah, it does everything we need.
- 5:56:04And let's use it here when we are calling our LLM here.
- 5:56:09Instead of using my LL, I'm going to use this agent that I created.
- 5:56:13And this agent has also this invoke method, so I don't have to change it.
- 5:56:18But as an input, it needs a dictionary.
- 5:56:21A dictionary that has a key called messages.
- 5:56:25okay and this way I provide my messages list for it and this
- 5:56:30is an important difference that you need to understand
- 5:56:34and another important difference is response is no longer an
- 5:56:38AI message it's a dictionary it's dictionary with this format
- 5:56:42and the messages entry contains a conversation history but here when we want to
- 5:56:47append that message to our conversation history we only need its final message
- 5:56:53So here I say response, access to its messages
- 5:56:57value, and then I only want the last item.
- 5:57:01Okay. And the same for printing the response.
- 5:57:04Here I say, I want messages entry, and
- 5:57:08then I want the, last message and I only want its content.
- 5:57:14And to clearly figure out that this tool is actually used,
- 5:57:19let me put a print statement here and I say called
- 5:57:24This way I can realize that this line in my own code is executed.
- 5:57:28So let's run our program.
- 5:57:31First, I enter a simple greeting.
- 5:57:34Okay, we don't have tool calling here.
- 5:57:36And then I ask what is the capital of France?
- 5:57:42You see that again, we do not have toolcalling and the exciting point is here.
- 5:57:47When I ask something about the weather, I ask, what is weather like in
- 5:57:52Paris. Paris today.
- 5:57:56Okay?
- 5:57:57And it says I don't have real time data.
- 5:58:02Probably we have something
- 5:58:04wrong here.
- 5:58:06Okay, it's saying that- Oh, I'm sorry.
- 5:58:09I put the print statement before my tag string.
- 5:58:14Okay.
- 5:58:16let's run it one more time and here I say hello
- 5:58:20and say what is the weather like in Paris
- 5:58:24today.
- 5:58:26okay this time you're seeing that we have
- 5:58:30we have tool called and this line that you have you see here has been executed
- 5:58:35and it says in paris today it's sunny with a temperature of about 22 degrees celsius
- 5:58:42and as you can see this response is based on this
- 5:58:45tool the information that this tool provided
- 5:58:47okay let's terminate it and great we have now transformed our simple chatbot into a
- 5:58:55tool using agent with just a few lines of code which is really fantastic and it's
- 5:59:00the power of this interface create agent and here before finishing the session
- 5:59:06let's take a moment to appreciate what LangChain is doing for us here
- 5:59:10it's really important to understand what's going on behind the scenes
- 5:59:15tool calling involves more than just asking an LLM to use a function
- 5:59:19in a manual implementation
- 5:59:22we need to check whether the model has requested
- 5:59:25the tool, identify which tool it wants to call,
- 5:59:27extract and validate the arguments, execute the
- 5:59:31function and send the result back to the model.
- 5:59:34Then we need check whether model is ready to produce
- 5:59:38its final answer or not, or it still wants to
- 5:59:41call another tool. And there are several edge cases here to consider. For example,
- 5:59:47the model might not call any tool at all. It might request multiple tools.
- 5:59:52Or it might request one tool, receive the
- 5:59:56result and then decide it needs another tool.
- 6:00:00So it has complexity and it is called a tool calling loop and it
- 6:00:05is a loop that we need to implement ourselves.
- 6:00:08Besides, we also need to think about invalid arguments.
- 6:00:12Maybe our LLM provides invalid arguments or maybe we
- 6:00:16have execution errors and some other unexpected cases.
- 6:00:20So here you can say that Create Agent has managed much of this complexity.
- 6:00:35Okay, in the last session, we use Create Agent to implement our agent.
- 6:00:39But Create Agent isn't the only way to do that.
- 6:00:42It isn't the the only to implement tool calling in Langchain.
- 6:00:45There are three broad approaches worth understanding.
- 6:00:49The first approach is binding tools directly to the model.
- 6:00:53And this is a lower level approach. First, we import a decorator.
- 6:00:58This way I say from Langchain dot tools import tool.
- 6:01:04this is a decorator and we put this decorator before the function we are defining.
- 6:01:10So here I say
- 6:01:12tool and this turns this function into a tool okay
- 6:01:17this way i mean this decorator helps lang chain
- 6:01:21turns the function's name description and turns the function's name description and input schema into a tool definition
- 6:01:26Next, we bind the tool to our model.
- 6:01:29So for example, inside our constructor, we could write something like this.
- 6:01:34We could say self.llm.
- 6:01:38This LLM object has a method called bindTools.
- 6:01:43So here I say bindTools and then I provide my tools inside a list.
- 6:01:49So I say getWeather.
- 6:01:52and to do this we should put this decorator here to use this approach i mean to use
- 6:01:58this bind tools method
- 6:02:01Okay, now the model knows about the available
- 6:02:04tool definitions and can request a tool call
- 6:02:07when appropriate.
- 6:02:08For example, we can store it in an attribute called LLM with tools.
- 6:02:14But there is an important distinction.
- 6:02:16Binding a tool to a model doesn't automatically
- 6:02:19execute the tool or manage the entire agent
- 6:02:22loop. create agent approach that could do that.
- 6:02:25In this approach, we need to invoke the model, inspect the response,
- 6:02:29determine whether it contains tool calls, execute the requested functions,
- 6:02:35and send the results back to the model.
- 6:02:37So in this approach there is a lot to do and we have the full responsibility.
- 6:02:42And as I told you, it is a lower level approach. So it was the first approach.
- 6:02:48And the second approach is using create agent that we did in the previous lesson.
- 6:02:54We create an agent, we provide tools, and then we
- 6:02:58simply invoke the agent with the conversation state.
- 6:03:02And LangChain manages the tool calling loop for us entirely.
- 6:03:07And the third approach is, using LangGraph.
- 6:03:10This becomes useful when we need more explicit control over the workflow.
- 6:03:16things like state transitions, branching, persistence,
- 6:03:20or other complex execution requirements.
- 6:03:23So we want to use it when we actually need it.
- 6:03:27For example, we might want an agent to follow a particular sequence of steps,
- 6:03:33or pause for human approval before performing an action,
- 6:03:37or route different types of requests through different processing paths.
- 6:03:42However, that additional control
- 6:03:45also comes with additional concepts and
- 6:03:47implementation complexity so the right choice
- 6:03:51depends on our requirements the requirements of our
- 6:03:54projects or application not simply on whether you
- 6:03:57are building an agent or not so these are the three
- 6:04:01main approaches but before finishing this lesson
- 6:04:04i should tell you that you might encounter some
- 6:04:08other methods in other tutorials for example you
- 6:04:12You might see something like this.
- 6:04:15Create tool calling agent
- 6:04:18or something like this agent executor these were commonly used in earlier lang chain
- 6:04:24implementations but in newer versions of lang
- 6:04:27chain create agent is recommended approach so
- 6:04:30these are kind of outdated okay that's it
- 6:04:43Okay, in this session, we are going to do something really exciting
- 6:04:48I'm sure you have seen this feature many times when working with LLMs
- 6:04:52Sometimes when you ask ChatChipity a question, you will
- 6:04:55see a message indicating that it's searching the web
- 6:04:59It's searching for the web to find an answer for you
- 6:05:01And you might notice this, especially when you asked about specific
- 6:05:06people, recent events or information that may have changed recently
- 6:05:12Well, it relates to some of the limitations we have already discussed
- 6:05:16including hallucinations and knowledge cut-offs
- 6:05:20and LLM's knowledge comes from the data it was trained on
- 6:05:24and that knowledge doesn't automatically
- 6:05:26include everything that's happening right now
- 6:05:28for example, if you ask whether a particular person has passed away
- 6:05:33that person might have died today so the model doesn't know about that
- 6:05:38and in situations like this
- 6:05:40Searching the web can help the model access more up-to-date information.
- 6:05:45And there is another scenario.
- 6:05:47Imagine you are asking about a person, a company, or a
- 6:05:51website that the model simply doesn't know much about.
- 6:05:54Maybe that information wasn't included in its training data at all.
- 6:05:59So, the model
- 6:06:00might not have a reliable answer on its own, but it
- 6:06:04could find relevant information by searching the web.
- 6:06:07to see a tangible example I run the program
- 6:06:11and I ask a question about myself I say
- 6:06:15Who is Mohammad SeyedAghaei?
- 6:06:21okay this is not a good answer as you can see this
- 6:06:24information is not accurate and it's not relevant
- 6:06:27to me let's ask about my academy the academy that
- 6:06:31i run so i say what is the neon learn website
- 6:06:38Again, the model doesn't necessarily have enough
- 6:06:41relevant information to give me a good answer.
- 6:06:44So, what we are going to do in this session is to connect our application to a tool.
- 6:06:50A tool that can search the net to find the relevant information.
- 6:06:54And the question is, how are we going to implement that? We are going use Tavily.
- 6:07:01It's a service that makes web search easy to integrate into AI applications.
- 6:07:05and Tavily provides a search API designed for AI-powered applications
- 6:07:10and it offers a free plan that you can use to experiment with it
- 6:07:14so you can create an account and you can generate an API key
- 6:07:18and you copy and have this API key with you like me
- 6:07:25keep it in your environment variables because we want to use it
- 6:07:30Okay, here I store it in a variable called tavily API key.
- 6:07:36okay pay attention that you need to use this name
- 6:07:40because the integration will look for this name
- 6:07:43now that we have our api key configured let's install
- 6:07:47its package its integration for lang chain so
- 6:07:50here i type q to quit and then i say pipenv install lang chain tavily
- 6:08:00okay once the package is installed we can
- 6:08:02import the search tool into our python code
- 6:08:05so
- 6:08:05on the top i say from lang chain tavily import tavily search
- 6:08:12Okay, and now creating the tool is very straightforward.
- 6:08:15Below our existing tool, which was for getting weather.
- 6:08:19I add this following line to add this new search tool. I say tavily search.
- 6:08:24It is a class and I only specify max results.
- 6:08:28And I set it to one and I store it in a variable called search tool
- 6:08:35Okay, that's it. We now have a web search tool that we can provide to our agent.
- 6:08:40And you can think of this tool as being similar to using a search engine.
- 6:08:45When you search for something on Google, for example, you search
- 6:08:49for an animal, you typically get a list of results.
- 6:08:53And these results come from different web pages.
- 6:08:57The max results argument is exactly for that.
- 6:08:59This parameter controls how many search results Tavily returns.
- 6:09:03and here we have set it to one which means we are asking for just one result
- 6:09:08now let's connect this tool to the agent in other
- 6:09:11words let's provide this tool for our agent
- 6:09:14so here in addition to this get weather tool i can add this
- 6:09:18search tool okay now our agent is equipped with two tools
- 6:09:23one tool for getting weather and another tool for searching the internet
- 6:09:28Now with that in place, let's run our program.
- 6:09:32First I'll ask the same question we tried earlier to see the difference.
- 6:09:35I say who is Mohammad SeyedAghaei?
- 6:09:43This information is not correct. This information
- 6:09:46is not about me, but this shows that it I mean our agent searched the internet
- 6:09:53used that tool to search the internet and found the information
- 6:09:57and now let's search with different keywords using
- 6:10:01Persian alphabet. So I say who is Mohammad SeyedAghaei
- 6:10:08I get the same result so nothing changed but
- 6:10:11let's ask about my academy what is new there
- 6:10:16offering okay this time it is returning information that is quite relevant
- 6:10:20because my website appears in the search results when it searches and it has
- 6:10:26gathered this information through its search and through my website
- 6:10:32okay so we successfully implemented what we wanted and I can tell you there is an
- 6:10:38interesting connection between what we have just built and the Rack system we
- 6:10:43implemented in the previous session
- 6:10:45In our Rack project, we extracted information from PDF documents
- 6:10:49and this time we are providing information via the internet.
- 6:10:54So we can say the overall idea is similar, but
- 6:10:57the source of the information is different.
- 6:11:00In our previous project, we retrieved information from our own document collection.
- 6:11:05And in this project, Tavily searches the web and provides search
- 6:11:09results that the agent can use to answer the user's question.
- 6:11:13So in both cases, we are giving the model additional information.
- 6:11:26Alright, there is a technology I know some of you have already heard about.
- 6:11:31And if you haven't, I'm sure you will hear about it a lot soon.
- 6:11:35It is called MCP, which stands for Model Context Protocol.
- 6:11:40So what is MCP and what problem is it designed to solve?
- 6:11:44Let's start with something we have already done,
- 6:11:47with something that we are already familiar with.
- 6:11:51We have built tools and made them available to our LLM.
- 6:11:55When the model needs to perform an action, it can
- 6:11:58request that our application call the appropriate tool
- 6:12:02Our application then executes the tool and sends the result back to the model
- 6:12:08But imagine that you move to a completely different project
- 6:12:12Maybe you are even using a different programming language
- 6:12:15And you need exactly the same tools Without a standardized way to expose those tools
- 6:12:22you might have to implement the same integrations all over again.
- 6:12:27and you will need to figure out how to describe the tools
- 6:12:31how to communicate with them and how to handle their inputs and outputs
- 6:12:37That's where MCP comes in.
- 6:12:39MCP is an open protocol introduced by Anthropic in 2024.
- 6:12:45And its goal is to standardize how AI applications connect
- 6:12:49to an external tool, to external tools in general.
- 6:12:53Not only external tools but also data sources and other capabilities.
- 6:12:58Instead of building a completely different integration for every application,
- 6:13:03we can expose capabilities through an MCP server
- 6:13:07and let compatible applications connect to them through this protocol.
- 6:13:13And this makes integrations more reusable
- 6:13:16and helps different AI applications work with the same services.
- 6:13:21Okay, for example, imagine that Gmail wants to make
- 6:13:24its capabilities available to AI applications.
- 6:13:28It could expose operations such as searching emails, reading messages,
- 6:13:33and creating drafts, and all these things through an MCP server.
- 6:13:37And an AI application that supports this protocol
- 6:13:41could then connect to those capabilities
- 6:13:44without having to implement a completely custom integration for every operation.
- 6:13:49And to understand how MCP works, let's start with two important components.
- 6:13:55MCP server and the MCP client. First, MCP Server.
- 6:14:00An MCP-server is a program that exposes capabilities
- 6:14:03to an AI applications through this protocol.
- 6:14:07For example, it can make tools available and describe what each tool does,
- 6:14:12what inputs it accepts, and what kind of output it returns.
- 6:14:17And this should sound familiar because we have
- 6:14:20already done something similar ourselves.
- 6:14:23Remember when we created our Get Weather tool.
- 6:14:26We defined a function, gave it a name,
- 6:14:28described its purpose, and specified the input it needed.
- 6:14:33MCP server provides a standardized way to
- 6:14:36expose tools and their descriptions so that
- 6:14:39compatible clients can discover and use them.
- 6:14:43Now let's elaborate on MCP client.
- 6:14:46The MCP Client is a component that connects an AI application to an MCP server
- 6:14:51and communicate with it using the MCP protocol.
- 6:14:55Imagine you are building an AI Application.
- 6:14:57Your application connects to an MCP server and discovers which tools are available.
- 6:15:03The client receives their descriptions and makes
- 6:15:06that information available to the agent or LLM.
- 6:15:10And when the model decides that it needs a particular tool, the application
- 6:15:16handles the request and invokes that tool through the MCP client.
- 6:15:20The result is then returned to the model so it
- 6:15:23can continue processing the user's request.
- 6:15:26So the important idea is that as the MCP client
- 6:15:29handles communication with the MCPServer,
- 6:15:32while the LLM decides which tool it needs based on the
- 6:15:36information and tool descriptions provided to it.
- 6:15:40And the model itself doesn't need to implement the
- 6:15:42MCP protocol or directly connect to the server.
- 6:15:45Okay?
- 6:15:46I'm sure many of you have already connected your Gmail account to an AI assistant
- 6:15:51or you have seen AI agents that can interact with email services.
- 6:15:54To implement this concept, MCP is the primary way.
- 6:15:58And let's walk through a simplified example to better
- 6:16:02understand how the different components work together.
- 6:16:06Imagine that we have an AI application and we want to work with Gmail.
- 6:16:11First, our AI application connects to a Gmail MCP server
- 6:16:15and the server advertises the capabilities it supports.
- 6:16:19For the sake of this example, let's imagine it exposes 10 different tools.
- 6:16:24These might include a tool for reading emails, a tool, for searching, a tools for
- 6:16:29reading spam messages, a tool for creating an email
- 6:16:32draft, and several other tools that might appear.
- 6:16:36The MCP client discovers those tools and makes
- 6:16:39their descriptions available to the agent.
- 6:16:41Now, let's say the user sends a message like this.
- 6:16:45Write an email to Alex asking to schedule a meeting for next week.
- 6:16:50Okay, and the LLM receives the user's request along with the available tools.
- 6:16:55and based on that information, it determines that it needs to create an email draft.
- 6:17:00It then generates a tool call requesting the appropriate
- 6:17:04operation, such as write draft with the necessary arguments.
- 6:17:08And our AI application receives that tool call request and
- 6:17:12invokes the corresponding tool through the MCP client.
- 6:17:16And the MCPServer responses the request and returns the result,
- 6:17:20such as confirmation that the draft was created successfully.
- 6:17:24The application then provides that result to the LLM.
- 6:17:27Finally, the LLM can respond to the user like this. I've created the email draft.
- 6:17:32You can review it in your Gmail before sending it.
- 6:17:35So, let's summarize the flow that I explained.
- 6:17:38The user sends a request to the AI application.
- 6:17:41The application makes the available tool descriptions accessible to the llm.
- 6:17:46The llm decides which tool to call and generates a tool call request.
- 6:17:51and the application invokes the tool through the protocol
- 6:17:54and the MCP server executes the requested operation and returns a result
- 6:17:59and the applications sends that result back to the LLM
- 6:18:03and the LLM generates a final response for the user
- 6:18:06and that's the basic idea behind MCP.
- 6:18:09It provides a standardized way to connect AI
- 6:18:12applications with external capabilities,
- 6:18:14making integrations easier to reuse across different applications and
- 6:18:19development environments. Welcome
- 6:18:21back aboard Neon Learned.
- 6:18:24Planet Agent is conquered. You're halfway there, five worlds to go.
- 6:18:29Fasten your seat belt. Next stop, Planet Structured.
- 6:18:32Here, you'll master prompt templates and make any LLM return clean structured data
- 6:18:37exactly how your app needs it.
- 6:18:40Your next world awaits. Enjoy the journey.
- 6:18:50Okay, starting from this session and over the
- 6:18:52next few sessions, we are going to explore
- 6:18:55a group of classes in LangChain called Prompt Templates.
- 6:18:58So far, we have been providing prompts to our LLM as simple text inputs.
- 6:19:03And depending on the role we needed, we create
- 6:19:06an instance of system message, human message,
- 6:19:08or AI message and pass a plain text string to it.
- 6:19:12And honestly, that's perfectly normal.
- 6:19:14I mean, for many projects, this approach works just fine.
- 6:19:18As our projects grow behind a certain size,
- 6:19:21having lots of plain prompt strings scattered
- 6:19:24throughout our code can make things messy.
- 6:19:27And it becomes harder to maintain our prompts.
- 6:19:30It makes it harder to reuse them across different parts
- 6:19:33of our application and keep our code organized.
- 6:19:36And for this reason, there are some libraries specialized for maintaining prompts.
- 6:19:42One example you might have heard of is DSPy.
- 6:19:46which provides a more programmatic approach to
- 6:19:49building and organizing LLM powered systems.
- 6:19:52And with approaches like this, instead of
- 6:19:54treating every prompt as an isolated string,
- 6:19:57you can organize your logic into reusable components.
- 6:20:01And of course, whether you need a dedicated framework
- 6:20:05like DSPy depends on the complexity of your project.
- 6:20:08For a small application, introducing an additional
- 6:20:12framework like this might be unnecessary.
- 6:20:14Besides DSPY, fortunately, LangChain also
- 6:20:17provides its own classes for prompt management.
- 6:20:21And these classes let us create dynamic reusable prompts
- 6:20:25without adding too much complexity to our application. The idea is pretty simple.
- 6:20:30Instead of hard coding every value directly into a string,
- 6:20:34we can define placeholders and inject different values into them
- 6:20:38whenever we need to use the prompt.
- 6:20:40If you have worked with Python formatted strings, This idea should feel familiar.
- 6:20:45We define a template with placeholders and then provide
- 6:20:49the values that should go into those placeholders.
- 6:20:52And the difference is that LangChain gives us dedicated
- 6:20:57object for managing these prompts and these templates.
- 6:21:00And because prompt templates can be used as runnable components, we can also
- 6:21:06connect them to other components using LangChains syntax, which is called
- 6:21:11L-C-E-L or Langchain Expression Language
- 6:21:14and that's where the name Langchain starts to make even more sense.
- 6:21:18We can chain different runnable components together to build more complex workflows.
- 6:21:25Now to save some time, I've already created a file called prompt template.py
- 6:21:30and placed some of the code we have written in previous session inside it.
- 6:21:35And there is nothing new. As you can see, it has a
- 6:21:38constructor here it has a chat method and it has an
- 6:21:41LLM call so if I run the program and as you can
- 6:21:45see I get a response so this is a code I've written
- 6:21:48and just write this code to follow along with me
- 6:21:53Oh
- 6:21:58now let's move on to our first prompt template class
- 6:22:01the first class we are going to look at is called
- 6:22:04prompt template it's one of the simplest classes in
- 6:22:07this family prompt templates are available through
- 6:22:10langchain core so if you want to import them we need to use langchain
- 6:22:14core so i say from langchaincore.prompts
- 6:22:20import i want to import prompt template okay this class
- 6:22:24and the purpose of this class is straightforward
- 6:22:27it lets us turn a regular string into a reusable template with dynamic placeholders
- 6:22:33let's see how that works with a practical example
- 6:22:36imagine that we want to generate an email whose
- 6:22:39content changes depending on the user's name and the
- 6:22:43company's brand name and we want the email to start with
- 6:22:47the user's name and end with the company's name.
- 6:22:50So instead of creating a completely new string
- 6:22:53every time, we can define a template and we
- 6:22:55can fill in the values whenever we need it. So first let's create a regular string.
- 6:23:02So I define a templat. I use triple quotes to write in multiple lines.
- 6:23:07I say here and here I have a placeholder and I name it a username.
- 6:23:14And then in the next line I say
- 6:23:17apologizing for the bad experience we will set an
- 6:23:21appointment to further discuss your problem and
- 6:23:26at the end sincerely and at the ends we have another placeholder called brand name
- 6:23:34okay so pay attention to these two username and
- 6:23:37brand name whenever we want a value to be dynamic
- 6:23:41we can put a placeholder just like when we have a
- 6:23:44formatted string and then inside these pair of
- 6:23:48curly braces we can have a variable name and in this case we have username and
- 6:23:53brand name that represents the company's name
- 6:23:56And at this point, this is still a regular string.
- 6:24:00We haven't converted it into a LangChain prompt template yet.
- 6:24:04So let's do that.
- 6:24:05To convert our string into a prompt template, we'll
- 6:24:09create an instance of the class we just imported.
- 6:24:12And then its constructor accepts the template and the variable names that we have.
- 6:24:17So let' s do it. I store the string inside a variable called template.
- 6:24:23and then we can have our prompt
- 6:24:27and our prompt is constructed using this class that I told you prompt template
- 6:24:33and then as I told first we have our input variables this is a dictionary
- 6:24:39this is the list sorry
- 6:24:40this is list and it contains our variables and we have a template
- 6:24:45So this is my template.
- 6:24:48and the variables that we have are username and
- 6:24:52here we have brand name okay we need to put them
- 6:24:56inside strings
- 6:24:58now this prompt that we have here gives us a method
- 6:25:01called format which we can use to insert the
- 6:25:04actual values into our template so let's say the
- 6:25:07user's name is Alex and the company's name
- 6:25:10is NeonLearn so we can generate a dynamic prompt like
- 6:25:14this i say prompt i use this method as i told you format
- 6:25:19and here i need to specify my variables the variables that i have in this prompt so
- 6:25:24the first one is username i want it to be alex and the latter one is brand name
- 6:25:32which is neonLearn
- 6:25:34okay so the method takes the values we provided and
- 6:25:38insert them into the corresponding placeholders
- 6:25:41and at the end let's print the result so i say print result
- 6:25:46Okay, I donate these lines and, oh, result.
- 6:25:49I forgot to store it in a variable called result.
- 6:25:52Let's run our program.
- 6:25:55Okay, that's it, dear Alex.
- 6:25:57And at the end, NeonLearn added was the whole purpose of this session.
- 6:26:01We have created a dynamic prompt template that
- 6:26:03we can reuse whenever we need to generate this
- 6:26:06kind of email.
- 6:26:07And if you want to generate an email for a
- 6:26:10different user or a different brand name, we
- 6:26:12don't need to rewrite the entire string.
- 6:26:14We simply call this method, I mean format, and we only change the values.
- 6:26:19And that's the main benefit of using prompt template.
- 6:26:22we separate the structure of our prompt from
- 6:26:25the specific values we want to insert into it
- 6:26:29you
- 6:26:35Okay, the next class we are going to explore is Chat Prompt Template.
- 6:26:39So what's the purpose of this one?
- 6:26:42In previous lesson, we used Prompt template to turn a simple
- 6:26:45string into a reusable template with dynamic placeholders.
- 6:26:49But in many applications, especially when we are building
- 6:26:53chatbots, we are not working with just one plain text.
- 6:26:57Instead, we're working with a sequence of conversation messages.
- 6:27:01and some messages come from the user some are generated by the AI and others provide
- 6:27:07system level instructions things that you saw before so if
- 6:27:11you want to create a reusable
- 6:27:13template we need a way to represent these
- 6:27:16different messages with different roles and
- 6:27:19that's exactly what chat prompt template is designed for
- 6:27:23so let's see how it works first
- 6:27:26I import it instead of prompt template I import
- 6:27:29chat prompt template.
- 6:27:31and i don't want these quotes here okay it was for our previous session
- 6:27:37okay this class that i imported provides a
- 6:27:40convenient class method called from messages
- 6:27:44and we use this method to create a chat prompt
- 6:27:47template from a list of messages and each message
- 6:27:50can be represented as a tuple something that we
- 6:27:54saw in open ai wrapper format it has two elements
- 6:27:57the first element specifies a role and the second
- 6:28:01element contains the message context so let's have it
- 6:28:04I told you that it is a class method.
- 6:28:07So I say chat from template, from messages, and it takes
- 6:28:11a list, a list of tuples, and each tuple is a message.
- 6:28:15So I want to have two messages, a system message and an AI message.
- 6:28:20So I specify its role like this.
- 6:28:23I say system and then I have its content and for user message, I say human.
- 6:28:30Let's create something practical with it.
- 6:28:32Imagine we are creating a translation application.
- 6:28:35We want the model to translate text from one language into another language.
- 6:28:40And we want the source language and the target language to be dynamic.
- 6:28:44In other words, we don't want to hard code the target
- 6:28:47language and source language into the prompt.
- 6:28:49We want to be able to change them whenever we use it. So let's create our prompt.
- 6:28:55As a system message I say,
- 6:28:57you are a translator from our source language to the target language and
- 6:29:03here
- 6:29:04i have another placeholder called input
- 6:29:07Let's take a closer look at it.
- 6:29:09Origin, as I told you, specifies the source
- 6:29:11language and this one is a target language and
- 6:29:14this input contains the text that we want to be translated.
- 6:29:19So, the system message tells the model about the task and
- 6:29:23the human message contains the actual text needs to be
- 6:29:26translated.
- 6:29:27Now that we have created our template, let's see how we can fill in these values.
- 6:29:32We can use format message method.
- 6:29:36from messages now we have a format messages so
- 6:29:39let's store this in a variable called prompt
- 6:29:43and then i say prompt format messages and then I'll specify these three values okay
- 6:29:51I set origin to
- 6:29:53English and
- 6:29:55I set to to German and I want the input to be I love you
- 6:30:02Okay, then I store it in a variable called
- 6:30:05messages because it returns a list of messages
- 6:30:09with different roles and I print it. I say print messages.
- 6:30:13Now let's run our program.
- 6:30:16and here you see the output as you can see
- 6:30:18Langchain takes our list of message templates
- 6:30:21fill in the placeholders and converts the result into message objects and
- 6:30:26internally the top will be provided are represented by the corresponding
- 6:30:30Langchain message classes and in this example we'll get a system message as you
- 6:30:34can see and we have a human message here okay
- 6:30:38that's the main idea behind chat prompt template but now
- 6:30:42I want to show you something even more interesting.
- 6:30:45How we can connect this prompt to our LLM using the syntax I told you about.
- 6:30:51LangChains LCEL syntax. This syntax is really simple.
- 6:30:57All you need is a pipe operator.
- 6:30:59And this way we can connected one component to another using a pipe.
- 6:31:04Then connect another component after that and continue building a chain as needed.
- 6:31:09The first component receives an input and produces an output.
- 6:31:13That output is passed to the next component as its input.
- 6:31:17And the next components processes it and produces another output and it continues.
- 6:31:22Okay, and this is how we can build a workflow.
- 6:31:25A workflow by connecting multiple different components together.
- 6:31:29And there is one important requirement to do that.
- 6:31:33Components used in this syntax needs to be runnable.
- 6:31:37needs to support the runnable interface. So otherwise you cannot do that.
- 6:31:43For example, this self.llm you see here is a
- 6:31:46runnble component and this class that you
- 6:31:49imported for generating your prompt is also
- 6:31:52a runable component or it's better to say
- 6:31:54it produces runnible objects for you and this
- 6:31:58means you can use these components inside
- 6:32:01the syntax lcel syntax. So let's
- 6:32:04do that!
- 6:32:06I told you we have two pronable components here.
- 6:32:09So here I can put the first component, which is our prompt.
- 6:32:14And then I use a pipe and then we have our LLM. Okay.
- 6:32:18And I store the result inside a variable called chain. So let's break it down.
- 6:32:23The first component is prompt and it receives
- 6:32:26the values needed to fill in our template.
- 6:32:29And it generates a list of chat messages and the second component.
- 6:32:33is our LLM and it receives those list of messages
- 6:32:37and it sends them to our language model to our LLLM
- 6:32:41and by connecting these two components we have created a single runnable chain
- 6:32:46so this component that we have here is also runnible
- 6:32:50and this means instead of manually calling the prompt
- 6:32:53collecting the resulting messages and then
- 6:32:56passing those messages to our lllm ourselves
- 6:32:59we can let the chain handle that sequence for us.
- 6:33:02And as I told you, this object that we have here is runnable
- 6:33:06and one primary feature of runnables objects is
- 6:33:10that you can call the invoke method on them.
- 6:33:13So here we can say chain and call the invoke method.
- 6:33:18And as i told you it's our first component
- 6:33:21and we need to provide the values that our first components in this change needs.
- 6:33:28so what values do we need we need origin to and
- 6:33:32input the exact values that we have here so i
- 6:33:35remove it and i move these values here and instead
- 6:33:39of equal sign i use colon because i want to have
- 6:33:42a dictionary here
- 6:33:44okay and i put curly braces around them and now i should put these inside
- 6:33:50quotations okay now everything
- 6:33:52is ready and i just need to store the result
- 6:33:56in a variable called response
- 6:33:59and I print the response.
- 6:34:02Okay, it's an AI message. I just want its content.
- 6:34:06okay let's run the application and it should return the german translation of
- 6:34:12uh the sentence so it says it uh so that's totally correct and that's a whole
- 6:34:18process instead of
- 6:34:20manually passing data from one component to another
- 6:34:23we have created a chain that handles the the flow
- 6:34:26for us and the best part is that we can extend
- 6:34:29this we can expand this chain whenever we need to
- 6:34:33but we can add more components after the LLM or build more complex workflows.
- 6:34:37We can add components before it and whenever we want,
- 6:34:40as long as the components are runnable objects.
- 6:34:44Ladies and gentlemen, we're passing through a little turbulence.
- 6:34:49If things feel harder right now, that's completely normal. It means you're climbing.
- 6:34:54Every AI engineer you admire flew through this exact same part of the journey.
- 6:35:00Take a deep breath, stretch your wings, maybe grab a glass of water.
- 6:35:04Now look out the window, see how far you've already come.
- 6:35:08The view only gets better from here, let's keep climbing.
- 6:35:21Alright, next let's imagine that we want to extend our previous example
- 6:35:26by including the conversation history between the user and the AI.
- 6:35:31And we want the system message to stay at the beginning, the latest user message to
- 6:35:36stay at end, and the conversation story to be inserted in between.
- 6:35:41And the question is why might we want do this? There are several reasons.
- 6:35:46We might want the LLM to take previous messages into account so
- 6:35:50that its responses are more relevant to the ongoing conversation.
- 6:35:54Or we might want to use few-shot prompting, where we
- 6:35:58provide a few examples of the kind of output we expect.
- 6:36:01And this way, we kind of control the output,
- 6:36:05something that we saw in previous projects.
- 6:36:08And for this example, our focus is on few-shots prompting.
- 6:36:13and imagine that we are building a translation application and
- 6:36:17we want the model to wrap its translations in three angle brackets on each side
- 6:36:22and we can provide a couple of examples
- 6:36:25a couple examples in the conversation history
- 6:36:28to show the model the exact format we expect
- 6:36:31The point is that currently this class, I mean Chat Prompt template by itself
- 6:36:36doesn't automatically insert a separate list of conversation messages.
- 6:36:40So we need another class and by using this new class we can create placeholder here.
- 6:36:47So
- 6:36:47at first I import this class that I told you which is called
- 6:36:51message placeholder, messages placeholder more accurately.
- 6:36:56and now before we use it we use the conversation history that we have
- 6:37:00so on the top I create a list called history
- 6:37:04and inside this list I have four messages
- 6:37:07two human messages and two responses by the LLM
- 6:37:10so I say human message and my message is I am tired
- 6:37:15okay and the answer is Ich bin mude with the translation more accurately
- 6:37:19so it is an AI message and as I told you I want three angle brackets
- 6:37:25three before and three after Ich
- 6:37:27bin müde let's duplicate these two lines, okay?
- 6:37:31I hate you Ich hasse dich
- 6:37:34These examples demonstrate the output format we want the model to follow
- 6:37:38and the translated text should be surrounded by three angle brackets on each side.
- 6:37:44Okay, now let's update our chat prompt template.
- 6:37:47Previously, our template contained a system message and a human message.
- 6:37:51This time we are going to insert a messages placeholder between them.
- 6:37:54So here I'm going to insert this messages placeholders that I told you.
- 6:37:59And here I need to specify a variable.
- 6:38:02like these variables that we had so here i set
- 6:38:06variable name and i set it to conversation history
- 6:38:10so now this is all we need and let's take a closer
- 6:38:13look the first message here is a system message
- 6:38:16it tells the model which languages to translate
- 6:38:19between and next we have this place folder it tells
- 6:38:22lang chain where the conversation history should be
- 6:38:25inserted the whole point of this session is that we cannot
- 6:38:29put this conversation history directly here.
- 6:38:32First, we need to create a placeholder and
- 6:38:35this placeholder contains a variable name.
- 6:38:38And when we want to invoke our chain, we
- 6:38:40need to specify this variable and our value would be this history.
- 6:38:45at the end that we updated our prompt we need to pass the conversation history
- 6:38:49and invoke our change so we only need to specify this new variable that we had here
- 6:38:55so i add conversation history and here the order
- 6:38:58doesn't matter so we have conversation history
- 6:39:02and i set it to my history the history that i
- 6:39:05stored in the list called history and that's it
- 6:39:09Let's run our program.
- 6:39:11It says messages placeholder object is not callable. What's wrong with it?
- 6:39:17Oh, here we forgot a comma.
- 6:39:20Okay, again, one more time.
- 6:39:23Okay, congratulations.
- 6:39:25And this time, as you can see, the model receives the
- 6:39:28system instruction and it also receives the examples.
- 6:39:32And this way it realizes the format that we expect.
- 6:39:36So the previous examples demonstrate the output format, the output format we want.
- 6:39:41And so the model has a reference for how to format its new translation.
- 6:39:45As you can, there are three angle brackets at the beginning and at the end.
- 6:39:50So exactly what we wanted.
- 6:39:51and we used few shot prompting to achieve this goal.
- 6:40:01Okay, the next topic in this course is something called Structured Output.
- 6:40:06This is a really useful and important concept
- 6:40:08because in many applications, we need our LLM
- 6:40:11to return its response in a special format.
- 6:40:14For example, imagine that we provide the model
- 6:40:17with several paragraphs describing a product,
- 6:40:20and we wanted to extract some specific information
- 6:40:24such as product's name, price, seller, and
- 6:40:27the product's weight. We don't just want a paragraph describing the product.
- 6:40:31We want a structured response containing exactly the fields we are interested in,
- 6:40:36such as product name, price, seller, and weight. And why? Why is that important?
- 6:40:41Because we are going to use these values in the rest of our application.
- 6:40:45We want to be able to access the product's price through a special field,
- 6:40:49retrieve the seller's name through another field, and so on.
- 6:40:53Depending on which library, framework, or wrapper we are using,
- 6:40:57the implementation can be different.
- 6:40:59So, for example, if we were working directly with the OpenAI SDK,
- 6:41:04we could use its supported structured output capabilities.
- 6:41:07And that way we could specify the expected output format and schema.
- 6:41:12But since we are using LangChain, I want to
- 6:41:15show you the best practice in LangChains.
- 6:41:18I mean, I wanna show you how we can approach
- 6:41:21this problem within the LangChai ecosystem.
- 6:41:24And remember something I've emphasized throughout this course.
- 6:41:28Understanding the concept is more important
- 6:41:31than memorizing every implementation detail.
- 6:41:34You don't necessarily need to remember every method or every library function.
- 6:41:38What's important is knowing that a capability exists.
- 6:41:42Understanding what it is used for and being able to
- 6:41:45find the relevant documentation when you need it.
- 6:41:49That's especially true in software development today. tools and APIs change rapidly.
- 6:41:55But something that is permanent is understanding the underlying concept.
- 6:41:59These concepts are permanent, but the tools, frameworks and other stuff change.
- 6:42:05So the question is, how are we going to implement structured output in LangChain?
- 6:42:11One of the best ways is using Pydantic.
- 6:42:14So first let's install it and I will elaborate on that. So here I say pipenv
- 6:42:21Install Pydantic
- 6:42:25Okay.
- 6:42:27But before we use Pydantic with Langchain, I want to spend
- 6:42:31a little time understanding how Pydantic itself works.
- 6:42:36And I need to say its main purpose is data validation.
- 6:42:39It tells us whether a piece of data follows a schema or not.
- 6:42:44Imagine you are building a web application, a web
- 6:42:47application with a form where users enter information such as their email address,
- 6:42:52username and age for example.
- 6:42:56and the user might accidentally enter invalid
- 6:42:59information or they might deliberately do that
- 6:43:02so that's really important to validate the data we receive
- 6:43:06we need some rules and Pydantic is a library that
- 6:43:09helps us define and enforce these kinds of rules
- 6:43:13before we look at Pydantic itself let's briefly review a python feature
- 6:43:17Pydantic builds on type hints suppose we define a variable like
- 6:43:22name and its value is for example John okay now if we want to indicate this name
- 6:43:29that we have here is expected to be a string we can write something like this
- 6:43:33we can put a colon after and we say it is a string what you see here is a type
- 6:43:38hint it tells tools that analyze our code that this variable is expected
- 6:43:43to contain a string and type hints make code easier to understand and
- 6:43:48and can help development tools provide better autocomplete,
- 6:43:52auto-intelligence, and static error checking.
- 6:43:55So, that's the purpose of type hint.
- 6:43:57However, there is an important detail to understand.
- 6:44:00Python type hints don't automatically enforce these types at runtime.
- 6:44:05For example, writing something like this doesn't
- 6:44:08prevent someone from assigning a different type.
- 6:44:11For example putting a number inside this variable.
- 6:44:14And the point is that Pydantic takes this idea further.
- 6:44:19I mean, when we define a Pydantic model, it uses the declared types.
- 6:44:23We also specify some validation rules and Pydantic uses them.
- 6:44:27Uses to process and validate the data at runtime.
- 6:44:31So you can think of type-ins as a way to describe the expected structure.
- 6:44:36While Pydantic provides a mechanism for checking that data against the structure.
- 6:44:42So that's the point.
- 6:44:44So, to get started, to learn about PyDantic, I
- 6:44:48create a new Python file called PyDantikValidation.
- 6:44:53okay that's it and now we will use this file to explore
- 6:44:57pydantic before integrating it with lang chain
- 6:45:01so the first step is to import base model from pydantic
- 6:45:06so i say from pydantic import baseModel okay what is baseModel?
- 6:45:12it's a base class that we inherit from when defining our pydantic data models
- 6:45:18let's have an example we want to define a user with three fields
- 6:45:22id name and age so first i create a new class called user
- 6:45:29it inherits from base model, as I told you.
- 6:45:32and i want to have three fields id name and age i
- 6:45:37want id to be an integer name to be a string str
- 6:45:43i need to say int sir and int and this way with these
- 6:45:47type hints i am defining the shape of my user
- 6:45:51my user is supposed to have these three fields okay now
- 6:45:56let's create an instance of this model and here's
- 6:46:00Here's where things get interesting. Let's create a user.
- 6:46:04I say user and I specify the value of these fields.
- 6:46:08I say ID is 1, but I put 1 inside quotation.
- 6:46:14and I say name,
- 6:46:15name is Alice
- 6:46:18and then age 25 okay and then i print it first
- 6:46:22i store it in a variable called user and then
- 6:46:26i print print user
- 6:46:29and you have probably noticed something unusual here
- 6:46:32I have 25 in start quotations technically a string and the same for id
- 6:46:37so these are now strings not integers
- 6:46:41but when I run my code you can see now that these are actually integers
- 6:46:47so as you can pidentic has automatically converted the type
- 6:46:52this kind of conversion is especially useful when working with web applications
- 6:46:57because data submitted through HTML forms often arrives as strings
- 6:47:02even when the user enters a number.
- 6:47:04So Pydantic can convert compatible values into the types declared by our model,
- 6:47:10saving us from writing some of that conversion logic ourselves.
- 6:47:15However, this doesn't mean Pydantic can convert every value into every type.
- 6:47:20Conversion depends on the input and the field's validation rules.
- 6:47:25if a value can't be processed successfully, Pydantic raises a validation error.
- 6:47:30For example, if we had something like this, ABC. Let me run our code.
- 6:47:35As you can see, it raises an error, so it's not possible.
- 6:47:39It is only possible when we have number.
- 6:47:42Okay, now that we understand the basics, let's look
- 6:47:45at another useful feature, the field function.
- 6:47:48So far, we have only specified the expected type of each field. For example.
- 6:47:54we have set that name must be a string.
- 6:47:57But what if we also want to specify a minimum and maximum length?
- 6:48:01Or what if you want to restrict the user's age to a particular range?
- 6:48:07This is where field becomes useful. So first I import it.
- 6:48:12I say field here
- 6:48:13and we can
- 6:48:15use field to attach additional validation constraints.
- 6:48:19And this way we can have more control.
- 6:48:22Here I want to have mean length 2, max length 50.
- 6:48:25So let's say field, mean
- 6:48:28length, 2 and max length, 50.
- 6:48:32And then for age, I'm going to say
- 6:48:36g e 0 and
- 6:48:37le 120 okay let's break this down for the name field
- 6:48:43we have specified
- 6:48:45mean length to max length 50 this means the
- 6:48:48string must contain at least two characters
- 6:48:52and no more than 50 characters and for the age we are specifying that i mean
- 6:48:58this g you see here means greater than or equal to and le means
- 6:49:03less than or equal to. So the age must be between 0 and 120 inclusive.
- 6:49:10Okay.
- 6:49:11And if we try to create a user with a name that
- 6:49:14violets the length or an age outside this range,
- 6:49:18Pydantic raises a validation error.
- 6:49:21And this allows us to define our validation rules
- 6:49:24directly alongside the field they apply to.
- 6:49:28Okay.
- 6:49:29There is one more feature of field that will be particularly important
- 6:49:33when we get to structure out within LangChain.
- 6:49:36Besides validation constraints, field also lets us attach metadata,
- 6:49:41including a description and examples. Why is that useful?
- 6:49:46When we ask an LLM to generate structured data,
- 6:49:49we want to clearly explain what each field represents.
- 6:49:52The name of the field that we have might not always be enough and even constraint.
- 6:49:58A description and examples can help the model a lot.
- 6:50:02So here we can have a description.
- 6:50:04For example, for this field, I set its description to
- 6:50:09the users.
- 6:50:10Full name.
- 6:50:12and I can provide some examples for example Alice and
- 6:50:19Okay?
- 6:50:20Now, in addition to specifying the type and length constraints,
- 6:50:24we have documented what the field represents and provided two example values. Okay.
- 6:50:31So now you have a good understanding of what Pydantic does.
- 6:50:37you
- 6:50:43Alright, let's put what we learned about Pydantic in previous lesson into practice.
- 6:50:48We are going to use Pydantic to control the structure of our LLM's output,
- 6:50:52so we can get the exact type of information we need in a predictable format.
- 6:50:57For this project, I'm going to copy a paragraph from Elon Musk's Wikipedia
- 6:51:02and pass it to my LL as input.
- 6:51:04Then I'll ask the model to extract four pieces of information.
- 6:51:09his name, his age, associated companies, and his net worth.
- 6:51:14So let's go to our code and do this.
- 6:51:16For this example, I'm going to use the project that we had prompt template. Okay.
- 6:51:21That's perfectly fine.
- 6:51:23So first I'm gonna import base model and field, which we saw in the last session.
- 6:51:28So I say from Pydantic import base, model and filled.
- 6:51:34Then I'll create a class called user that inherits
- 6:51:37from base model with the field that I want.
- 6:51:40So here I'm going to create a a class call user.
- 6:51:44It inherits it from
- 6:51:46baseModel.
- 6:51:48and the field that I have are name
- 6:51:52H
- 6:51:54Companies and Net Worth.
- 6:51:58now it's turn for typins i want it to be
- 6:52:02stir this one integer and companies are a list of strings
- 6:52:08and net worth can be a simple string okay
- 6:52:12and we need four fields for each one containing a description and this
- 6:52:17description is going to guide our element so here i add a field with the
- 6:52:22description i say the name of the user
- 6:52:26Let's just copy it and paste it here.
- 6:52:30Here I say the age of the user.
- 6:52:33And here say the
- 6:52:36companies associated with the user and finally
- 6:52:40for net worth I say the net worth of the user.
- 6:52:43Okay, that's it.
- 6:52:45And as you can see, we are specifying the
- 6:52:48expected data type for each field, along with
- 6:52:51a description explaining what information it should contain.
- 6:52:54Next, for this particular task, it's a good idea to use a lower temperature.
- 6:52:59For example, it is okay to say 0.1 because we don't want any creativity at all.
- 6:53:05We are dealing with facts and a lower temperature is more suitable here.
- 6:53:10So it's for temperature. Now I remove what we have inside our chat method
- 6:53:16because we are going to replace it with the implementation
- 6:53:20that we have for our structured output project.
- 6:53:23Okay, so the question is how do we get structured output in LangChain?
- 6:53:28And the answer is that our LLM provides a method called withStructuredOutput
- 6:53:34and this method allows us to specify the schemas that the models should follow.
- 6:53:40So I'm going to call this method. I say self.llm with structured output.
- 6:53:46and here i need to specify two things the first
- 6:53:50thing is the schema that we defined here and the
- 6:53:53other one is a method okay i say method to be json
- 6:53:57schema okay it's suggesting really great okay
- 6:54:00so
- 6:54:01we are passing two arguments here the first argument
- 6:54:04is our user schema which defines the fields
- 6:54:06and their expected types the second argument is json-eschema
- 6:54:11this way we tell Langchain to use the json schema
- 6:54:14approach for structured output now let's pass our
- 6:54:18input and as i told you our input is going to be
- 6:54:21a paragraph from Elon Musk's wikipedia page so
- 6:54:25i'm going to store this llm with structured
- 6:54:28output in a variable for example structured llm
- 6:54:32then I'm going to invoke it. So I say structured LLM, invoke,
- 6:54:37and I paste this paragraph here. I save the result in a variable called response,
- 6:54:43and then I print the response. Print response.
- 6:54:47So let's run our program and see what happens.
- 6:54:51Okay, as you can see here, we have name,
- 6:54:53We have age and it's really nice because the age wasn't provided here.
- 6:54:57It was his birthday.
- 6:54:59And as you can see, it is calculated and companies are Tesla and SpaceX. Net worth.
- 6:55:06Uh, okay. Good for him.
- 6:55:08And great. Our model returns extracted information in the structure we defined.
- 6:55:13And instead of asking the model to return an arbitrary
- 6:55:17paragraph and then to try to parse it ourselves.
- 6:55:20We have specified a schema with four fields, name, age, companies, and net worth.
- 6:55:25Plankchain uses the schema to request a structured output,
- 6:55:28and Pydantic helps validate the resulting data against our expected types.
- 6:55:33Welcome back aboard Neon Learned. Planet
- 6:55:36Structured is conquered.
- 6:55:37Six worlds down, four to go.
- 6:55:41Fasten your seat belts. Next stop, Planet Memory.
- 6:55:44Here, you'll save every conversation to a database
- 6:55:47so users can start new chats and pick up old ones, just like ChatGPT.
- 6:55:52Your next world awaits. Enjoy the journey.
- 6:56:01Now let's move on to our next topic, next important topic, which is check pointers.
- 6:56:07This is a more advanced topic, but it's also extremely
- 6:56:10practical and useful in many real world applications.
- 6:56:14Let's gradually build an understanding of what check pointers
- 6:56:18are, why we need them and what we can use them for.
- 6:56:22CheckPointers can have different use cases, although in this course their central
- 6:56:27role is to save and restore an application's state, including conversation history.
- 6:56:34To save time, I've already created a file called
- 6:56:37checkpointers.py and I've put this code that you see
- 6:56:41here.
- 6:56:41There is nothing fundamentally new here.
- 6:56:43Here I created an agent using create agent function that I taught you.
- 6:56:49And I've created an Agent that doesn't have any tool yet.
- 6:56:53So we can think of this agent as a basic conversational chatbot.
- 6:56:57Let's test our program. Let's run it and see the result. So I say hello. Okay.
- 6:57:03What is the capital of France?
- 6:57:06What about president?
- 6:57:08As you can see, it's a normal chatbot that we implemented
- 6:57:12and it has history and there is really nothing new.
- 6:57:15I just wanted to save your time. Okay, let's get back to our question.
- 6:57:19Why do we need a check pointer?
- 6:57:21When we first started building chatbots, you quickly learned
- 6:57:24that LLMs don't automatically remember previous conversations.
- 6:57:28If you want a model to have access to conversation history, we need to store
- 6:57:33that history ourselves somewhere and send it back to the model each time.
- 6:57:37And this way the model has history. Even
- 6:57:40here you can see this approach.
- 6:57:42You can see that we are recording this conversation history.
- 6:57:46Here we have a list for messages. Here we a system message or first message.
- 6:57:51And then each time the user asks something, we append that message.
- 6:57:56And whenever we get the response, we append that message as well.
- 6:57:59This way we are recoding messages and when we want to
- 6:58:03send a new request, we pass this list of messages.
- 6:58:06And this way, the model receives the previous messages as context
- 6:58:10and can respond based on the conversation we have had so far.
- 6:58:15But there is a problem. There is a problems with this approach.
- 6:58:18Our code has become a little more complicated.
- 6:58:21We have to remember to append the user's message
- 6:58:24and we have to do the same for assistance response.
- 6:58:27We also need to think about different edge cases.
- 6:58:29For example, what happens if we encounter a connection error?
- 6:58:34What if the response is empty? What if we had providers token limit?
- 6:58:37So we have these edge cases.
- 6:58:39And another important limitation is that our conversation
- 6:58:43history is stored in a regular Python list.
- 6:58:46And where does that list live? It lives in memory or RAM.
- 6:58:50The thing is that as soon as the program terminates, that list disappears.
- 6:58:55And if we restart the application, the previous conversation is gone.
- 6:58:59So what if we want to preserve conversation history?
- 6:59:02In that case, we need to store it on database,
- 6:59:05instead of keeping it in a simple Python list.
- 6:59:08However, that introduces additional complexity.
- 6:59:11We need to save records, retrieve them from database, organize conversations,
- 6:59:15handle database errors, and test all those operations.
- 6:59:19We also need to distinguish between different users.
- 6:59:22So, you can see that there are many things that we need handle.
- 6:59:26Think about an application like ChatGPT.
- 6:59:29user might have separate conversations,
- 6:59:31conversations about hundreds of different topics,
- 6:59:34and the user wants to switch between them.
- 6:59:37And each conversation needs its own history,
- 6:59:39and that history needs to be associated with the correct conversation.
- 6:59:43So as you can see,
- 6:59:45building all of that complexity ourselves would
- 6:59:48require quite a bit of additional code.
- 6:59:50and this is where check pointers in LangChain become useful.
- 6:59:54Sorry, check pointers are in LangGraph. It's a feature in Lang Graph.
- 6:59:58LangGraph provides a mechanism for saving and restoring graph state.
- 7:00:03And we will talk about states in more detail in future.
- 7:00:06And when we use an agent built with LangGraph, a check pointer can manage its state,
- 7:00:12and it can do it across multiple invocations. That means,
- 7:00:16we don't have to manually maintain the conversation history in a list.
- 7:00:20So that's the whole idea.
- 7:00:22This feature, I mean check pointer, is going to handle the state of an agent for us.
- 7:00:27you
- 7:00:33When working with check pointers, the first thing
- 7:00:36that we need is a saver in the first place.
- 7:00:38And in this session, I'm going to use In-memory saver,
- 7:00:42which saves the information on memory, on your RAM.
- 7:00:45And later we will switch to something else, something
- 7:00:49that can persist your data in a database.
- 7:00:51So for learning how check pointers work, first
- 7:00:55we will use In memory saver which is simpler.
- 7:00:58And later when you want to store your data on database,
- 7:01:02it is as simple as changing your saver. So here I say from LangGraph
- 7:01:07as I told you it's a feature in LangGraph not Langchain
- 7:01:10and you don't need to install LangGraph
- 7:01:12because it is automatically installed when you install Langchain
- 7:01:16and then from checkpoint and from memory if I'm not wrong import in memory saver
- 7:01:22as a result whenever we want to save state in memory
- 7:01:26we use this option InMemorySaver and as a rule of thumb
- 7:01:30When developing, always use this option.
- 7:01:33And when everything works and when you want to
- 7:01:36switch to production, you can use another saver.
- 7:01:38And as I told you in the previous session, we no longer
- 7:01:42have to manage the conversation history ourselves.
- 7:01:46So I want to remove all the code we had for managing history in our code.
- 7:01:51First, we had this list for recording the messages.
- 7:01:54and every time we had to append the user's message and the AI's message.
- 7:01:59So I don't want these two lines as well. And here we don't have this list anymore.
- 7:02:05So I put three dots here and later I will show you what else we should put here.
- 7:02:11But what I did raises a question.
- 7:02:13If we remove the message list, what happens to our system message?
- 7:02:17Previously, we were storing the system message inside that list.
- 7:02:21Now we need another way to provide those instructions to agent.
- 7:02:24And yes, fortunately create agent has an argument
- 7:02:28for explicitly introducing these instructions.
- 7:02:31In other words, it gives you, I mean, this create agent gives
- 7:02:35you an option to specify a system prompt and it's an argument.
- 7:02:38So I say system prompt, and I say give short answers, just like before.
- 7:02:44And we need to pass our check pointer here. So I specify check pointer.
- 7:02:49and I create an object from this class, okay? That's all we need.
- 7:02:53And this tells the agent to use in-memory check pointer
- 7:02:56to save and restore its state between invocations.
- 7:03:00Now the check pointer can manage the conversation history for us
- 7:03:04and it can track our conversation. It tracks the conversations through a thread ID.
- 7:03:11So another thing you need to pass is thread ID,
- 7:03:15The thread ID is an identifier, an identifier
- 7:03:18that LangGraph uses to associate checkpointed
- 7:03:21state with a particular conversation thread.
- 7:03:24In other words, it tells LangGraph which conversation
- 7:03:27we want to continue, and without this identifier,
- 7:03:30the system wouldn't know which conversation
- 7:03:33history to receive when we make another request.
- 7:03:36The useful part of it is that we are no longer
- 7:03:39limited to a single conversation, we can have
- 7:03:42multiple conversation, multiple independent conversation histories,
- 7:03:46each identified by its own thread ID. So that is the reason we need a thread ID,
- 7:03:51previously we didn't need that because we only had one conversation,
- 7:03:55but now we can have multiple and we need a thread id for each conversation.
- 7:04:00For example, we could have one conversation in
- 7:04:02which we are asking questions about friends
- 7:04:05and another in which we are discussing cooking.
- 7:04:09So the two conversations have separate histories.
- 7:04:12And when we return to the conversation about friends,
- 7:04:14we can continue from its previous context.
- 7:04:18So the question is, how do we specify the thread ID?
- 7:04:22The answer is that we need a configuration dictionary.
- 7:04:25So first we need such a configuration
- 7:04:28dictionary and then we will pass it when we are invoking the agent.
- 7:04:32Before we invoke our agent,
- 7:04:35I create this dictionary, I store it in a variable called config.
- 7:04:40And in this dictionary we need to have a key called configurable.
- 7:04:44So I say configurable, this
- 7:04:47is itself is a dictionary.
- 7:04:49And this dictionary has a key, called thread ID.
- 7:04:53The thread ID we were talking about. And here you specify an arbitrary name.
- 7:04:58So I call it session one. Okay, that's it. And then.
- 7:05:03when we invoke our agent, we specify a second argument. That is config.
- 7:05:09Let's say config and I set it to the config I created about. Okay,
- 7:05:13that's it.
- 7:05:14And what about this part? Here we don't have our messages list anymore.
- 7:05:19Here I still need to have a list, but this list has only one message
- 7:05:24because we are tracking the conversation history somehow else.
- 7:05:27So here I add a human message.
- 7:05:30this message is basically the last message that the user has sent
- 7:05:35so i say user input okay
- 7:05:37now i guess everything is in order
- 7:05:40and the only step left is to test our program to make sure everything is
- 7:05:45working okay so let's run it and let's ask a few questions
- 7:05:49hello okay
- 7:05:50what is the capital of france who is its president
- 7:05:57So at this point, if I get the right answer, it
- 7:05:59means our code works correctly because here
- 7:06:01we didn't do anything about managing our
- 7:06:04conversation history and it is done only through
- 7:06:07the check pointer that we added here. Okay, so I enter...
- 7:06:11and it answers correctly so we achieved what we
- 7:06:15wanted as you can see the agent can use the context
- 7:06:18from the previous question to understand what we
- 7:06:21are referring to and this was the purpose of this
- 7:06:24session and in the next session we will do something really
- 7:06:29interesting
- 7:06:34all right now let's get to the exciting part as i
- 7:06:37mentioned earlier we can have multiple sessions each
- 7:06:41with its own conversation history and now we
- 7:06:43are going to implement that feature we are
- 7:06:46going to Implement this feature in our command
- 7:06:48line application and fortunately this is really
- 7:06:51straightforward in this example we are gonna have two
- 7:06:55separate sessions and whenever the user types ch
- 7:06:58the application will switch to the other session so let's implement this first a
- 7:07:03at the beginning of this method, I mean chat loop method,
- 7:07:07we're going to have a flag and I call it active session.
- 7:07:11and I set it to session1
- 7:07:14now let's add a new condition after this if condition that we have here
- 7:07:19when we check if the user has entered Q or not
- 7:07:22after this condition I add another condition and I check if the user had entered CH
- 7:07:29if the users has entered CH in that case we want to switch the session
- 7:07:33so in that I want to update this active session so I say active session is equal to
- 7:07:39and if the current session is session1
- 7:07:42i say if the correct active session is equal to session1
- 7:07:47in that case i want to switch to session2 otherwise in the case that active session
- 7:07:54is equal session2 i want update our active session to session 1
- 7:08:01and then i print a message i print done
- 7:08:04so this way we can realize that it is switched successfully
- 7:08:08And then I use the continue keyword, the continue statement.
- 7:08:12And it skips the rest of our current loop and prompts the user for another input.
- 7:08:19next let's update the label that we have here
- 7:08:22i want to display the current active session
- 7:08:25so here i use a formatted string i place a placeholder here
- 7:08:29i say active session so the user can see which session they are currently in
- 7:08:35and then i add here or type c let's change it to double quotations
- 7:08:41it's nicer yes okay or type ch to change sessions and
- 7:08:48I don't want to hard code this part. I want to use our flag or active session flag.
- 7:08:53So I replace it with active session. Let's run our program.
- 7:08:58Okay. I say.
- 7:09:00hello we are in the session one okay now i ask what is the capital of france
- 7:09:07now i want to switch now i am in session two i say what is capital of spain
- 7:09:15okay, moderate. now I want to switch. now, I ask who is its president.
- 7:09:21in this session, we were talking about France.
- 7:09:25it says Emmanuel Macron and I switch again and
- 7:09:29this time I again ask who is its president
- 7:09:34and we were talking about Spain in this session
- 7:09:37so it says yes Spain doesn't have president
- 7:09:41prime minister okay yes we successfully implemented
- 7:09:44what we wanted we wanted two separated sessions
- 7:09:47with their own conversation histories and we
- 7:09:50successfully implement this concept this
- 7:09:54feature.
- 7:09:54Ladies and gentlemen, the captain has asked me to pass along a personal message
- 7:10:00from the flight deck.
- 7:10:01We've now reached cruising altitude and every like you leave helps
- 7:10:05us climb even higher.
- 7:10:08To keep your seat reserved for every future flight, simply subscribe.
- 7:10:13And the cockpit door is always open.
- 7:10:15Leave a comment and the captain will read every single one.
- 7:10:20Cabin crew prepare for the next lesson.
- 7:10:22Nooo!
- 7:10:31All right, the next step is to store our conversation
- 7:10:35history in a database instead of keeping it only in RAM.
- 7:10:38And as I mentioned earlier, we can use different databases for this purpose.
- 7:10:43But the choice for this project is PostgreSQL.
- 7:10:47And honestly, the choice depends on our application and
- 7:10:50requirements, your experience and many other factors.
- 7:10:53But for the sake of this project, we are going to use Postgre SQL.
- 7:10:56And I've already installed PostgreSql.
- 7:10:59it's running on port 5432 and installing Postgres SQL isn't part of this course.
- 7:11:05So if you want to follow along, you need to install Postgres on your machine.
- 7:11:10And that's not difficult at all. You can download it, install it.
- 7:11:14And if you face any problem, you can ask ChatGPT or any other AI assistant for help.
- 7:11:19So that's really easy. And after installation, I can test it.
- 7:11:24So I say PSQL...version.
- 7:11:28it says the version of PostgreSQL on my machine
- 7:11:31and then I say PG is ready
- 7:11:35and it says it is running on this board 5432 you can connect to it
- 7:11:41you can to our database server
- 7:11:43so I say psql-u postgres
- 7:11:48here dash u specifies the username in my case the username is postgres
- 7:11:54and then when I press enter
- 7:11:56it asks for my password the password that I set during installation so I say admin
- 7:12:03and I can connect to our database so everything is
- 7:12:06working correctly and you can also inspect your
- 7:12:09database visually using a database management tool
- 7:12:12something like data grip and this way you can
- 7:12:15visually observe what's going on in your database
- 7:12:19that's it now that our database is ready we need
- 7:12:22to install the package that allows Langchain to
- 7:12:25save checkpoints in our Postgresql database so
- 7:12:29So I open up my internal terminal and I say
- 7:12:33pipenv install langchain checkpoint Postgres.
- 7:12:38let's check if it's okay i don't have any typo no i guess i have a typo here
- 7:12:44oops the problem is that here i'm saying lang chenit is lang graph
- 7:12:49lang graph checkpoint postgres so lang graph now it should be okay
- 7:12:56okay it is installing this package and i don't have
- 7:12:59any typo and it is actually resolving this package
- 7:13:02now let's take a look at the database connection string i've
- 7:13:07already added to my.n file and you should have one like me
- 7:13:13So this is my connection string and as you can see it has different components.
- 7:13:19This Postgresql you are seeing here specifies the connection scheme.
- 7:13:25This PostgreSQL admin specifies
- 7:13:28the username and password and this localhost
- 7:13:31states that the database service is running on
- 7:13:35the same machine, the same as our application and this is a port.
- 7:13:40and this is a database that we need to have on our database service
- 7:13:45and as you can see i've disabled ssl mode since you are working in a local
- 7:13:50development environment so we don't need this setting again
- 7:13:55one point is that it's better to have another
- 7:13:57database because i used this database before
- 7:14:00recording this video to test things that i
- 7:14:04wanted to teach you so here it's best to add
- 7:14:09a new database and I call it chat history 2
- 7:14:14Okay.
- 7:14:15Okay, chat history 2. That's great. And I change it to chat history2.
- 7:14:19Now, everything is in order.
- 7:14:21Now, let's get back to our application and connect our agent to Postgres.
- 7:14:26First, inside my main function, I want to read the
- 7:14:30connection string from our environment variable.
- 7:14:33So, I say OS environment and the name of my
- 7:14:37variable is, let us copy it to avoid typo.
- 7:14:42okay that's it and i store it in a variable
- 7:14:46called db url db URL next instead of importing
- 7:14:50in memory saver we'll import postgres saver so here i say postgres and i import
- 7:14:57postgressaver that's exactly what i need
- 7:15:01this Postgres saver that we imported provides a method called from connection string
- 7:15:07so we're going to use it here I say PostgresSaver from connectionString
- 7:15:13and I need to provide this URL
- 7:15:16and then I can use it with the with statement
- 7:15:20I say with and I get a check pointer
- 7:15:25Now this check pointer object provides a method called setup.
- 7:15:30This method creates the database tables and other structures
- 7:15:34that Postgres CheckPointer needs to store checkpoint data.
- 7:15:38And the point is that we need to execute it just for the first time.
- 7:15:43Once necessary tables exist, we generally don't need to run it manually every time.
- 7:15:48So for the next executions, you can comment it out or have a script for it for the
- 7:15:53first time, although if you run it again, it's
- 7:15:56normally fine and it doesn't lead to duplication.
- 7:15:59now inside this with statement i want to create
- 7:16:02my agent object or in other words i want
- 7:16:06to move these lines here and yes that's it now the
- 7:16:10only step left is to make a small change to our
- 7:16:14agent class previously we created an in memory
- 7:16:17saver now we no longer want to do that
- 7:16:20because we have another check pointer so what i need to do is to pass
- 7:16:26this check pointer that I have here to the constructor of this class.
- 7:16:31And then I add it to my constructor here. Okay. Check pointer.
- 7:16:36And then we can use it here.
- 7:16:38So instead of in memory saver, we can have this checkpointer. Okay. That's it.
- 7:16:44Everything else about conversation logic can remain
- 7:16:47the same, including our use of thread ID to keep,
- 7:16:50to keep different sessions separate and that's it,
- 7:16:54We have now switched from storing conversation state in memory to storing
- 7:16:58checkpoints in PostgreSQL. Now we need to test our program.
- 7:17:03So here you can
- 7:17:05find an object, which line
- 7:17:11Oops, here I'm using apprentices now.
- 7:17:14It's a dictionary. Okay. Okay. We run our program one more time.
- 7:17:20My first question is hello.
- 7:17:22What is the capital of France?
- 7:17:26Okay, now I want to terminate the program.
- 7:17:29I run it one more time.
- 7:17:32And now if I ask about its president and it answers correctly, it
- 7:17:37means that we have successfully saved the data in our database.
- 7:17:42So I ask who is
- 7:17:44its president?
- 7:17:48Okay, it answers correctly and it means we simply
- 7:17:51and successfully implemented this feature.
- 7:17:55And if you want to visually see the tables, you can come here.
- 7:18:00Yes, I want to see tables in public. Public schema.
- 7:18:05Okay.
- 7:18:07yes public and you can see that we have these
- 7:18:10tables these tables are automatically made for
- 7:18:14us by LangGraph itself by this packet that we installed
- 7:18:19Oh
- 7:18:25Okay, there is one more thing I'd like to show you.
- 7:18:28What if we want to access the conversation messages ourselves?
- 7:18:31So far, LangGraph has been retrieving the conversation
- 7:18:34history and making it available to the model automatically.
- 7:18:38But in many applications, we may need direct access to that history.
- 7:18:42For example, we might want to display previous
- 7:18:45messages to the user in a chat application.
- 7:18:49So we need that and the good news is we don't
- 7:18:52need even query the database and we can use
- 7:18:55the feature that LangGraph offers to do that.
- 7:18:58I mean the agent we have already created provides a
- 7:19:02method called getState and if we pass it the thread ID,
- 7:19:07we can get access to the conversation.
- 7:19:10So let's
- 7:19:11do that after this while loop when our program terminates.
- 7:19:17I want to print the messages. So first I need config.
- 7:19:21So let's copy.
- 7:19:25config from here. Oh, we are in the middle of our while loop.
- 7:19:30OK, here I copy it.
- 7:19:32and after the while loop
- 7:19:36I'm going to have this line
- 7:19:38and then I want the messages in session one. Okay.
- 7:19:42We specify the thread ID.
- 7:19:44And then as I told you, the agent we created offers a method called getState.
- 7:19:51and we pass our config here let's call it config history
- 7:19:57and I store it in a variable called state because it is a state.
- 7:20:02And then I say state.values and I want to access to messages.
- 7:20:08It's a dictionary.
- 7:20:11I want to print it.
- 7:20:14Now let's run our program.
- 7:20:17I just say hello and
- 7:20:19I quit okay and as you can see all the messages from this
- 7:20:24session which was hello and the last session are here we asked about friends
- 7:20:29and its president and now we said hello and you can
- 7:20:32see this message here okay that's it
- 7:20:36Before we finish this session and this project,
- 7:20:38there is a useful thing that I want to tell you.
- 7:20:42In our project, we use two conversation IDs, session 1 and session 2.
- 7:20:47That's perfectly fine for this educational example, for this
- 7:20:51demonstration, but you know that in real world applications,
- 7:20:55we often see something else.
- 7:20:57For example, when you are working with an application like
- 7:21:01chatgpt when you look at the url you will see
- 7:21:05a long string of numbers and characters
- 7:21:08this is the best practice that is used in real
- 7:21:11world applications we need a longer identifier
- 7:21:14and one common option to have something like
- 7:21:18this is using uuid which stands for universally
- 7:21:22unique identifier and generating a uid in python is pretty straightforward
- 7:21:28We don't even need to install an additional package
- 7:21:32because Python already provides a module called UUID.
- 7:21:36Let's see that. I say import UUid.
- 7:21:39Now, in order to show you at the beginning of my
- 7:21:42main function, I'm going to generate a UU ID.
- 7:21:45I print, I say uuid, uuid4, and I need to convert it into a string.
- 7:21:53I say...
- 7:21:54str built-in function and then i return here i don't
- 7:21:57want the rest of my function to be executed so
- 7:22:01i run my program and as you can see a uid is
- 7:22:04generated so in your real world applications
- 7:22:08use something like this welcome back aboard neon
- 7:22:11learned planet memory is conquered seven worlds down
- 7:22:15three to go
- 7:22:17Fasten your seat belts. Next stop, Planet Middleware.
- 7:22:21Here, you'll use middleware to keep a human in the loop
- 7:22:24so your agent asks for permission before it acts.
- 7:22:29Your next world awaits. Enjoy the journey.
- 7:22:38Now let's move on to another very important
- 7:22:41component in Langchain which is middleware.
- 7:22:44We can either create our own middleware or we can use some of the
- 7:22:48built-in middleware implementations that Langchain provides.
- 7:22:52But before we start writing code, before we go any
- 7:22:56further, let's understand why middleware exists.
- 7:23:00Remember when I explained that
- 7:23:02we can use create agent to build an AI agent
- 7:23:05without having to deal directly with all the
- 7:23:08complexity of lang graph.
- 7:23:10The idea is that create agent already handles many of the complexity, many of
- 7:23:15the things we need to handle when building an AI Agent.
- 7:23:19But what happens when it covers most of our
- 7:23:22requirements, but we still need a few additional
- 7:23:25customization.
- 7:23:27Middleware gives us exactly that customization.
- 7:23:30It gives us a way to customize an agent's behavior
- 7:23:34without having to implement everything from scratch.
- 7:23:37The main question is that where does middleware fit into the agent life cycle?
- 7:23:43To answer this question, we first need to understand
- 7:23:46what is the life cycle of an agent?
- 7:23:49Let me try to explain it by saying that so far we
- 7:23:52have been using CreateAgent to build our agents.
- 7:23:55As we discussed earlier, an agent typically sends messages to an LLM.
- 7:23:59If the model requests a tool call, the application executes
- 7:24:04that tool and sends the result back to the model.
- 7:24:07And the model can then continue processing and eventually produce a final response.
- 7:24:12And there are different stages in this life cycle.
- 7:24:16there are different points, more accurate to say.
- 7:24:19Middleware allows us to intervene at specific points
- 7:24:23in this lifecycle and customize what happens.
- 7:24:26For example, we can define a middleware that runs
- 7:24:30before the agent starts processing a request.
- 7:24:33We can also define middleware, that runs, before the model is called.
- 7:24:37We can go even further by customizing the model calling process itself.
- 7:24:41For example, imagine that a connection to our model fails.
- 7:24:46We could implement custom logic to retry the
- 7:24:48request up to five times before giving up.
- 7:24:51This gives us more control over how model calls are handled.
- 7:24:55We can also run middleware after the model response and
- 7:25:00at any stage of this life cycle that I explained.
- 7:25:04Okay, so... The key idea is that middleware gives us some hooks
- 7:25:09and we can use these hooks to customize the behavior of this life cycle.
- 7:25:16Let's see where middleware lives and how we can import it.
- 7:25:20For example, let's say we want to take an action
- 7:25:24before the life cycle of an agent starts
- 7:25:27and before when the model is going to be called.
- 7:25:31For these two points, we can use two middlewares, before agent and before model.
- 7:25:37so to access these things we can use i mean we can
- 7:25:41go to lang chain agents and middleware and when i
- 7:25:45say import there are lots of middlewares that you
- 7:25:48can import for example before agent before model
- 7:25:51and these are two middlewares that i was going to show you and these
- 7:25:55are basically decorators and decorators let us define some functions
- 7:26:01It's better to say that they allow us to turn functions to something else.
- 7:26:05For example, we had a decorator for converting a function into a tool.
- 7:26:10And now we have these decorators to turn a function into a middleware.
- 7:26:15So after this, we can define a function.
- 7:26:18For example we can find a function called log before model.
- 7:26:22And it takes state and runtime. Don't worry about these.
- 7:26:26These are related to blank graph.
- 7:26:29And the only thing that I want to tell you is that by using these decorators,
- 7:26:34you can turn a regular function into a middleware.
- 7:26:37Now, the next question is that how do we register it with our agent?
- 7:26:42It's really simple. There is an argument here called middleware,
- 7:26:46and it's a list like the list that we had for tools.
- 7:26:50And you can put your middlewares here.
- 7:26:52And this way we can have a customized middleware
- 7:26:55I'm going to show you something really exciting.
- 7:26:58Langchain provides a large number of ready-made
- 7:27:01middlewares that you can add them to your application.
- 7:27:05And by doing that, you can add a lot of capabilities to your app.
- 7:27:09The exciting part is that you don't have to write any code.
- 7:27:12You just use these middlewares.
- 7:27:14To number some of these useful middlewares, I
- 7:27:17can refer to a summarization middleware.
- 7:27:20Remember our earlier chatbot project when we
- 7:27:23discussed summarizing conversation history.
- 7:27:25As a conversation gets longer, its history can become very large.
- 7:27:29We had this problem and that means more tokens may be sent to the model.
- 7:27:33It leads to more cost and the other thing is that we may fill our context window.
- 7:27:40So that's really bad and in that project I showed you
- 7:27:43a strategy to summarize your conversation history
- 7:27:47We kept the most recent messages, for example, 10 most recent
- 7:27:51messages, and we summarized early messages using an LLM.
- 7:27:56As you saw, implementing something like that introduced additional complexity,
- 7:28:02and we needed to track the conversation, decide when summarization should happen.
- 7:28:07And we considered lots of other things and many other edge cases
- 7:28:11that we didn't take into account.
- 7:28:13However, with this middleware, I mean with summarization middleware
- 7:28:17that LangChain provides, things become much more simpler.
- 7:28:21We just use this middlewares and we will do that in the next session.
- 7:28:25Another useful option, another useful middleware
- 7:28:28is human in loop, human in the loop middleware.
- 7:28:31This is for scenarios that we need human approval.
- 7:28:34Imagine that you're building an AI agent that can send emails.
- 7:28:39and you might want the agent to prepare an email draft
- 7:28:43but you don't want it to send the email without your approval
- 7:28:46so you can introduce a human approval step by using this middleware
- 7:28:51and you can review the email before it proceeds
- 7:28:56so this is particularly useful in general in a
- 7:28:59situation that actions have real-world consequences
- 7:29:03and another useful middleware is PII
- 7:29:07and it stands for personally identifiable information
- 7:29:12and this includes information that can identify a person
- 7:29:16such as certain personal identifiers or contact details
- 7:29:20imagine that an employee accidentally includes sensitive information in a prompt
- 7:29:25and this information is about to be sent to an external model provider
- 7:29:30we might want to detect that information
- 7:29:33mask it before the request reaches the model
- 7:29:36PII can help us implement this kind of behavior.
- 7:29:39Another useful middleware can be model fallback.
- 7:29:43Imagine that we are using Groq as our primary
- 7:29:45model, but we have also configured a model from
- 7:29:48another provider like OpenAI.
- 7:29:51And our goal is that if the primary model
- 7:29:54becomes unavailable or it fails, we may want
- 7:29:57our application to try the other model.
- 7:30:01Model fallback middleware can help us implement this behavior.
- 7:30:06If a model fails, it automatically
- 7:30:08switches to another model.
- 7:30:10Now, after introducing these useful middlewares, in the next session,
- 7:30:15we will see a summarization middleware in action
- 7:30:18and you will learn how to use a middleware
- 7:30:21in your Langchain application.
- 7:30:31all right as i told you in this session we are
- 7:30:34going to use a built-in middleware called
- 7:30:36summarization middleware first we need to import it so i say from lang chain
- 7:30:42agents middleware import summarization
- 7:30:45middleware
- 7:30:47this is a middleware that we are going to use and this is inherently a class
- 7:30:51so we need to create an instance
- 7:30:55and the question is that where should we put this middleware
- 7:30:58we should put it inside this middle-ware argument list that I showed you
- 7:31:02so this is the list and I put it here
- 7:31:06okay I create an Instance
- 7:31:08i need to pass some arguments the first argument that we need to
- 7:31:12pass is the model the model that we are going
- 7:31:15to use to summarize our messages so here i say
- 7:31:19model self.llm okay and the next two arguments are
- 7:31:24max tokens before summary and messages to keep
- 7:31:29and these two arguments specify how many
- 7:31:33messages we want to keep unchanged and what
- 7:31:37is
- 7:31:37are threshold before summarization after how many tokens we want to summarize.
- 7:31:43But these two arguments are deprecated. Let me show you.
- 7:31:47Max tokens before summary and messages to keep.
- 7:31:53These two arguments recently deprecate it and honestly before recording this session
- 7:31:58I realized that.
- 7:31:59And now the new way to introduce these two values are trigger and keep.
- 7:32:06I say trigger and I set it to a top hole. The first element is tokens.
- 7:32:12I set to 500. And honestly, 500 is a very small value
- 7:32:17because in real world applications, we want a much higher threshold.
- 7:32:21But to make our test quick, to trigger summarization quickly and see the result,
- 7:32:27I want to set it a very, very small, value.
- 7:32:31And the next one is keep the number of messages we want to keep.
- 7:32:35again it is a tuple the first element is messages
- 7:32:39and the next one is the number of messages we want
- 7:32:43to keep okay these are the settings that we need
- 7:32:46and in order to have a different session let's set
- 7:32:50active session to for example three it should be
- 7:32:54okay and here again i set it to three to have
- 7:32:58a clean conversation history now let's run our
- 7:33:02program and test it okay here i say hello
- 7:33:06What is the capital of France?
- 7:33:11Who is its president?
- 7:33:14and tell me a story
- 7:33:20till another one.
- 7:33:23Another.
- 7:33:25I want to make sure that I will hit this number 500 tokens
- 7:33:30Tell me a longer one about its dishes.
- 7:33:36okay now let's quit and if we inspect the messages
- 7:33:42I think we didn't have summarization
- 7:33:47Okay, let's test it one more time. I can reduce this number.
- 7:33:53I set it to just 100 and I change it to 4 and this one also to 4
- 7:34:01okay let's run our program one more time
- 7:34:04Hello.
- 7:34:06What is the capital of France?
- 7:34:10Tell me a story about France...
- 7:34:15Another story.
- 7:34:17again and
- 7:34:18let's terminate the program now i'm sure that we have summarization
- 7:34:24okay this is the point that we terminated the
- 7:34:27program and as you can see it says here's
- 7:34:30a summary of the conversation so we had summarization
- 7:34:33successfully and we could implement summarization
- 7:34:36as simple as you see with this summarization middleware welcome
- 7:34:41back aboard neon learn
- 7:34:44Planet Middleware is conquered, eight worlds down, two to go.
- 7:34:49Fasten your seat belts. Next stop, Planet LangGraph.
- 7:34:52Here, you'll build an agent with LangGraph that grades its
- 7:34:56own work and keeps improving it until it scores above 7.
- 7:35:00Your next world awaits. Enjoy the journey.
- 7:35:10Alright?
- 7:35:11Throughout this course, you have heard the name LangGraph many times.
- 7:35:15I've mentioned several times the create agent function we
- 7:35:18have been working with uses LangGraph behind the scenes.
- 7:35:22And starting from this point, we are going to take a closer look at LangGraph.
- 7:35:27So the question is, what exactly is LangGraph? The name gives a pretty good hint.
- 7:35:32Instead of building just a chain, we're going to build a graph.
- 7:35:36And the question again is, What are the fundamental building blocks of a graph?
- 7:35:41And the answer is nodes and edges We have a set of nodes
- 7:35:45and these nodes are connected to each other through edges
- 7:35:49Inside LangGraph nodes define actions Actions we want to perform
- 7:35:54For example, a node might make an LLM call
- 7:35:57run a query against a database or execute a Python function
- 7:36:01And what about edges?
- 7:36:03Edges define where we go next In other words
- 7:36:06After an action finishes, an edge determines which node should execute next.
- 7:36:11Okay, and there is another important component, state.
- 7:36:16State is the data that flows through our graph and it is available to its nodes.
- 7:36:21For example, if you have three nodes, all
- 7:36:24three nodes can access the same graph state
- 7:36:27and they can read the data in that state and they also return updates to it.
- 7:36:33So in other words, nodes don't just read state. They can also modify it.
- 7:36:38Modify it by returning updates to its fields.
- 7:36:41Now that we understand these three important components,
- 7:36:45I mean nodes, edges and state, let's see how they work in code.
- 7:36:50Okay, so in our project, I add a new file called graph.py.
- 7:36:56OK, and we want to write our code here.
- 7:36:59As I mentioned, we have three fundamental building blocks. Nodes, edges, and state.
- 7:37:04And when you are building a graph in the first
- 7:37:07place, you need to define the shape of your state.
- 7:37:11For that let's consider an example.
- 7:37:13Suppose I want to use AI to write an essay.
- 7:37:17I'll ask LLM number 1 to write the essay. Then I'll send that essay to LLM number 2.
- 7:37:23And the LLM number 2 will evaluate it and give it a score. A score from 1 to 10.
- 7:37:30And if this score is greater than 7, we are done.
- 7:37:34otherwise we will send the essay back to the first LLM number one
- 7:37:39and ask it to improve it and this way we have a graph
- 7:37:42so in this example that is the workflow we are going to build
- 7:37:46now the question is what information do we need to keep track of
- 7:37:50first we need the essays topic the topic that we want to write about
- 7:37:54next we want the content it is a string and initially this field is empty
- 7:38:00and the first llm will fill it
- 7:38:03Finally, we need score, which will hold the
- 7:38:05valuation score and we will use a floating point
- 7:38:08number and initially it is also empty.
- 7:38:11So we need a data structure that can hold these fields, these key value pairs.
- 7:38:17And when you hear key value pair, you can think of a Python dictionary.
- 7:38:21Yes, we could use a regular Python dictionary, the point is that however,
- 7:38:26a regular dictionary doesn't explicitly describe the expected types of its value
- 7:38:32so it's a better option here to use type dict type
- 7:38:36dict is available in python
- 7:38:38and you don't need to install an additional package
- 7:38:40and it lets you define the expected keys and values like a dictionary
- 7:38:45and the ultimate result will be dictionary when you want to create an object of it
- 7:38:50so let's do that I say from typing import typed dict.
- 7:38:56OK. And now we need to create a class. I name it Essay.
- 7:39:02And this class should inherit from this typed dict class.
- 7:39:07something like Pydantic that we had and as I told
- 7:39:11you we have three fields topic content and score
- 7:39:15topic is string and we need to specify that content
- 7:39:19is again string but it can also be none and at
- 7:39:23first
- 7:39:24it is none because when we execute our graph it
- 7:39:27doesn't have any value and the same to a score
- 7:39:30and as i told you it's a class and we can create an
- 7:39:34instance of it okay so i say essay and i need to
- 7:39:38specify these keys topic content and score for example topic can be this is a topic
- 7:39:46Okay, content, it is a string. This is content.
- 7:39:51And we have a score, which is a floating point number.
- 7:39:53Oh, I set it to a string, I say float. Okay, 2.2 can be good.
- 7:40:00Okay, and the interesting part is that, as I told you, the
- 7:40:05resulting object is still an ordinary Python dictionary.
- 7:40:09So let me show you. I store it in a variable called SA, and then I print it.
- 7:40:15I print this essay, to make sure, I also print its type.
- 7:40:23and if I run my program as you can see it's
- 7:40:25a regular dictionary and this is a regular
- 7:40:28dictionary that you're seeing here okay great
- 7:40:31so I'll remove these print statements and our
- 7:40:35first step is complete we have defined the
- 7:40:38state and you need to pay attention here the
- 7:40:41essay class you're saying here describes the structure of our state
- 7:40:46while this essay you're saying here is a dictionary
- 7:40:50instance and it contains the actual data
- 7:40:53so it is what that is going to flow in our graph
- 7:40:57it is only the blueprint kind of now in
- 7:41:00the next step we need to create our nodes
- 7:41:04remember a node represents an action something we want to do as part of our workflow
- 7:41:09and what's a natural way to represent an action in python a function
- 7:41:14So we'll start by defining two functions for two nodes.
- 7:41:18One function to write the essay and another to score it.
- 7:41:23So we are going to have two functions. The first function is write essay.
- 7:41:29SA and its type is SA.
- 7:41:31So it refers to its structure, its shape, and it is the name
- 7:41:36of the parameter that this function is going to receive.
- 7:41:40And for now, we are going to use simple dummy implementation.
- 7:41:45Later, after learning the core concepts of langGraph, we
- 7:41:49will replace these dummy functions with real LLM calls.
- 7:41:53So here's my first function.
- 7:41:55As I told you, this function takes the current essay state
- 7:41:59as input and returns an update for the content field.
- 7:42:02So here I say return. I want to update content.
- 7:42:08and I say, for example, the essay is about the topic.
- 7:42:13placeholder here to dynamically inject this field here.
- 7:42:18Say SA, which is a dictionary and I want its topic. Okay,
- 7:42:23pretty good.
- 7:42:24Now let's define the other function, which is a scoreSA.
- 7:42:30we only want to update the score of this state
- 7:42:34okay so I set it to for example 10
- 7:42:38Here's the important point you're seeing here.
- 7:42:41A node only needs to return the fields it wants to update.
- 7:42:45So here we are not returning content or topic
- 7:42:48because we only want to update the score.
- 7:42:52And that's the point about LangGraph.
- 7:42:55Now we have our two node functions and we have already defined our state.
- 7:43:01Now at this point, we are ready to start building the graph.
- 7:43:04So first, let's import state graph from blank chain. Here I say from land graph.
- 7:43:10Graph import.
- 7:43:12state graph okay we need
- 7:43:14the state graph and it allows us to define a graph around a particular
- 7:43:20state schema and we'll pass our essay class to its
- 7:43:24constructor and then it gives us a builder so
- 7:43:27let's do that
- 7:43:29here i use this state graph
- 7:43:31and i pass my schema the shape of our state and it gives me a builder object
- 7:43:38okay and now we can use this builder object to
- 7:43:42register our nodes define the edges and configure
- 7:43:45the graphs workflow and once we are finished once we
- 7:43:49have finished defining the graph we can compile it
- 7:43:53so first let's add our nodes so i say builder i use the add node method
- 7:43:59And I add WriteSA as my first node and ScoreSA as a second node.
- 7:44:06and the order doesn't matter here
- 7:44:08notice that we are passing the functions themselves not calling them here
- 7:44:12i don't put parentheses
- 7:44:15pay attention here and when we add a function this way
- 7:44:19lang graph uses the functions name as a nodes name
- 7:44:22so our nodes will be called writeSA and scoreSA if you wanted a different name
- 7:44:28you could pass a first argument here defining the name that you preferred okay
- 7:44:34now it's time to define the edges for our initial workflow
- 7:44:39we want the graph to execute these steps in order start write
- 7:44:44SA score SA okay and the start node is a special
- 7:44:49marker that represents where execution begins and
- 7:44:53we don't need to implement it ourselves we just
- 7:44:56import it so here in addition to state graph i
- 7:44:59import start node
- 7:45:01and it can play the role of starting out for us so here after having
- 7:45:07these nodes i can define edges and we do it using the add edge
- 7:45:12method so here i say builder.add edge and then you will specify the
- 7:45:18nodes that you want to connect to each other so i want to
- 7:45:22connect my start node to the write-say node and pay attention you should
- 7:45:28put the name of your node inside quotations. So I say writeSA.
- 7:45:33And then we need an edge between writeSA and scoreSA. So let me duplicate this line.
- 7:45:39Here I say WriteSA and here I say scoreSA
- 7:45:45okay that's it and these are my edges and as i told
- 7:45:49you the final step is to compile it to compile
- 7:45:53the graph that we shaped with this builder object so i say builder.compile and i
- 7:46:00store the result in a variable called graph and this graph object that we have here
- 7:46:06has a runnable interface it's a runable object so we can call the invoke method
- 7:46:13I say graph.invoke
- 7:46:16and then we pass an object and we specify the
- 7:46:20values of our state so here i say topic topic is
- 7:46:25a
- 7:46:25field that should be specified by us so i say programming but
- 7:46:30content and score are are valued by these two functions
- 7:46:35So the initial value for them is none. Say content none.
- 7:46:41and score none as well okay and
- 7:46:44I don't need this line it was just for demonstration
- 7:46:49And now it returns a state.
- 7:46:52I store it in a variable called state and then I print it.
- 7:46:56Okay, notice that here we are passing a state, a regular Python dictionary.
- 7:47:01And I told you that we could also construct an essay using, I mean, by
- 7:47:07creating an object of this class, but the result is not different.
- 7:47:12So here I'm passing an s state and as this state goes through
- 7:47:17this graph, these fields are updated and then I will get a
- 7:47:21final state and then, I'm printing this final state.
- 7:47:25OK, let's execute our program.
- 7:47:29And as you can see, these fields are updated.
- 7:47:32And this was the core idea of building a graph in LangGraph.
- 7:47:38Ladies and gentlemen, this is a message from your captain.
- 7:47:43We've now reached planet LangGraph
- 7:47:45and your captain wants you to know he's truly proud of you.
- 7:47:48You've stayed with us all the way here and you never gave up.
- 7:47:52This journey has been made many times, but very
- 7:47:55few passengers have ever made it this far.
- 7:47:59Your captain is certain you'll reach your destination.
- 7:48:01You'll become an AI engineer and land a great job at a great company.
- 7:48:07Please have your passports ready. Your AI Engineering Visa is about to be stamped.
- 7:48:21One of the most important capabilities of AI agents is decision making.
- 7:48:26An AI agent needs to decide what action to take next.
- 7:48:30But what exactly do we mean by decision making?
- 7:48:33For example, if condition A is true, go to
- 7:48:36one node, otherwise go to a different node.
- 7:48:39This is essentially the same if-else logic we already have from regular programming.
- 7:48:44And in LangGraph, we can implement this behavior using conditional edges.
- 7:48:49for this session let's start with a very simple example
- 7:48:53first here i'm going to remove the previous
- 7:48:56code from this file we want to have an example
- 7:48:59but before explaining the example let's import
- 7:49:02things that we need for example first let's
- 7:49:05important our state graph so i say we want
- 7:49:08to build a graph we say lang graph graph import state graph
- 7:49:13And what does the state graph expect as input? It expects the schema as you saw.
- 7:49:18And in previous example, the state represented an essay
- 7:49:22and it contained fields such as topic, content, and score.
- 7:49:27But in this example, we are going to keep things much simpler.
- 7:49:31Our state will be a simple integer number. So we are gonna have something like this.
- 7:49:36I'm going to say state graph and its state is just a regular integer. That's it.
- 7:49:43And I store it in a variable called builder and
- 7:49:46another very common name is also workflow.
- 7:49:50So here in this example, I use Workflow.
- 7:49:54as i told you each node represents an action we are going to create two nodes we are
- 7:49:59going to have two actions the first node will
- 7:50:03multiply the current state by two the second node
- 7:50:06will add one to the current the state so let's
- 7:50:09implement that here i have these two functions
- 7:50:13function add it receives state and we can specify type hint or not okay
- 7:50:20and it adds one to the current state
- 7:50:24as it's a regular integer we don't need these curly braces
- 7:50:28and we have the multiply function
- 7:50:30okay and it's going to multiply the current state by 2.
- 7:50:35that's it now let's decide how our graph should behave
- 7:50:39this is the behavior that i expect we have a starting point we have
- 7:50:43the multiplication node and then the add node and we
- 7:50:48have a conditional decision and our condition is that
- 7:50:52If the current state is greater than 10, we are done.
- 7:50:55Otherwise, we want to go back to the multiply node.
- 7:50:59So when we start, we multiply the state by two,
- 7:51:03we add one, and then we check if the current
- 7:51:07state is, greater than ten, we terminate.
- 7:51:10And if it's not greater than, ten, We want to jump back to, to the multiply node
- 7:51:16and now the important question is how do we implement this conditional decision
- 7:51:21just like our other nodes we can define a regular python function here
- 7:51:25so let's define another function called should continue
- 7:51:31receives the state and the function returns
- 7:51:33the name of the destination node as a string
- 7:51:36whenever we want to jump okay whenever we
- 7:51:39want to specify the next node and we have two
- 7:51:42destinations either we have the end node
- 7:51:46or the multiply node and if the state is greater than 10 we want
- 7:51:51to terminate we want return the end note otherwise we want
- 7:51:54to go to the multiply note so here i say return end
- 7:51:59if state is greater than 10 otherwise
- 7:52:04We want to go to the multiply node
- 7:52:07We don't have this end here. We
- 7:52:09need to import it. So here I say.
- 7:52:13it's like the start node that you saw in the last
- 7:52:17session okay that's it and now we have defined our
- 7:52:20node functions we have to find our conditional logic
- 7:52:24now we can build our graph so here i can add
- 7:52:27the two nodes i mean multiply and add so i say workflow add node
- 7:52:33multiply okay
- 7:52:34as well as add node
- 7:52:37and here we need a conditional node we need
- 7:52:40a conditional note after the add node but before that let's establish an edge
- 7:52:45between multiply and add so i say
- 7:52:49Edge.
- 7:52:50multiply and add
- 7:52:53okay and then we need a conditional edge and for
- 7:52:56establishing conditional edges you shouldn't use
- 7:52:59this add edge method you need to use its
- 7:53:02specific method which is add conditional edges
- 7:53:05so here i say workflow add conditional ages i want to connect add to shoot continue
- 7:53:13In other words, it's saying that execute should continue after the add node
- 7:53:18and use its return value to determine where execution should go next.
- 7:53:24If the function returns end, the graph stops.
- 7:53:27If it returns multiply, it goes back to the multiply node.
- 7:53:32and the final step is to set the entry point
- 7:53:35in the previous lesson we used the start node
- 7:53:38but there is another approach and in this session we're going to use that approach
- 7:53:43that approach is using the set entry point method
- 7:53:47so I say set entrypoint
- 7:53:51and I want this multiply node to be my first node this
- 7:53:55is my first note
- 7:53:57and now everything is in place we can compile it
- 7:54:01So I say compile and then I store the result in a variable called
- 7:54:08This is another common name when you compile your workflow or your graph.
- 7:54:13And then you want to invoke it.
- 7:54:15I invoke it and my initial state is 1 and then invoke.
- 7:54:22forgotten book.
- 7:54:23Okay, then I print it.
- 7:54:25Okay, so here's
- 7:54:26my initial state.
- 7:54:28It goes through my graph and then I'll print the final result.
- 7:54:33If I execute it, you can see 15 here.
- 7:54:37Let's take a closer look to see why we see 15.
- 7:54:42First we have one.
- 7:54:44and it reaches the first node which is the multiply node
- 7:54:48and it multiplies the state by 2 so we will have 2
- 7:54:53the next node which the add node adds 1 to it so it becomes 3
- 7:54:59then we have the conditional node is 3 greater than 10?
- 7:55:04it's not so we go back to our multiply node so it is multiplied by 2
- 7:55:10it becomes 6
- 7:55:12and the next node it becomes 7 and we have this conditional node
- 7:55:17is 7 greater than 10? no it's not so it returns to multiply node
- 7:55:22so 7 is multiplied by 2 it becomes 14 and then 1 is added to it and it becomes 15
- 7:55:31now when we go to this node to this conditional note 15 is greater than ten
- 7:55:38So it is over and 15 is printed on the terminal.
- 7:55:43you
- 7:55:49Okay, now that you have learned all the fundamental
- 7:55:52concepts we need to build the graph, we are
- 7:55:54ready to move on to a more realistic project,
- 7:55:57which is using LLMs to write an essay, evaluate
- 7:56:01it, and revise it based on feedback.
- 7:56:04And this is going to be a really useful project
- 7:56:07because it brings together many of the concepts
- 7:56:10we have covered throughout the course.
- 7:56:12First, let's load our environment variables
- 7:56:15because we are going to have an LLM call.
- 7:56:18We are going contact with the provider. So import.env.
- 7:56:23and then i load my environmental variables using the dot env method so i say dot env
- 7:56:29load dot env and this way i have my environment
- 7:56:33variables now let's import type dict so i
- 7:56:37say from typing import typed dict and our
- 7:56:41state is going to contain several keys we have
- 7:56:46a topic we have a draft and we have score and we also have a feedback because the
- 7:56:51The first LLM needs to have a feedback to improve itself.
- 7:56:56Then we need our provider and we are going to use the Langchain integration.
- 7:57:00So I say from Langchain Groq import ChatGroq.
- 7:57:06And finally, we need to import things that are necessary to build our graph.
- 7:57:10So I say from langgraph.graph import end start and stateGraph
- 7:57:19We will use StateGraph to build a graph start to define the
- 7:57:23starting point and end to stop the workflow when we are done.
- 7:57:27And then we need define the shape of our graph, the shape our state.
- 7:57:32So I create a class called content state and it inherits from type dict
- 7:57:39And then we have our fields. We have topic, which is a string.
- 7:57:43And we have draft, which is a String again. And we've score, it is an integer.
- 7:57:50And we our feedback, it is a String again. And we have revision count.
- 7:57:56Revision count.
- 7:57:59which is a integer number. So first here we have topic.
- 7:58:04This is a topic we want the essay to cover and we
- 7:58:07will specify it ourselves when we run the graph.
- 7:58:10For example, we might ask for an essay about
- 7:58:13health or explain why sleep matters for learning.
- 7:58:16Next we have our draft and this field contains the most
- 7:58:20recent version of the essay written by our first LLM.
- 7:58:24Then here we had a score.
- 7:58:27integer number representing the score assigned by our second LLM.
- 7:58:31And we also need a feedback. This field contains the reviewer's feedback,
- 7:58:37the second LLLM's feedback explaining what needs to be improved.
- 7:58:41For example, the review might say the writing is unclear, or it needs more evidence,
- 7:58:48or it need more examples and things like this.
- 7:58:51And the first LL will use this feedback to improve the next version of the essay.
- 7:58:56And finally, we have Revision Count.
- 7:58:59This tracks how many times the essay has been revised.
- 7:59:03And this field is especially important because we need to
- 7:59:06prevent our graph from getting stuck in an infinite loop.
- 7:59:10Imagine that our goal is to achieve a score greater than, for example, 9.
- 7:59:15And the essay might never reach that score or
- 7:59:18it might take many iterations to get there.
- 7:59:21and without having stopping condition, our graph could keep running indefinitely.
- 7:59:27We will solve this problem by adding a guard and for that guard, we need this flag.
- 7:59:32Then we need to initialize our LLM. So let's borrow some code from here.
- 7:59:38Yeah, that's exactly what we need.
- 7:59:41Okay.
- 7:59:42So here I'm going to define my LLM. It is a global variable here and I'm
- 7:59:47going to use this model, but a higher temperature is better. And to clarify, I'm
- 7:59:53going to the same model for both writing and reviewing the essay.
- 7:59:58But you could have two LLMs.
- 8:00:00This way it can be your first LL and it could be your second LL.
- 8:00:05And you can use another model. Okay. It is as simple as that.
- 8:00:09So it's not a big deal.
- 8:00:11And honestly, in a real-world workflow, using different models can be better
- 8:00:15because a model might be biased toward its own writing.
- 8:00:20Now, let's identify the components of our graph.
- 8:00:24We need a node that writes the essay, another node that reviews it,
- 8:00:28and a conditional function that decides whether
- 8:00:31the essay needs another revision or not.
- 8:00:34So, let us define them. Let's say dev write draft.
- 8:00:38gets a state.
- 8:00:41and this way with this type hint, we specify
- 8:00:44its type, which is going to be content state.
- 8:00:47Okay.
- 8:00:48Currently, we don't have any implementation yet
- 8:00:51and I need two other functions, review draft
- 8:00:54and the third one should continue
- 8:00:58or should revise. Okay.
- 8:00:59These ones also don't happen implementation yet.
- 8:01:03Before implementing these functions, first we want to set up the graph itself. So.
- 8:01:08To do that, I use state graph that I imported. I give it the shape of my state,
- 8:01:14and then I store it in a variable called Builder.
- 8:01:18Now let's register our nodes So I say Builder Add node
- 8:01:24Okay, I say Write draft
- 8:01:27And again, review draft.
- 8:01:30and as i told you you can specify the name of these
- 8:01:33nodes yourself for example let's do that so you can
- 8:01:36see and here the name our function and the name
- 8:01:40of our node is the same you could use different names
- 8:01:44as well and next we need to specify our edges
- 8:01:47first i'm going to connect the starting node to
- 8:01:51the right draft node so i say builder add edge
- 8:01:55start and this node, write draft.
- 8:01:58And then after this node after write draft, we need to move to the review node.
- 8:02:03So let's duplicate it. I'm going to go from write draft to review draft.
- 8:02:09to a word typo let's copy paste these things after
- 8:02:12the review node finishes we need to decide
- 8:02:15what happens next should we stop or should we
- 8:02:19return to the writing node for another revision
- 8:02:22and we need handle it using a conditional edge
- 8:02:26so here i say builder add conditional edges
- 8:02:30and here i'm going to connect my review draft
- 8:02:34and then after it i need to put this function should revise
- 8:02:39this function is executed after our review draft node
- 8:02:43and finally let's compile it i say builder compile
- 8:02:48and I store the result in a variable called graph.
- 8:02:52our graph structure is now in place we just need
- 8:02:55to implement the logic inside the functions but
- 8:02:58again before implementing the functions let's see how
- 8:03:01we'll eventually run our graph i mean i want to
- 8:03:04pass an initial state containing the topic
- 8:03:07and initial values for other fields so here
- 8:03:11i'm going to invoke this graph because
- 8:03:13it is a runnable object we can invoke it
- 8:03:17And then we provide our state here. And I think we can borrow some code from here.
- 8:03:23Yes, let's do that.
- 8:03:26I paste it here, but these fields needs to be inside quotations.
- 8:03:32and the topic is going to be why sleep matters for learning and then
- 8:03:38we need some commas between
- 8:03:41these fields okay now the draft is empty at first the
- 8:03:46score is uh zero we don't have any feedback yet
- 8:03:51and revision count is zero
- 8:03:54Okay, so this is our initial state and once the nodes are implemented,
- 8:03:59the graph will execute the workflow and return the final state.
- 8:04:02So these values will be updated. And then when we have the state,
- 8:04:08we can print the result. I want to print the final draft,
- 8:04:12the number of revisions, and the final score. So here I say print.
- 8:04:18I use a formatted string
- 8:04:20and I say
- 8:04:21final
- 8:04:22draft and then I specify the number of revisions so here I say I access this final
- 8:04:30state and I want this key revision count okay
- 8:04:35and I say revision one revision or several
- 8:04:39revisions and then i want backslash n and let's have another print statement to make
- 8:04:46our code more clear and then I want the draft itself So I say result draft.
- 8:04:53And finally we want the final score. So here I use a formatted string.
- 8:04:58First I go to the next line. Then I say final score, and this is our final score
- 8:05:04I say, result. Okay. Final score.
- 8:05:08okay so the only thing left is to implement the functions that we don't have
- 8:05:14any implementation for them yet let's start with the simplest one
- 8:05:19here we only need a condition our target score is
- 8:05:23eight so if the score is greater than eight we can
- 8:05:27terminate and if it's not we go back to to the right
- 8:05:31draft node we also need another guard that i told you
- 8:05:34if after three revisions we didn't reach the score
- 8:05:38we want to terminate this way we will also enforce
- 8:05:41a maximum of three revisions and this prevents
- 8:05:45an infinite loop okay so let's have it
- 8:05:48First, I add an if statement and I say if our score
- 8:05:53is greater than or equal to 8 we want to terminate and i return the end node
- 8:06:00OK, and another if statement and I say if our revision count.
- 8:06:05is greater than three, greater than or equal to three.
- 8:06:10Again, I want to return.
- 8:06:13And then finally, if neither of these conditions
- 8:06:16is matched, I want to return to the right draft.
- 8:06:20So I say return right draft
- 8:06:24Okay, so that is the implementation for this function.
- 8:06:28Now let's implement review draft.
- 8:06:30The job of this node is to ask the LLM to evaluate the current
- 8:06:35state, assign a score from 1 to 10, and provide a feedback.
- 8:06:40So let's start by providing a prompt. It is going to be my prompt.
- 8:06:44I say I use a formatted string and I use triple quotation.
- 8:06:49And I say rate this paragraph from one to 10 and give one sentence of feedback.
- 8:06:58okay then we need to provide our paragraph so i say paragraph and
- 8:07:03i extract it from my state
- 8:07:06It is our draft and then we specify a guideline for it.
- 8:07:11I say respond in exactly this format.
- 8:07:16I specified the score.
- 8:07:18It is going to be a number.
- 8:07:20And then we want that one sentence feedback.
- 8:07:25fixed.
- 8:07:26So this is going to be our prompt.
- 8:07:29We are explicitly asking the model to return its
- 8:07:32answer in a simple, predictable text format.
- 8:07:34We could also use structured output and we covered that in an early session.
- 8:07:39And that would generally be better, a more robust way to do that.
- 8:07:43But in this example, I just wanted to show you how
- 8:07:47to extract information from plain text as well.
- 8:07:50because you may encounter this scenario and it's good to be familiar with.
- 8:07:56So next we need to send our request.
- 8:08:00So I say LLM invoke and I'm going to send this prompt.
- 8:08:05and I get a response.
- 8:08:07and I only want its text so I say response content it's an AI message and I
- 8:08:14only want it's text field
- 8:08:16and I store it in a text.
- 8:08:19now we need to parse it we need to extract the score and feedback from that text
- 8:08:24and since we asked the model to put the score
- 8:08:27and feedback on separate lines we can split the response into individual lines
- 8:08:32and look for the lines that start with score colon and feedback colon
- 8:08:38so that's really simple and we can do it using a list comprehension
- 8:08:44So first I'm going to split this text into its lines.
- 8:08:48So I'm gonna use the split method.
- 8:08:52and my separator is going to be backslash n to separate lines
- 8:08:57and i'm going to iterate over it so i say for line in these lines
- 8:09:04and then i'm gonna put a condition i say if line start with
- 8:09:11with exactly this thing. In that case, I want you to return that line.
- 8:09:19and I put the whole line between brackets and
- 8:09:23this way I will have a list of one element so
- 8:09:27here in this list I will only have one element I
- 8:09:31extract it now I have a line containing the score
- 8:09:34that I'm looking for so eventually I will something like this
- 8:09:39score blah blah blah
- 8:09:40Okay, I store it into a variable called scoreline.
- 8:09:47Okay, and I do the same thing for feedback. I say feedback line.
- 8:09:53Okay, uppercase F. And now I have two separate lines.
- 8:09:58At the beginning of each line, we have score and feedback.
- 8:10:02So the only thing left is to get rid of these two things.
- 8:10:06I mean, score and feed back.
- 8:10:09and to do that we can use the replace method
- 8:10:13so here first i say scoreline dot replace
- 8:10:17Replace what? Replace this part.
- 8:10:20with empty.
- 8:10:22and then I strip it to get
- 8:10:24rid of any additional space at the beginning and at the end.
- 8:10:29Now I want to convert it into an integer. So I use the built-in integer function.
- 8:10:35And I store the final result in a variable called score.
- 8:10:38And then I'm going to do the same for feedback,
- 8:10:43but I don't need to convert into integer because it's a string.
- 8:10:48okay feedback line here now it is ready and finally we are going to update our state
- 8:10:56we return the fields we want to update score and feedback so here i say return
- 8:11:02I want to update our score field to this score
- 8:11:06that we extracted from the LLM's response
- 8:11:10and the same for feedback.
- 8:11:13okay here we go one thing that is important and you
- 8:11:17need to keep in mind is that this simple parsing
- 8:11:20approach assumes that the model follows the
- 8:11:23requested format exactly if it changes the labels
- 8:11:26or returns an invalid score the code could fail so for this reason i told you that
- 8:11:32it's better to use structured output and it is more robust but you
- 8:11:37Here it is perfectly fine and we are using a relatively
- 8:11:41strong model so it can handle this scenario.
- 8:11:44Now the final function is writeDraft.
- 8:11:46This node is responsible for generating the initial
- 8:11:50essay, also improving it whenever we have a feedback.
- 8:11:54So let's begin with a simple prompt.
- 8:11:57They say,
- 8:11:59write a short paragraph about instead of saying essay
- 8:12:05I'm saying paragraph because I don't want it to be long
- 8:12:09and then we need to provide the topic so I say state topic
- 8:12:15okay I want a paragraph about this topic and this prompt tells the LLM what topic
- 8:12:21what topic to write about but what happens when we are revising an existing draft
- 8:12:27in that case we need to give the model the reviewers feedback. So we need check.
- 8:12:33We need to check if we have a feedback. We want to add it to our original prompt.
- 8:12:37So here I add a condition and I say in case we have feedback.
- 8:12:44we want to add something to our prompt so i say
- 8:12:48prompt plus equal i want to concatenate this new
- 8:12:51string to it i say first i put
- 8:12:56I put double backslash and I say previous feedback to address.
- 8:13:02and then I provided, I say state feedback.
- 8:13:07okay that's all we need if feedback exists we
- 8:13:10append it to the prompt and this gives the model
- 8:13:13additional guideline when generating the next version now let's send the prompt
- 8:13:18and get the response
- 8:13:20Okay, like here.
- 8:13:22And finally, we
- 8:13:24need to update two things. The draft and the revision count field.
- 8:13:29So I say return.
- 8:13:32draft and it is LLM response
- 8:13:36And then we have Revision count.
- 8:13:40Okay, say...
- 8:13:43Revision count plus one. I want to add one to it.
- 8:13:48it was the final function that we implemented now
- 8:13:51every time this node runs the counter increases by
- 8:13:55one and if you have a new draft the draft will
- 8:13:58be updated so let's put everything together and
- 8:14:02run our graph okay let's run it
- 8:14:06Okay, what's the problem? Oh, here we have a type I guess.
- 8:14:11starts with
- 8:14:14Okay, one more time.
- 8:14:18okay everything is working correctly after just one
- 8:14:22revision we have this essay and as you can see
- 8:14:26it is biased and in the first try it got the score score nine so
- 8:14:32let's make it harder i change this number to nine and one more time
- 8:14:37Maybe the next time it gets 8.
- 8:14:40Okay, again after the first time it got 9.
- 8:14:44So let me change the prompt a little bit. I say be very strict.
- 8:14:52Okay, one more time.
- 8:14:54okay this time after two revisions we have this
- 8:14:57essay and this is our final score finally made it
- 8:15:01to get a score more than or equal to nine let's
- 8:15:04change this condition to only show you that we
- 8:15:07won't have an infinite loop i'm going to say
- 8:15:11if it is greater than 11 so it will never
- 8:15:14happen because we are telling model that it
- 8:15:17needs to generate a number between 1 and 10. so let's run our program
- 8:15:23And you will see that after three revisions, we will have this paragraph here.
- 8:15:28And that happens because we intentionally changed this
- 8:15:33condition and we went out of the range that was possible.
- 8:15:38Welcome back aboard Neon Learn.
- 8:15:40Planet Lang graph is conquered, nine worlds down, only one remains.
- 8:15:46Fasten your seat belts. Final stop, Planet Multi-Agent.
- 8:15:49Here you'll build a team of specialist AI agents,
- 8:15:53each one an expert, all working together as one.
- 8:15:58Conquer it and become an AI engineer.
- 8:16:01you
- 8:16:07Alright, by now you are familiar with the concept of an AI agent.
- 8:16:11An AI agent is essentially an LLM that has access to a set of tools.
- 8:16:16Whenever the model needs to use one of those tools, it requests a tool call.
- 8:16:21We execute tool call, provide the result back to the model,
- 8:16:25and the agent continues working based on that result.
- 8:16:29Now in our next project, we are going to explore multi-agent systems.
- 8:16:34So what exactly does that mean?
- 8:16:36Instead of having just one agent, you are going to work with multiple agents.
- 8:16:41For example, imagine we have agent 1 which specialized in a
- 8:16:45particular task and has access to a specific set of tools.
- 8:16:49Then we have Agent 2 which specializes in another
- 8:16:53task and have access to different tools.
- 8:16:56Finally, we have the agent 3 which coordinates the other agents.
- 8:17:01We can think of this agent as our orchestrator, receives a request, decides
- 8:17:06which agent should handle it, and delegates the task accordingly.
- 8:17:11Let me make this concept clearer with an example.
- 8:17:15Imagine that agent 1 has a tool that can reduce the file size of an image.
- 8:17:21Agent 2, on the other hand, has a tools that can improve image quality.
- 8:17:26And
- 8:17:27then we have a decision-making agent. and this agent can coordinate the workflow
- 8:17:33and this is our orchestrator and suppose we receive something like this
- 8:17:38the user asks I want to improve the quality of this image
- 8:17:42in that case it routes the request to agent 2
- 8:17:45because that agent specializes in improving image quality
- 8:17:50Now imagine the user says, I can't upload this file because its size is too large.
- 8:17:56In that case, the orchestrator agent routes a request
- 8:18:00to agent 1, which can reduce the file size.
- 8:18:03So what have we done here?
- 8:18:05Instead of having a single agent with two tools, now we have three agents.
- 8:18:10Three agents working together with two tool.
- 8:18:14At this point, you might be asking yourself a reasonable question.
- 8:18:18why would we use three agents with only two
- 8:18:21tools instead of simply having a single agent
- 8:18:24with two tools it doesn't make sense and yes you are absolutely right
- 8:18:28For this simple example, a single agent with only two tools would be sufficient.
- 8:18:34And I used this example simply to make the concept easier to understand for you.
- 8:18:39So the question is, when do multi-agent systems actually become useful?
- 8:18:43They become useful when we are dealing with a large number of tools.
- 8:18:48In real-world applications, we might have 30, 40, or even 50 tools.
- 8:18:53And many of these tools may have very little to do with one another.
- 8:18:57and if we give all those tools to a single agent
- 8:19:00and also provide it with a very long complicated prompt
- 8:19:04the system can become harder to manage okay and it is less reliable
- 8:19:10I mean the model might struggle to determine which tool it should call
- 8:19:14sometimes it might not call a tool at all
- 8:19:17in other cases it might select the wrong tool
- 8:19:20or fail to follow the intended workflow
- 8:19:24So the idea is that instead of giving every tool to one
- 8:19:28agent, we divide the workload among specialized agents.
- 8:19:32Each agent focuses on specific tasks and has
- 8:19:35access to a relatively small set of tools.
- 8:19:38And this makes each agent's responsibilities clearer
- 8:19:42and can reduce the complexity of the overall system.
- 8:19:46So the goal isn't simply to increase the number of agents.
- 8:19:50The goal is to organize responsibilities.
- 8:19:53And of course, multi-agent systems aren't always the best choice.
- 8:19:58For simple workflows, a single agent might be a better option.
- 8:20:03Now, there are two major approaches that we can implement this concept using them.
- 8:20:09The first one is the supervisor approach and the second one is handoffs approach.
- 8:20:16The first approach, I mean supervisor, is the approach that I discussed.
- 8:20:21And the second approach is called handoffs.
- 8:20:24In this approach, we don't necessarily need a separate central supervisor.
- 8:20:29Instead, agents can cooperate.
- 8:20:32Agents can transfer control to one another when they think it is appropriate.
- 8:20:37Imagine that we have agent 1 and agent 2. Agent 1 starts handling a request, but
- 8:20:44At some point, it determines that Agent 2 is a
- 8:20:47better option, can better handle the situation.
- 8:20:50So Agent 1 can then hand off the conversation
- 8:20:54and the relevant state goes to Agent 2.
- 8:20:58And Agent 2 takes over and continues working from that point.
- 8:21:03So rather than having a central supervisor, we can
- 8:21:07give the ability of controlling the situation
- 8:21:10to the agents themselves. That's the basic idea behind handoffs.
- 8:21:15And in our next project, we are going to focus on the first approach,
- 8:21:19the supervisor architecture,
- 8:21:21because it builds on the concepts you are already familiar with.
- 8:21:26you
- 8:21:32all right now let's implement a multi-agent system using the supervising pattern
- 8:21:38what are we going to build we are going to
- 8:21:40build a very simple project with only two specialized agents
- 8:21:45one agent will be an expert on fruits and the other will be
- 8:21:48an expert and vegetables the fruit expert will have access to a tool that provides
- 8:21:54information about fruits while the vegetable expert will have access a tool
- 8:22:00that provides information about vegetables.
- 8:22:02And as you can see here, I've already prepared these two tools.
- 8:22:06And as we can see, there are just two simple Python functions.
- 8:22:11The first function takes a fruit name and the second one takes a vegetable name.
- 8:22:16And the type hints here are telling you that the
- 8:22:19input parameter is expected to be a string.
- 8:22:22And the function returns a string
- 8:22:25and these are as you can see simple almost dummy tools
- 8:22:30our fruit tool only knows about tomatoes and apples
- 8:22:34while our vegetable tool only know about carrots and potatoes
- 8:22:38and that's it they don't have any additional knowledge
- 8:22:42if we pass a fruit or vegetable that isn't included the function
- 8:22:47returns a message saying that no specific information was found so
- 8:22:52let's build our multi-agent system first
- 8:22:55we need to load our environment variables because
- 8:22:59we have LLM calls so here I import dot-env
- 8:23:03and then I load environment variables then
- 8:23:08I need to import ChatGroq so I say from Langchain Groq import ChatGroq.
- 8:23:15and then we need an llm so let's copy some code from for example here
- 8:23:23Yeah, that's perfectly fine.
- 8:23:25Okay, so it is going to be our LLM.
- 8:23:29Now let's move on to creating our sub-agents.
- 8:23:33So first I import create agent from LangChain.
- 8:23:38Agents, import, create agent.
- 8:23:42okay we'll
- 8:23:43create an agent that uses our LLM and has access to the fruit info
- 8:23:48fruit info tool that you are seeing here and we will do the same for VG info
- 8:23:53okay so at this point we need to create two agents
- 8:23:58so I say create agent our
- 8:24:00model is going to be this LL that we stored in this variable and then we need
- 8:24:06to specify the tools for this agent we only have this
- 8:24:12tool and
- 8:24:13we should define here below these functions
- 8:24:16okay great and then we need to specify a system
- 8:24:20prompt and in this system prompt we are going to say
- 8:24:24it is a fruit expert and we will ask it to respond
- 8:24:27in one sentence okay so i say system prompt
- 8:24:31you are a fruit. Expert respond in one sentence.
- 8:24:38Now let's create the other agent.
- 8:24:42Come on.
- 8:24:43I forgot this comma. Another one.
- 8:24:47for the veggie expert, so here I say
- 8:24:50Oh, I didn't store it in a variable. Okay. So here I say fruit agent.
- 8:24:56And this one is VEG Agent.
- 8:25:00okay and this agent has access to this
- 8:25:04tool
- 8:25:05here we
- 8:25:06should say VEGExpert pay attention when you copy paste these lines
- 8:25:12that's good so at this point we have two specialized agents
- 8:25:16each one has its own responsibility and access
- 8:25:19to a tool relevant to that responsibility
- 8:25:22now it's time to create our third agent, the coordinator
- 8:25:26our main agent which acts as a supervisor
- 8:25:29and let's start by defining it, we need a new agent Here I say create agent.
- 8:25:36and I store it in a variable called mainAgent.
- 8:25:40Then, like before, we specify the model.
- 8:25:44and then we also need to specify tools and system prompt
- 8:25:50But for the moment, I'm leaving this list and system prompt MPT and I will explain.
- 8:25:56Okay.
- 8:25:57uh well the main question is that how do we give
- 8:26:01our main agent access to these two sub-agents
- 8:26:05and the key is that we need to make these sub-Agents
- 8:26:09available to the main agent as tools in this step
- 8:26:12we don't simply put the agent objects directly into the tools list instead we wrap
- 8:26:19each sub agent in a regular python function that
- 8:26:22invokes the corresponding agent and returns its
- 8:26:26response okay so at this step we need to wrap each
- 8:26:30of these functions in a function to make it a tool
- 8:26:35so first for the fruit expert i define this function i call it ask
- 8:26:42Fruit Expert
- 8:26:44It received a question.
- 8:26:46And the type is string and it also returns a string.
- 8:26:51then i put a description because i'm defining
- 8:26:55a tool so here i say ask the fruit expert
- 8:26:59Use for all
- 8:27:01fruit questions.
- 8:27:04and that's it then we need to invoke the corresponding agent
- 8:27:09so i say fruit agent
- 8:27:12invoke
- 8:27:13and as you remember we need to provide a dictionary
- 8:27:17and this dictionary has a key called messages
- 8:27:20and as a value of this key we need provide a list and
- 8:27:24in this list we have a user message so here i say role
- 8:27:29user and content
- 8:27:32question the question that this function receives
- 8:27:35that's fine and let's do the same thing
- 8:27:38for the vegetable expert so i just copy and
- 8:27:42paste this line i say ask a veggie expert
- 8:27:47and they say ask the
- 8:27:50ask the veggie expert used for all questions and here instead of invoking
- 8:27:55This agent I'm going to invoke VEG Agent.
- 8:27:59OK, there is nothing else to change.
- 8:28:05But here we forgot to store the result in a response and return it.
- 8:28:11here I should return response and I need to
- 8:28:14extract the last message and its content only
- 8:28:18so here I say messages and then the last item content
- 8:28:24And the same
- 8:28:26for here.
- 8:28:28okay response oh we have a typo here i guess yeah response
- 8:28:34okay so we have wrapped each subagent in a python function
- 8:28:39these functions can now serve as tools for our main agent
- 8:28:43and when the main agent calls one of these functions
- 8:28:47the function invokes the corresponding specialist and returns
- 8:28:51its answer now we can update our mainagent and provide
- 8:28:56these tools along with a prompt
- 8:28:59so here i say ask fruit expert which is a
- 8:29:03tool now and ask VEG expert. Okay.
- 8:29:07And then as a system prompt,
- 8:29:10I say you have two experts.
- 8:29:13and we specify the name of these experts it's better although we have provided
- 8:29:20these here and here okay and then i say
- 8:29:24always delegate questions to
- 8:29:29the appropriate expert and this
- 8:29:31is our system prompt
- 8:29:34so this is a core idea behind our supervisor architecture
- 8:29:38the main agent receives the user's request decides which specialist to call
- 8:29:43and uses the specialist's response to answer the user
- 8:29:47okay so let's the entire system now I invoke this main agent
- 8:29:54Then as usual, we provide a dictionary containing this key and then a list.
- 8:30:00And inside this list, we have our user message.
- 8:30:04And as a content
- 8:30:06we specify our question, we say, is a tomato a fruit?
- 8:30:12for sure inside quotations and then we store the result
- 8:30:17and then we print it.
- 8:30:20let's say messages, its last item and all its content
- 8:30:26Now let's run our program and see what happens.
- 8:30:30okay as you can see our system is working correctly
- 8:30:34and it says yes botanically a tomato is a fruit
- 8:30:37it develops from the ovary of a flower and contains
- 8:30:40and the other information and if you pay attention
- 8:30:44this answer is based on this information that we have developed here botanical fruit
- 8:30:51and yes this information is augmented so it was our goal and now we have a
- 8:30:58multi-agent system and we have successfully implemented a simple multi-
- 8:31:02agent system using the supervisor approach Ladies
- 8:31:07and gentlemen, you have arrived
- 8:31:10All 10 worlds are conquered.
- 8:31:15Take a moment, you've earned this.
- 8:31:18You've built chatbots, smart search, rag systems,
- 8:31:21AI agents, and teams of agents that work together.
- 8:31:26You are now an AI engineer. Thank you for flying with Neon Learn.
About this transcript
This page contains the full transcript of AI Engineering Full Course | 8+ Hours | LLMs, RAG, LangChain, LangGraph, AI Agents by neonlearn, generated from the public captions YouTube serves with the video. The transcript has 72,203 words across 7,637 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.